@shaunlebron: the indentation behavior you describe in the video is something i’ve been thinking about a lot lately. currently my editors are just indenting/dedenting by two spaces but i’ve definitely noticed a need for something more intelligent. i’m going to tackle that next.

sekao: See here for the logic I implemented: https://github.com/shaunlebron/parinfer/issues/89#issuecomment-202176311

There’s a couple of bugs in the current release, I’m hoping to make a release fixing them today.

@cfleming: @sekao: I just proposed a solution you can read about here: https://github.com/shaunlebron/parinfer/issues/89#issuecomment-205525034

I’ll put that together tonight if we can agree it’s okay

@shaunlebron: In the example there, the let binding vector should not be a tab stop, I think.

It’s shadowed by the (foo bar)

good catch

Here’s how I calculate it:

Mine is interactive on the tab action, so I work backwards from the caret line until I find a line with content at offset 0, i.e. a top level form.

(corrected the example)

Then I work out the list of tab stops for that line.

Then, for subsequent lines I trim all tab stops that are greater than the indent level for that line, and add any new tab stops.

Once I reach the caret line, that’s my list of stops.

> I trim all tab stops that are greater than the indent level for that line

that seems wrong I think

unless I’m misreading

Isn’t that the rule? When would you want a tab stop that appears to the right of the subsequent line’s indent?

nevermind, yeah that’s it

I also return extra stops as I mentioned in my comment, they’re all trivially able to be calculated except for the one where the user may want to align parameters.

as in checking the previous sibling line?

here’s an example:

or subsequent sibling line too

(println x
         y)

Here, you want a tab stop where the parameters appear.

oh yeah, first arg alignment

That’s the only case the parinfer caller won’t be able to trivially add themselves.

Adding another inside the collection for the various open delimiters is trivial, obviously.

i suppose they could run a regex for the first space after the open-paren I would return them

well, first space outside a string

and outside any collection in the first position...

yes, non-trivial

Yeah, I guess that’s trickier for you, I’m doing this based on a lexer so I just look to see if after the delimiter I have symbol/keyword then whitespace then something else on the same line.

And if so, I add the start offset of the something else.

awesome, so you’re using cursive’s lexer here

I’m actually about to re-work the Cursive lexer, the current one is a horrible JFlex monstrosity.

Once I have it, I think it’ll actually be some pretty simple Java code. It might be worth porting to JS and using that for parinfer.

yeah, we should talk about standardizing the method here

When I profiled parinfer, a lot of the time was spent updating all the state per char - by using a lexer, you could do it per token.

And you also would remove all the inString, inCode, inComment stuff totally.

right, I consider that stuff a mistake

Well, you have to do it, but if you used a lexer it wouldn’t be required.

have to do what now?

Since you’re processing per-char, you have to track your current state - are you in a string, or a comment, etc.

right

I wonder how switching to a lexer would affect compatibility with different lisps

Yeah, that would be an issue.

I straight up don’t care about anything except strings and collections and comments

Parinfer actually broke my code the other day, I haven’t had time to reproduce it, but it was using an sexp comment.

sexp comment?

I’m not sure if that’s a bug in the Cursive implementation or not, I would have thought that vanilla parinfer should have handled it.

I had some code I had commented out with #_#_

That pushed the code over to the right, and indent mode broke it.

What I’m not sure of is why paren mode didn’t fix it - I wanted to reproduce it to see what was going on there.

#_(defn foo [a b])
  (+ a b)

Right.

yeah, that is actually the case that made me start thinking about paren mode

But I didn’t make the change in parinfer, it was old code, so the paren mode cleanup should have fixed it.

So either it wasn’t run correctly, or it didn’t fix it for some reason.

paren mode cleanup, not sure what you mean byt that

When paren mode is run upon opening the file.

To correct the indentation.

oh, yeah, paren mode should fix something that would look like this:

#_(defn foo [a b]
  (+ a b))

before indent mode ever corrupts it

Right.

It didn’t, for some reason, but I haven’t had time to investigate why.

just pasted that code into the demo editor just to verify

so I wonder what happened

let me know if you see it again I guess

It’s entirely possible it’s a bug in my integration.

(demo editor corrected it after pasting)

BTW I’ve been thinking about how a potential hybrid mode might work.

I’m going to try to implement this once I get the current implementation working.

I think one of the biggest problems for people with parinfer is that the changes have to be global.

Several people have commented to me that they can’t use it at work because it modifies the whole file, not just where they’re working.

oh sorry, before we talk about this, is the tabstops implementation an agreeable thing, since @sekao will be needing something like that too?

And in particular, the fact that paren mode tidies up hanging closing parens is a problem.

Yes, definitely - if you can provide that I think it will be very helpful. I definitely think you’ll need the first parameter align, but I’m not sure how to implement it as parinfer is right now.

I think a regex is probably your best option, ideally parinfer would do that internally.

There are lots of edge cases there.

I’ll have to leave that out until a lexer makes it into the implementation I think

but I wouldn’t be returning the extra tab stops for 1-space or 2-space indentation after open-parens

Yeah, that’s fine - those are easy to add, and may depend on the user’s config in the editor anyway.

that’s entering into convention territory that parinfer isn’t touching right now

alright, I’ll add that in

i.e. whether you want one or two space indents for lists. I’m not handling that case right now, nor am I handling the two-space-for-everything indent case.

But they’re both quite easy to add.

agreed

interesting that you mention people commenting on global changes

valuable feedback

Yes, it’s an issue.

yeah, and I suppose paren mode is sort of a linter when it comes to hanging close-parens

In particular, people use dangling closing parens… right.

I was going to ask if that was an essential part of the algorithm, but indent mode would tidy them anyway.

hmm

So here’s what I’m planning.

I’d have to revisit to see if it’s essential, but go ahead, yeah

In Cursive, when changes happen to the document, I get before and after ranges for each modification. I use that to determine the total range affected by a particular action, but I’m not using that final range right now.

However using the lexer, I can use that range to find the innermost form which completely covers the change.

If the change is to top-level forms there may be multiple forms affected by a change, but that’s fine. Basically, I get a minimal range of the document I have to run parinfer over.

ah I see

On any change, I’m going to run a slightly modified paren mode, so the indentation will always be maintained.

And the number of doc changes I’ll have to make will be minimal.

I can actually check - if none of the before or after ranges for a particular action contain any of ()[]{}”\n then the change only affects a single line and doesn’t affect indentation - I can optimise that case and just insert spaces.

This covers the very common cases of typing, backspace, delete etc.

If any of the changes do contain those characters, then I’ll run Cursive’s indentation on the range of lines affected by the innermost form I worked out previously.

So the indentation will be semantically accurate.

can you describe the before and after ranges more?

If you type a space, I’ll get a change saying something like: at offset 10, old-length 0, new-length 1, old-text “”, new-text “ "

If you then backspace over that, I’ll get the opposite change: at offset 10, old-length 1, new-length 0, old-text “ “, new-text “"

perfect, yeah

Or if you select a word and paste a new word in it’s place, you might get: at offset 10, old-length 4, new-length 3, old-text “foos”, new-text “bar"

that’s inline with the change object in codemirror, cool

There may be multiple changes like that for a particular action, but I use those deltas to find the total range in the resulting document affected by the change.

Using that and the lexer, I can then fairly trivially find the innermost form enclosing the whole change.

After any change, I’ll run my modified paren mode, so I’ll maintain indentation with minimal changes.

I’m with you so far

Then I’ll have explicit indent and dedent actions, using tab and shift-tab as I have now.

Those actions also affect a range, since they affect either the current line or the range of selected lines.

So on those actions, I’ll work out that range. I’ll then run vanilla paren mode over it, to ensure that the indentation is within the bounds required for indent mode.

Note that the indentation only has to be correct for that range.

And I can correct it on the action, not at file open.

If the indentation requires correction, I’ll do that instead of the indent action, and provide an “indentation corrected” action to the user.

Otherwise I’ll indent the lines, and run indent mode over the range to correct the parens.

So I’m basically treating parens as the source of truth, but allowing the user to explicitly use indentation to adjust scope when they want to.

this is intereseting

And only affecting the part of the doc that the user is touching.

@cfleming: the cursive lexer would be amazing to have in js, but I'm afraid it's kind of your secret sauce

so tab will dispatch to paren mode if indentation is incorrect, but if it’s correct, it’ll run indent mode

It only makes very local changes

Right.

@snoe: I’m planning to OSS it, it’s not that secret and it would be really useful for this sort of thing.

I think that will provide the best of both worlds in a single mode, and also address a lot of the complaints I’ve seen about parinfer.

I think this is promising. I wonder about auto-closing of parens

Yeah, there are probably edge cases to be considered, I’ll try it out.

it’s weird because indent mode is really doing two things

correcting structure based on indentation, and auto-closing parens as you type them (based on indentation)

You do lose some things from vanilla parinfer: being able to indent using spaces, and the slurping/barfing.

(and of course, it’s good at breaking expressions inside a line in certain cases…)

I can probably fix the first by checking if the user has hit space, backspace or delete inside the leading indent of the line, and doing the indent/dedent action in that case.

I’m also starting to think that a lot of this functionality should be in composable pieces rather than all-in-one modes.

Currently I have a parinfer mode, a paredit mode, and a none mode.

But I had to make a change because users missed some of the electric handling of open parens when in parinfer mode.

oh, I see. yeah, it was my hope that parinfer would be compatible with paredit

And really, parinfer mode is mostly the backspace handling over closing delimiters at that point, since the actions work in any mode.

electric, like moving close-parens when pressing enter?

So with this proposed change, the actions would always work anywhere, and users would just choose the options they want.

In parinfer mode, if you backspace before a closing paren, it jumps over it rather than deleting it.

right

Which is wonderful or maddening, depending on preference.

end-of-line close-parens, yes

which I why I think it’s important to dim them in the editor to communicate their nature

Sorry, I meant in paredit mode there.

parinfer too, in indent mode

That’s something I explicitly added rather than something that falls out of the parinfer algorithm.

Right.

it works that way in the demo editor

So instead of the mode switcher I have right now, I’m imagining that popping up a palette of options - want tab bound to scope manipulation, want auto-indent, want clever close-paren handling, etc

And there are no explicit modes at all.

Users will be able to mix and match the parts they like of each mode.

so, to recap:

I’d need to try it - I suspect that removing the global nature of parinfer is a good thing, but I’m not sure until I experiment.

it’s all about representing the intention of the user

in parinfer, that intention is in modes

with this new mode, intentions are represented from a mapping of keys -> actions

tab or shift-tab -> scope manip

or, not so much mapping of keys as mapping of the types of changes being made by the user

so I insert ( -> auto-insert ) somewhere

of if I add or remove spaces at the leading indentation point of a line -> scope manip

so you can mix and match these things

Hopefully, yes

I think it will be possible to get something which is as intuitive for the user as parinfer right now, but without some of the confusing edge cases and without the whole-doc modification.

about the whole-doc modification

I do have some users who don’t like any sort of paren matching at all, I’d have to ensure that even when the doc is fairly broken these operations do something sensible.

I thought the diffs might be a problem for people at work, yeah

but I also thought it was making the code follow uncontroversial standards across all Lisps

One thing I have learned about lisps is that there are no uncontroversial standards

it’s like running a linter on your code base, there’s be some diffs, but only once

So for example, one user wrote to me saying that they have some big file which for whatever reason is only ever added to.

I mean, hanging parens, that’s done mostly out of convenience for line insertion

So they always maintain a blank line at the end so that they don’t get diff conflicts when someone adds to it.

Parinfer messed that up, so the guy always had to remember to turn it off before opening that file.

Another common problem is forms with hanging parens with comments inside them, ; new tests go here or whatever

yeah, @chrisoakman was thinking about that when proposing standard comment directives for disabling parinfer

The parens get pulled over the comments.

yeah, that’s a tricky one

The other nice thing about all this from the implementer’s point of view is that it all happens as a direct result of user action. The async stuff plays hell with undo-redo.

oh, I remember warning about managing the history stack with parinfer

Yeah, it’s really hard.

because of the async?

if the ops were done in sync, maybe not?

I think Chris had problems with that, and it was killing me in IntelliJ. I actually found a much better way of handling the changes which alleviated that somewhat, but it’s very editor dependent.

Right, depending on the editor it’s easier. I found a listener in IntelliJ which allows me to insert new changes into an undoable change so they all get synchronously packaged up.

i think it was something like replacing the top of the undo stack when parinfer runs, or something

And the fact that it’s synchronous helps responsiveness a lot.

but I’m missing the subtleties obviously

Well, I’d guess it’s editor dependent if you can even do that.

I don’t think IntelliJ allows it, but I may be wrong.

Anyway, think about it a bit, feedback very welcome.

I’ll try to implement that at some point relatively soon, although I have some other fixes that need some attention too.

I’m going to re-work the lexer soon and benchmark it against the old one, and I’ll OSS that - that’ll be Java but porting to JS should be easy.

not sure I can use a clojure lexer for parinfer

it’ll be interesting to see though

Yeah, I guess it’ll depend on whether you want to support other lisps or not.

definitely do

it’s working for racket, and guile so far

In that case it’s difficult, yeah.

Do no other lisps have tricky string handling?

they seem to have the same strings I think

i.e. allowing strings delimited by single quotes, or something?

oh

that would suck

Or chars like ‘c'

luckily I think that’s reserved for quoting forms

Oh, of course

(no tea yet this morning)

haha

well, I just burned my last hour at work thinking about parinfer

Haha

I’ll keep thinking about the auto-mode, I think it sounds cool

can’t wait to try

i’ll add the tabstops thing too

(Full disclosure requires me to point out that it’s basically what the Emacs parinfer-criticism doc suggested)

oh, and especially the locally-bound changes sounds really useful