awesome

thanks for setting that up

I was planning on playing around with it, but will focus on something else

haha - I like that you kept lots-o-clojure

Definitely

man - JS is beating the pants off this JVM version

but now we know that with confidence

Really?

and can optimize

What’s the timing for JS on that test?

the values in the table here: https://github.com/shaunlebron/parinfer/pull/88

Is this the equivalent of the really_long_file values?

indent mode for really_long_file is around 12ms, 23ms for paren mode

Interesting

There are definitely improvements to be made, I’m working on them now

sweet

I am going to set up travis-ci to run the test suite on every commit

I think that should be not-too-hard to set up

wonder what happens to the performance if you put SENTINEL_NULL back in?

Hmm, not sure

I’d be surprised if it made much difference, but I was surprised with JS too

You could run the bench against the original version I guess

I might give that a try; going to set up travis-ci first

~/d/p/parinfer-test (master)> lein with-profile bench run
Evaluation count : 4920 in 60 samples of 82 calls.
             Execution time mean : 12.537784 ms
    Execution time std-deviation : 436.185958 µs
   Execution time lower quantile : 11.935113 ms ( 2.5%)
   Execution time upper quantile : 13.447069 ms (97.5%)
                   Overhead used : 1.707955 ns
Evaluation count : 5280 in 60 samples of 88 calls.
             Execution time mean : 11.775496 ms
    Execution time std-deviation : 408.468619 µs
   Execution time lower quantile : 11.253086 ms ( 2.5%)
   Execution time upper quantile : 12.753575 ms (97.5%)
                   Overhead used : 1.707955 ns

Found 5 outliers in 60 samples (8.3333 %)
	low-severe	 5 (8.3333 %)
 Variance from outliers : 20.6507 % Variance is moderately inflated by outliers

I replaced a few of your functions with stdlib equivalents that are more efficient, especially around repeated String concatenation.

Perhaps V8 is better at optimising that?

oh man 😄 😄 😄

this is quite exciting

Is it true that http://result.ch is always a character, or empty?

I believe so

Thinking about it, parinfer doesn’t do anything special for #( and #{, right?

hmmm

I'm not sure; Shaun would know the answer to that question

I know he had to make some change in order to support Racket

Are there tests around those cases?

there might be

it's his first week working at Stripe, so he's not around as much

although I spoke with him last night; he is pretty excited about parinfer-jvm as well as the JS speed-up

Fair enough

Here’s hoping Stripe will soon be running on Clojure and CLJS

yeah

here's hoping

Then I can sell them tons of licences - booohahahahaha

haha

gotta add parinfer to cursive first!

how hard do you think that will be?

I’m getting there, I’m getting there...

there can be some trickiness adding parinfer to editors

I can probably do a naive impl relatively easily, but there’s a few bits around the edges making it user friendly.

And I suspect I can make it more efficient with the lexer I use internally

yeah - are you familiar with the "parent expression" hack that atom-parinfer uses?

Shaun and I were talking about this last night; with the performance we're at now we might be able to just take it out

it's so fast already

Yeah, I am. I’m actually planning to benchmark just parsing the top level forms from the top of the file until, say, the last one in really_large_file, and then parinfer’ing that.

haha - mental slip

So no hack, but parsing to find the top level forms is faster.

you named a folder "paredit"

Whoops

I will fix it; I have to commit something in order to test travis-ci

Ok.

there might be a flurry of small commits in order to test that actually - nothing major

is it a convention to use the performance namespace?

I was going to change that to parinfer-perf/core ?

Sure, go for it.

ok

wasn't sure if there was a convention there; I assume there is for tests

does have DOUBLE_SPACE assigned to it, so it’s not just a char.

ahh

yes

it would be faster if it were just a Char on the JVM?

Not sure, I assume so, since that’s basically just a native int, not an object

Looks like that’s only used for replacing tabs

Ok, lunchtime, back in a bit.

I just pushed the perf improvements.

so sweet

that test takes forever to run on my systme

but I get similar results

and your commit incidentally tested the build script

which worked perfect the first time

so we now have an automated build / test

Yeah, the benchmark takes ages to ensure that the JIT has stabilised

should we mark a v0.2.0 with your changes?

Yeah, that’s probably most of the low-hanging fruit.

do I need to add that number to build.gradle ?

in JS and Python people just include the file in their own builds

in Java people use actual packages and .jar files, right?

Right.

version '1.0-SNAPSHOT'

Is what you need to change

and is it ok to follow semver format there?

version '0.2.0'

Sure, that’s what Maven usually uses

I suppose another question I have is: what should the public API look like?

right now it's ParinferKt.indentMode()

I’m actually not sure why it started at 1.0 and didn’t use three digits

drop the Kt ?

Yeah, I think so - no-one using it cares

ok

because it has to pass the type system I suppose

that sort of thing is pretty important in JS

also I went with 3-arity options instead of a hashmap of options

In fact, that class name is automatically generated from the filename, looks like.

since Java doesn't have literal data types

I’ll check what the best way to expose that API is.

(I’m still fairly new to Kotlin too)

ok

And while we’re confessing, that was the first Gradle file I ever wrote

I guess the question is: if this already existed and you were just adding it to Cursive, would what you want the API to look like?

lots of firsts for us both here

I think that the concept (a public object with two static methods) is fine.

I think we should probably create the object explicitly so we can control the name, rather than have it be autogenerated.

yeah - I saw there was a way to do that in Kotlin

@Jvmname

or something

but I couldn't figure it out

No, I think we can just use:

public object Parinfer {
  … api goes here …
}

I’ll double check and make that change if you like.

It’s interesting that in the V8 version, the paren mode is much slower

I wonder why

yeah - I was going to look at that tonight

that discovery yesterday gave me the itch to squeeze out more performance

it's not too often that that sort of thing is the best way to spend your time

working on perf like that vs working on features or design

(at least in my experience)

but in this case it's worth it

parinfer has to run constantly as you're typing

speed is essential

Yeah. I’ll have to check how fast it is in Cursive because it’s not just manipulating strings, it’s fiddling with documents that have locks and so forth.

Will you make that API change? I think I prefer just Parinfer to ParinferKt

Yeah, I’ll make it.

In the end, this is a cleaner solution:

@file:JvmName("Parinfer")

Then from Clojure it’s just (Parinfer/indentMode …)

And from Java Parinfer.indentMode(…)

Unfortunately getting this into Cursive is a little trickier than I thought.

I’ll need some help from someone at JetBrains I think.

In Atom, are files ever changed in the background? There’s nothing like refactorings or anything like that, right?

So these changes are only ever applied to a document while a user is interactively editing it?

Actually, what about if a file is updated from VCS, which is effectively in the background as far as Atom is concerned?

hey guys, just started catching up on messages here

@snoe: I like that idea you had yesterday! tracking here: https://github.com/shaunlebron/parinfer/issues/89

@thomas: by “block comment” do you mean “comment the selected lines” or “comment the lines of the cursor’s sexp”?

@cfleming: parinfer looks at all parens outside of comments, strings, and character literals. so it defaults to treating #{, #[ and #( correctly.

it doesn’t actually know anything about the preceding #, so it’s treated as if it wasn’t there

and secondly what you said here is good

I’m actually planning to benchmark just parsing the top level forms from the top of the file until, say, the last one in really_large_file, and then parinfer’ing that.

hey man

hey

like the activity happening here!

Colin is kicking total JVM ass; the benchmark on parinfer-jvm is cut in half from my original implementation

I saw it was around 40ms for indent-mode in really_long_file?

that's old news; now it's down to ~12ms

well then

that is good news

i can’t find the confetti emoji

I'm playing around with more JS optimizations

I'm a little obsessed with it now

little worried that the "clean implementation" is going to have more JS-specific stuff

that might not translate as easily to other languages

but that's not really a big deal; we can always link people to an older version

i think it’s fast enough

yeah

just seeing very little difference between indentMode and parenMode on the JVM

and thinking that probably that can be achieved in JS too

I think in reality locating the parent expression is the important part

and I feel like that circuit is finally closing

colin can just use his lexer to locate that

emacs already has fast support for that as well

and for editors without their own native support for locating it, I can put the fast reader back in the API

parinfer without all the transformation business can run through really_long_file in about 7ms

idk why i didn’t think of this earlier. chatting with the emacs guy today helped I think

https://clojurians.slack.com/archives/parinfer/p1454558963000385 atom-parinfer only ever looks at the current buffer

@shaunlebron: I added a couple examples to the issue.

@cfleming: in atom, if a file is changed in the background, atom-parinfer won't run until the user does something to trigger it, like press any key on the keyboard

( 🎉 btw)

ah, thanks it’s :tada: 🎉

I think we’re moving into paredit territory

from your example @snoe, you want an indent-next-sexp function, not a indent-next-line function, since it indents the child expressions with it

secondly, you want it to indent two spaces

the offset from the tabstop would have to be configurable

I think the parinfer API can provide some primitives that allow parinfer plugins to handle this behavior themselves

@shaunlebron: !!!!!!!

I just figured out the problem with paren mode

?

parinfer.js is about to get faster

https://gist.github.com/oakmac/7639728fc162045c4f3b

I agree, maybe parinfer could return something like indentMode(..., {...}) => {:text "..." :tab-stops [1 4 8 11]}?

or maybe a separate function 😕

The make edit, goto next line, select form, indent/dedent selection dance is still something I find myself doing all the time. I wonder if others do the same thing?

@chrisoakman: sheeeeeiiit

https://github.com/shaunlebron/parinfer/pull/90

that must have been preventing some v8 optimizations

@snoe: I think that info will be enough for one-space indentation

you’ll need more info for ( to be two-space, and for (expr arg to be n-space indent

yeah, the important part is making the structural change. formatting can/should be handled outside.

and whatever rules required for determining that stuff

I see what you’re saying, so do a separate auto-indent operation after parinfer moves it to one-space indentation

@snoe, seems like we need either indent-selected-lines option instead of a single line

right, I'm not sure if there's a real difference between forms and lines

or some way to specify the lines of an sexp

Maybe : 1) go to next line -> 2) find first char -> 3) keep going down until you find a char before step 2

gah - I am super-happy about that find

I had a gut feel it was a JS thing

parinfer.py and then parinfer-jvm showed that the algorithm should be returning similar result times for both functions

something was clearly happening in the JS version

I'm not sure we're going to get faster than that; I was playing with replacing the if/else with dispatch and such

but doing object key lookup in JS isn't free; in most cases it was hurting perf

@chrisoakman: cool, I wish we had something Criterium for node

it's v8 optimizations

that's what this boils down to

i can’t figure out why the last two tests are any faster than the previous

I considered changing the perf.js to run multiple iterations and then show the average, etc

I only included one result on that PR, but ran it several times on my machine

it might be saving some data from the first test

and it was consistently faster when Object.preventExtension was removed

some caching I mean

to be honest, I "discovered" that earlier today on a Windows machine running an older version of node

Have you guys run parinfer.js through the the chrome profiler?

saw a huge bump when I removed that

yes - but I can't seem to make heads or tails out of the .log files it produces

in both of these last cases of "discovering" big speed improvements, it's clear they are v8 optimizations

not fundamental algorithm or data structure changes

I did see some perf improvement in parinfer.py by changing that if/else chain to a Dict lookup

https://github.com/oakmac/parinfer.py/blob/master/parinfer.py#L278-L304

but when I did that in JS it was slower across the board

I'm looking at Python string concatenation now

@chrisoakman: you were asking about the next language to tackle maybe if you do C/C++ you could then use emscripten for js (or c node module)

Shaun already went down that road a bit

with c++

and decided to scrap it

because the JS version is fast enough now

I'm thinking emacs lisp should be the next target

I don't use emacs, nor terribly wish to write the plugin

but I think writing the parinfer algorithm would be fun

I also want to do it in F#

but only because I want to do something in F# 😉

I don't know what editor would benefit from having an F# implementation; I don't think there are too many people using MS Visual Studio to edit Lisp code

it could be the reason they start

true

another benefit of writing Parinfer in emacs lisp is that I get to use parinfer to write it in !

which is cool and recursive

not sure if I added enough context about finding the parent expression in a file

but cursive, emacs, and parinfer stripped down to its reader form only, all of these can locate a parent expression very quickly and reliably

so it’s like using atom-parinfer’s parent expression hack, except without the corner cases

I am seriously considering dropping the parent expression hack for files under a certain N length

is what I said clear

I mean - parinfer.js can do a 2800 line file in under 10ms

that's the debounce rate

just to put that in perspective

yeah for what it's worth I haven't bothered with expressions in vim because the whole file is pretty much fast enough

I am willing to wager that 99% of lisp files that people regularly work with are under 1000 lines of code

there are obviously exceptions to that, like if you're working on cljs.core

what i’m saying is that it’s now possible to hit 100% reliably and quickly

are you going to add another function to the API?

is what I said about the parent expression thing clear?

vim's select top form: https://github.com/guns/vim-sexp/blob/b4398689f7483b01684044ab6b55bf369744c9b3/autoload/sexp.vim#L1054-L1211 hehe

maybe not?

you use regex

emacs and cursive can do the same as your regex, but reliably and as fast

so can parinfer’s reader

by just keeping a running data structure of the whole file?

i’m not sure how emacs and cursive do it

but emacs’ paredit and smarparens plugins have fast implementations for it

the point is that data is available for parinfer to use

I think atom uses regular expressions for the grammar engine

actually, I know it does

for syntax highlighting and whatnot

which is probably a decent decision for most languages, but probably the wrong one for lisp-based languages

where it would be cheaper to just keep the actual AST

right, that will not work for parinfer

breaking syntax highlighting will not corrupt your code, only its coloring

thanks @snoe! this looks how most editors would locate the parent expression: https://github.com/guns/vim-sexp/blob/b4398689f7483b01684044ab6b55bf369744c9b3/autoload/sexp.vim#L1054-L1080

i want to test that in vim to see how well it works

heading out for the night

@snoe, I’ll keep thinking about the indent thing

did you bump parinfer.js on npm?

@chrisoakman: thanks for speed improve, will publish

kk

I don't think we're going to get too much faster than this, btw

would be interesting to see how Chakra or SpiderWhatever performs

thanks man

Has anyone got parinfer on emacs yet?

Last I saw, the strategy was to rewrite parinfer in emacs lisp, instead of using the lib

I thought there was a link on the parinfer web site… but not sure what it said

There is. The first is a very limited rewrite in emacs lisp, the other is a node server of some kind, which you've got to start to use.

Maybe I'm just spoiled by @snoe's excellent neovim integration