Fork me on GitHub
#calva
<
2018-12-17
>
mseddon13:12:11

https://github.com/mseddon/calva-fmt/tree/restartable-formatter so this is now still a bit too eager, but is incrementally lexically analysing the entire source file correctly.

mseddon13:12:32

need to optimize it a little bit more- since e.g. if you type " in a file, now you have inverted the ENTIRE document's state as inside and outside a string. So this needs to be lazily computed when necessary rather than rescanning everything below the line that changed.

mseddon13:12:04

still, once this is handled, we should be able to amortize everything really nicely, and have a very fast way to find the enclosing sexpr at any part of the file

mseddon13:12:37

I will be tracking the enclosing sexpr, the indentation rules for that sexpr based on the start symbol + a table, as well as the index this expr is in it's parent, which should be enough to recreate cljfmt

mseddon13:12:20

but then- finding the required indent will be O(1) for extremely small 1, so everything should indent instantly like it does with e.g. typescript

mseddon14:12:01

it will also allow us to grab exactly the lines required to pass to cljfmt and parinfer

pez14:12:54

Awesome. I'll have a look tonight.

pez14:12:57

We will need to be able to feed it custom cljfmt configs, but from what I hear that is not a problem, right?

mseddon14:12:30

no problem at all- we just strap an edn reader on front and we're done πŸ™‚

mseddon14:12:39

that'll be how we get the original cljfmt stuff too

mseddon14:12:54

some more advanced settings- i am not sure how they work yet, but we can address those later πŸ™‚

pez14:12:04

Reading EDN isn't hard for calva-lib.

mseddon14:12:16

nope πŸ™‚

mseddon14:12:29

right, I have to walk this dog. catch you later!

pez14:12:16

I use a js edn reader for Calva today, but found that it is to flaky, will switch to doing it via calva-lib, since now Calva has that in reach.

pez14:12:26

catch ya!

manas.marthi15:12:31

hi guys I see there has been a lot of chat since I visited this channel last time. Recently I heard of Tree-Sitter API added to Atom for source code parsing. Is it going to be added to code? Any idea if Calva can use it?

manas.marthi15:12:24

I have one issue to report though

manas.marthi15:12:47

if a source file has errors in it, and if I evaluate the file, I am not seeing the stack trace

manas.marthi15:12:59

I had to evaluate the file in normal repl

manas.marthi15:12:04

by restarting it and then found it

mseddon15:12:16

regarding tree-sitter- I am implementing a similar thing but specifically for calva/clojure

mseddon15:12:31

the stack trace thing I am not sure, perhaps @pez knows?

manas.marthi15:12:23

oh okay. why not just take the Tree-Sitter bindings. The reason is we can get all the future bug fixes for free

manas.marthi15:12:36

(I am not sure how to create bindings though)

mseddon15:12:45

that's true, although Tree-Sitter's back end is in c, so that may be overkill

manas.marthi15:12:07

I think they were creating API bindings for other langs

manas.marthi15:12:13

as per what I saw on the youtube talk

mseddon15:12:16

ah, no, i mean the actual parser itself is in c

mseddon15:12:33

and I am unsure how we distribute the executables

mseddon15:12:38

oh, it has typescript typings too

manas.marthi15:12:01

It allows for creating new parsers

mseddon15:12:21

huh, I will take a look at that later this week, thanks!

mseddon15:12:45

it is interesting, since it's fairly easy to write an incremental parser for lisp, but they are using some far more complex research to handle lalr languages

manas.marthi15:12:33

My opinion is leaning on deep pockets is better

manas.marthi15:12:54

Let them take care of the complexity of the parser

pez15:12:56

Tree-Sitter is probably not coming to VS Code. They are considering adding it temporarily until they get the parser ported from Visual Studio.

manas.marthi15:12:14

and let them pay for the maintenance and bug fixes

pez15:12:52

Yes, that was the issue I read. Along with a lot of linked issues.

pez15:12:11

Doesn't mean we can't use Tree Sitter, though.

pez15:12:45

But we need only parse clojure code, which is less complex than the task Tree Sitter has to perform.

mseddon15:12:17

which can be done very easily since it's an LL(1) language

pez15:12:49

If anything we should be looking at rewrite-clj, me thinks.

manas.marthi15:12:25

oh okay. noted. I have not heard of rewrite-clj before. I will bing it

mseddon15:12:38

It's great, although it's not an incremental parser per-se

mseddon15:12:09

but the nice thing is I can build incremental edits to sync that up further down the line

manas.marthi15:12:12

just curious, have you seen the source code of emacs packages that do code parsing?

manas.marthi15:12:31

I don't know if emacs has any incremantal parser package

mseddon15:12:54

I am kinda basing the incremental parser on how emacs does it's thing

mseddon15:12:40

really it doesn't do much 'parsing', but it tracks lexical state across lines so e.g. it doesn't get confused by an opening '(' inside a string while looking back

mseddon15:12:22

for the bulk of the work, we just need to know at any position in the file what the extent of the sexpr is, and the formatting rules that apply

mseddon15:12:04

so this incrementally lexes the lines as you type, and marks lines as 'dirty' if some change has caused us to e.g. move into a multiline string, which is very cheap

manas.marthi15:12:11

as clojure has square brakcets and curly braces

manas.marthi15:12:18

we need to track them as well I guess

mseddon15:12:43

yep. so in addition to the state 'are we in a string?' we store the stack of parenthesis types, along with their opening symbol and indent

mseddon15:12:00

but we don't compute this every keystroke, we amortise it when we need it

mseddon15:12:39

but this lets me VERY quickly determine what the indent should be when you press return, so we will lose that delay before everything lines up correctly

mseddon15:12:45

(which is basically why I'm doing this at all)

manas.marthi15:12:55

someone complained about how slow emacs can be when editing large files

mseddon15:12:04

emacs is not great on long lines

manas.marthi15:12:08

I hope we might have to watch for those as well

mseddon15:12:33

currently I have similar issues with my lexer, although hopefully it won't be too terrible. But if you try creating a file currently with say 25k lines of clojure

mseddon15:12:49

and then press 'enter' in it, you'll see we have a terrible delay before the indent happens

manas.marthi15:12:59

will writing the parser in rust and generating wasm bindings help with the performance?

manas.marthi15:12:27

I started working on Rust wasm generation sometime back when I heard of wasm and webcomponents

mseddon15:12:36

it would help with raw speed, but currently we have lots of low hanging fruit

mseddon15:12:56

so since this is relatively simple to write, I will start here and then port it over to something faster if we need it

manas.marthi15:12:16

wt's the repo link?

manas.marthi15:12:21

got it.. thanks..

mseddon15:12:34

it doesn't do anything to the editing experience yet, but it's mirroring the state of the lexical analyser inside the calva-fmt/docmirror directory

mseddon15:12:42

(all very typescripty atm sorry about that :))

mseddon15:12:55

oh, it's now also not updating the lexical information for lines that weren't directly edited unless flushChanges is called, which should bring everything back to consistency again.

mseddon15:12:12

(and nothing calls it in that repo atm, it's commented out in processChanges ;))

manas.marthi16:12:45

oh okay. I don't know how to test vs-code plugins yet.. a bit behind on the learning curve

mseddon16:12:51

In the case of this- you should be able to check out that branch, npm i the dependencies, and start ts compiling with npm run watch-ts.

mseddon16:12:25

Then press F5 and you'll launch a vscode debug instance. It'll warn that calva-fmt has been replaced (but that's expected, since.. it has been swapped out for the new version here)

mseddon16:12:00

you can then stick breakpoints in etc and poke around and see what is happening, although other than maintaining that data structure, there is no user-visible behaviour yet

manas.marthi16:12:17

oh okay.. thanks a lot

manas.marthi16:12:23

I will try that

mseddon16:12:37

I am hoping to have it working somewhat at the end of the week though, and it would be great to have some early testers on it to flush out all my bugs πŸ˜„

manas.marthi16:12:08

I will try. I hope calva matures soon and competes with other IDEs.

manas.marthi16:12:20

tired of so many things that don't work behind proxy

manas.marthi16:12:31

spacemacs, atom, lighttable

manas.marthi16:12:37

all are useless behind proxy

mseddon16:12:42

desperately trying to get calva there. I am sick of having to switch to emacs whenever I use clojure πŸ™‚

mseddon16:12:13

I have high hopes for the project though, the issues primarily are due to lack of time and other contributors.

manas.marthi16:12:32

I can understand that. It's a pain all over the clojure community. So many developers for JS but too few in CLJS

mseddon16:12:05

It was the same with Common Lisp πŸ™‚

pez15:12:34

About the stack trace, @manas.marthi, I am aware of that those are often swallowed by Calva, but I would appreciate an issue filed so that I get reminded about it, preferably with some small repro.

manas.marthi15:12:32

I will reproduce the issue and file a ticket

mseddon16:12:22

huh. with a 25k line file even reconciling the lexer for the entire file is fast for me. πŸ™‚

mseddon16:12:51

this is also tracking the parenthesis, too

mseddon16:12:40

Definitely we need to rebind Ctrl+W to use this datastructure instead of parinfer

mseddon16:12:09

I need some aggressive tests, and I realised I can be even lazier with reconciliation, because I can ask "I need everything from this position up to the next N close parens", so it can bail early there, too

mseddon16:12:20

and.. i need to handle parens within the line, but that's easy

mseddon16:12:08

basically at the moment I can instantly get the parenthesis stack and the source locations of them for the start of any line in the file. so that's nice.

mseddon16:12:36

(I can also immediately detect mismatched close parens, but I don't do anything about that atm)

pez17:12:05

Can you correctly find the right current enclosing form in an expression like '(foo)?

mseddon17:12:04

Well modulo any bugs :)

mseddon17:12:59

And if that close paren is a few lines down and we are deeply nested inside expressions i only have to reconcile up to the first ) in that case

pez17:12:52

Cool. Maybe it's time to replace paredit.js, at least for when we use it to figure out the current form. There are a few evaluation commands in Calva that use that.

pez17:12:46

Also, of course, the whole Calva Paredit suffers from that paredit.js is abandoned.

mseddon17:12:31

Yeah. It's probably not too hard to do lexical slurping and barfing with this, actually

mseddon17:12:40

But... small steps :)

mseddon17:12:32

Oh, also we can use this to add breadcrumb support quite easily πŸ˜€

manas.marthi17:12:26

what is breadcrumb support?

mseddon17:12:34

Oh, they are those little bits along the top of the editor. Currently you have the directory path but we can show the heads of each expression, too.

manas.marthi17:12:53

oh okay.. thanks..

pez17:12:03

Also, maybe to implement the # command?

pez17:12:30

Press CMD+P, then #. In a TS file.

mseddon17:12:55

Ooh, right yeah

pez17:12:06

Of course that can be done via nrepl, but seems to be a perfect match for static analyze.

pez20:12:38

Hi there, @shane. Welcome!

marc-omorain21:12:14

I have an interesting edge-case. I have the following in my editor:

;(clojure.test/test-vars [#'get-namespace-works])
(clojure.test/run-tests)
When I have my cursor on the second line, in column 0, before the opening paren (the editor highlights the matching parens around (clojure.test/run-tests)), and I hit β€œEvaluate current top level form”, it evaluates the common on the first line:
Evaluating:
;(clojure.test/test-vars [#'get-namespace-works])

Evaluation failed for unknown reasons. Sometimes it helps evaluating the file first.

pez21:12:22

I can’t reproduce that. Maybe it is something else in the file tripping paredit.js up? What happens if you have only those two lines in the buffer?