Fork me on GitHub
#calva
<
2021-07-05
>
bringe03:07:58

Thanks, @U01LFP3LA6P, for the output window improvement!

Tomas Brejla08:07:14

Glad to have helped. Unfortunately the output still gets quite sluggish after 10k lines so I still often find myself jumping to that window and manually deleting all the content. But I guess the improvement helps quite a lot. Printout of 500 lines took almost 25 seconds for me without batching (with empty repl window), now it's under second.

bringe00:07:07

Yeah, I'm not sure how much more performant we can really make printing to an editor. Adding an option to print to an output channel would be helpful, though.

Tomas Brejla08:07:54

@U9A1RLFNV what is actually quite interesting is the fact that... 1. when I keep appending lines via evaluation of some expression containing println, it gets quite slow after line count in output.calva-repl reaches approx 5000 - 10000 lines. 2. but now when I enter the output.calva-repl window and perform Ctrl+a, ctrl+c, ctrl+v (copy all + paste) several times, the editor is actually responding really quickly. Including syntax highlighting etc. 3. (when I'm combining steps 1 & 2, I often end up in situation, that my output.calva-repl and evaluation somehow completely freezes 😞 ) Any idea why copy-pasting acts so quickly, compared to printing from expression?

pez08:07:19

We should provide an option to let stdout go somewhere else.

☝️ 2
bringe16:07:50

Yeah, this is the plan ^. @U01LFP3LA6P I'm not sure. It could be that pasting operates differently behind the scenes than when we use the API to apply an edit.

Tomas Brejla16:07:51

But it's weird, isn't it? Is there perhaps an API which would mimic the paste operation, so that one could try to perform some benchmarks and comparions?

Tomas Brejla16:07:57

When I think about it now, I wouldn't be surprised if the editing api actually performed (perhaps even multiple?) saving operation of the content, while copy-pasting doesn't do that. Also I remember that in the past I was trying to tail-pipe the output.calva-repl file to stdout and it seemed that the content doesn't get appended to that file, but the file is always written "from scratch". Not sure if that's really true or not, it just behaved like that. Btw having option to forward stdout to somewhere else would be super-nice, but it is also good to have integrated window with your repl output, directly in your vs code window.

pez20:07:02

Is writing the same amount of output in one edit operation slower than pasting the same chunk? Or is it writing as the output arrives that is slower?

Tomas Brejla22:07:27

my observation is that _vscode_._workspace_.applyEdit(edit) itself starts to get sluggish very quickly. Even when the edit effectively appends a single line, the operation takes... • around 10ms when appending to empty document • around 90ms when appending to doc with 1000 lines • around 200ms when appending to doc with 2000 lines • around 900ms when appending to doc with 4000 lines • around 3000ms-1000ms when appending to doc with 10000 lines sweet spot for testing the delays seems to be around 8k lines for me. What is interesting is that • when you manually keep appending yet another 8k lines again and again and again... (via ctrl+v) it's instantious. Like super-quick. • but when you manually select last line (or just a single character, really) and try to delete it, it takes a 1-2 seconds delay. • when I open foo.txt, put the same 8k characters, try the same operations... it works just fine, appending is quick, deletions are quick.. everything just flies. So perhaps the syntax highlighting somehow triggers on full document instead of just the appended part?

pez22:07:11

> but when you manually select last line (or just a single character, really) and try to  delete it, it takes a 1-2 seconds delay. This could be because what’s really happening is Paredit delete backwards. Try with alt+backspace.

pez22:07:16

> when I open foo.txt, put the same 8k characters, try the same operations... it works just fine, appending is quick, deletions are quick.. everything just flies. Does that include _vscode_._workspace_.applyEdit(edit)?

Tomas Brejla22:07:14

can't say, really, I'm just comparing user editing directly in the editor window.. ie. in both cases not using API

Tomas Brejla22:07:06

can the paredit support be disabled or short-circuited somehow quickly? just for testing..

Tomas Brejla22:07:55

when I manually add 8k lines into output.calva-repl , try to manually delete last line, it gets stuck. I can add new letters, but any attempt to delete (either via Backspace, del, or alt-backspace, alt-del) doesn't work.. seems to be blocked. I've seen this behavior before in the editor and we were already trying to pinpoint the root cause of this.

Tomas Brejla22:07:35

funny thing is that one can delete the line super-quickly by selecting the row and pressing space 🙂

pez22:07:10

I’ve seen that behaviour with deletes not working too. Never figured out what gets stuck.

Tomas Brejla22:07:38

now it seems to me that it can quite easily be reproduced

pez22:07:41

Paredit delete is structural. So for a large file, lots of things are going on. Pressing space is not structural. But alt+backspace shouldn’t be structural either, so that should be as quick as pressing space.

pez22:07:11

> now it seems to me that it can quite easily be reproduced That gives hope. 😃

Tomas Brejla22:07:39

i deleted the content of output.calva-repl window by using the "select-all + space" hack. But I can stil not perform any real deletion (delete, backspace, alt+delete, alt+bacspace)

Tomas Brejla22:07:12

is there any obvious place in calva's code which could prevent such delete event to be handled?

pez22:07:41

The mirror document is probably out of whack. Closing and re-opening the output window should make it start working again.

Tomas Brejla22:07:09

(and ctrl+s also does nothing)

Tomas Brejla22:07:27

yes, closing + reopening fixed the situation

pez22:07:47

We should add that “fix” to the documentation.

Tomas Brejla22:07:18

or even beter, we should try to find out the root cause for all this 😄, but I get your point

pez22:07:55

Indeed, hence “fix”. 😃

Tomas Brejla23:07:41

Situation 1 "all works" • start new calva in debug mode by making some change in calva code • open output.calva-repl file from previous session, delete the content • add 8k lines of plain text this is a weird bug • try to delete last few lines by selecting them and pressing Del - works fine • try to append 123 to last line.... works fine • try to delete that line ... works fine now different scenario. • start new calva in debug mode by making some change in calva code • open output.calva-repl file from previous session, delete the content • add 8k lines of plain text this is a weird bug • add 123 on the end of the last line (after the text) • sometimes deletion stoped working for me, but if nothing else, Enter key stops working ◦ even if I close and reopen the calva repl window, same problem remains

Tomas Brejla23:07:33

ha #2 I take that "enter stops working" back. It's just terribly slow. Enter performs evaluation. And that evaluation is extremly slow even after 2000 lines. After a lot of appending, that evaluation takes more than 30 seconds, so the editor appears stuck.

Tomas Brejla23:07:02

if it takes ~30 seconds for 2000 lines, I understand that it takes forever for 8k

pez23:07:43

I’m a bit slow here as well, because tired. Can you tell me the difference between scenario 1 and 2?

Tomas Brejla23:07:31

yeah, sorry, me too 🙂. I'll probably look more into this tomorrow afternoon/evening and possibly record a video and describe the reproduction in better way.

Tomas Brejla23:07:18

several times it seemed to me that syntax highlighting might be the culprit. As when the lines contained just the text which didn't get highlighted, I never got the "hangup" behavior. But now it seems that REPL needs to be connected as well to get into this "cannot delete/enter" state.

pez23:07:51

If the contents of the editor is mostly a lot of plain text, then there will be very long sequences of tokens on the same “level”, the top level probably, in your test there. That will be slow for the token cursor to navigate. So this is a reason why we might want to offer the option to send side effect output to some other place. b/c that kind of data is often unstructured.

Tomas Brejla23:07:21

ok, but still... 30s delay for 2000 lines?

Tomas Brejla23:07:32

I mean for 2000 very short lines of just a few words?

pez23:07:08

enter gets bound to something different when the REPL is connected (I think)

pez23:07:43

I don’t know what’s going on really. But I often have 10K+ lines in the output window without it getting that slow.

pez23:07:55

The bracket coloring tries to work on what’s visible in the view. But it needs the start and end of forms to do this. If it is all unstructured, I think maybe it works on the whole file.

pez23:07:56

^ super sloppy and tired theory ^ ¯\(ツ)

Tomas Brejla23:07:28

I think it's a good time to leave it for today. I'm just running around the issue and trying to guess the correct steps to reproduce it in that way that it would pinpoint the cause. But no luck there. If nothing else, it's really easy to make the repl window behave super-slow with just ~2-3k lines, definitely with 8k lines. When the REPL is NOT connected, any manual appending or deleting lines works instantly.. including syntax highlighting. I'll try to add some elapsed time logging tomorrow to the code that performs the "enter key" evaluations when the REPL is connected. Since it also starts to behave slow with increasing amount of lines in the repl window, maybe it will lead me to some slow place deeper in the code, who knows.

pez23:07:42

> When the REPL is NOT connected, any manual appending or deleting lines works instantly.. including syntax highlighting. This is quite a lot of progress on this issue!

Tomas Brejla01:07:32

state.extensionContext.subscriptions.push(vscode.window.onDidChangeTextEditorSelection(event => {
        let submitOnEnter = false;
        var before = new Date().getTime();

        if (event.textEditor) {
            const document = event.textEditor.document;
            if (isResultsDoc(document)) {
                const idx = document.offsetAt(event.selections[0].active);
                const mirrorDoc = docMirror.getDocument(document);
                const selectionCursor = mirrorDoc.getTokenCursor(idx);
                selectionCursor.forwardWhitespace();
                if (selectionCursor.atEnd()) {
                    const tlCursor = mirrorDoc.getTokenCursor(0);
                    const topLevelFormRange = tlCursor.rangeForDefun(idx);
                    submitOnEnter = topLevelFormRange &&
                        topLevelFormRange[0] !== topLevelFormRange[1] &&
                        idx >= topLevelFormRange[1];
                }
            }
            console.log("onDidChangeTextEditorSelection subscription took " + (new Date().getTime()-before))

        }
Maybe I found something. When appending a single line to a window with ~8000k lines, the code above ^^^ takes 1149ms out of the full 1186ms that is elapsed within applyEdit . Will investigate further tomorrow. It's 3AM facepalm 🙂

Tomas Brejla01:07:18

const topLevelFormRange = tlCursor.rangeForDefun(idx); this seems to be the slow one

Tomas Brejla01:07:13

yup, when I short-circuit it (put return [0,0] to that method's body), applyEdit takes 50ms instead of 1186ms. 💤 🛏️ 😴

bringe03:07:18

Great investigation!

pez06:07:53

That’s great. I can reproduce it with Select Current Form. I need some 5K lines, though, but anyway. The way rangeForDefun works is that it starts at the token-cursor position (0, in this case and most often) and moves forward sexp until it has passed the editor cursor position. In my test I have 60693 words in the window. Which translates to as many tokens. token-cursor.forwardSexp() has to do its work 61K times so it will take some time. Hmmm…

pez06:07:58

I wonder why selectionCursor.atEnd() is not enough… Can you recall anything around that, @U9A1RLFNV?

pez06:07:18

In this particular case we can find the range by starting at the end and go backward sexp instead. We could consider generalizing that and always start from the closest end, or change the way we find top level forms entirely. Like: • If upList(): ◦ Continue upList() until we can’t anymore ◦ note the current position as end of current top-level form ◦ go backwardSexp() ◦ note the current position as start of current top-level form • Else: ◦ Current form is the current top level form That should be quick in any document. Maybe I am missing something obvious though, because why didn’t I do it this way to begin with? 😃

Tomas Brejla06:07:31

awesome ! 🤞 There might be other smaller bottlenecks, but this one seems most pronounced with increasing amount of lines in repl window. After I disabled that slow function, I was able to append relatively quickly even to 200k-lines repl window. It choked for a while (10-20s) after I pasted all that content (pasting itself was quick), but then after those 10-20 seconds, additional succeeding append operations took around 200ms each, which is not too bad considering the amount of lines.

pez06:07:08

Up already? 😃 Those 10-20 seconds is probably spent in tokenizing the document. Something we potentially can speed up if we go for a deterministic regex implementation in the scanner. (Which probably is a pretty huge undertaking.)

Tomas Brejla07:07:53

Yeah, lately I find that about 6 hours of sleep is often enough for me. I mean I don't mind sleeping for longer period of time, really.. but even after less than 6, I no longer feel like zombie :male_zombie: 😄

pez07:07:05

I’m a zombie for sure. 😃 Didn’t catch fully 6 h, though. Was too wound up.

Tomas Brejla11:07:38

@U0ETXRFEW / @U9A1RLFNV so, you'll eventually try to improve the performance of that rangeForDefun as described above, right? It's very far from my level of knowledge (sexps, their processing, in-memory representations in calva etc), so it would probably take me ages to start understanding even some small bits and pieces around all that.

pez11:07:33

I was hoping you would do it. 😃 But that particular change is small enough for me to assign myself. It’s actually the most fun parts of Calva to work with. Do we have an issue where this fits, or should an issue be created?

pez11:07:27

I’m still a bit confused on why we need those extra checks to enable the submit-on-enter key bindings…

Tomas Brejla11:07:22

hmm, if you think it's easy, then perhaps I can later try to re-read those bullets you posted above and try to figure out what is exactly needed to be changed and how. regarding the issue: AFAIK there's this generic one, that's related: https://github.com/BetterThanTomorrow/calva/issues/942

pez11:07:15

We can update that issue then, I think.

pez11:07:21

I’d be happy to pair program a bit with you @U01LFP3LA6P and guide you a bit if you like to give this a try. I might be over simplifying it, but at least I can guarantee it is a fun task. 😃

pez11:07:50

At least while I haven’t thought it through fully, it looks to me that the change could be isolated to that one function. Which is implemented on a pretty high abstraction level so knowledge about the turtles below is probably not needed.

Tomas Brejla11:07:35

Haha, turtles bellow 🙂 🐢🐢. Ok. Not sure whether I'll have some time today or tomorrow, but if so, we can try that. In the meantime, can you please update that issue and write a few sentences there regarding what change you think should be needed in that function?

pez11:07:52

I can. And it is best, I think, if you give it some context with your findings. Deal?

👍 3
pez11:07:33

Something you can test manually in Calva, to get a feel for the idea to change I had tonight, is to have the cursor in some form. Then do Paredit Forward Up Sexp (`ctrl+alt+down`).until the command doesn’t move the cursor. That’s the end of the current top level form. Then do Paredit Backward Sexp. That’s the start. Two things complicate this a tad more: 1. The token-cursor primitive for upList() doesn’t move up unless the cursor is at the very end of the list (iirc), so the first upList() needs to be preceded by a forwardList(). This might make determining if the current form might actually be the current top-level form a bit tricky. (But maybe it doesn’t, we’ll find out.) 2. comment forms create a new top level. (But hopefully this will already be handled by the function and will just work.)

Tomas Brejla12:07:41

Hmmm, ctrl+alt+down doesn't get handled by vscode/calva in my setup. Prorobly my desktop environment stealing that shortcut somehow. It works via ctrl+alt+p though. Update: fixed.

bringe16:07:57

> I wonder why selectionCursor.atEnd() is not enough… Can you recall anything around that, @U9A1RLFNV? I do not recall anything around that, fyi. > Those 10-20 seconds is probably spent in tokenizing the document. Something we potentially can speed up if we go for a deterministic regex implementation in the scanner. (Which probably is a pretty huge undertaking.) Interesting..

bringe16:07:22

Is there not prior work in this area that can be utilized/borrowed? (re: regex implementation in the scanner)

pez16:07:14

There is probably prior work. I think all XML-parsers are using deterministic regex engines. But still, we have quite complex regexes as it is. Converting them to DFA … I think it will be tricky! 😃

bringe16:07:10

Ah.. DFA = ?

pez16:07:28

Also, 200k lines is quite a lot. We rather might throw in the towel there and tell the user that the document is not getting scanned. Like we do with long lines. (Like VS Code does with long lines too.)

👍 3
pez16:07:51

DFA is Deterministic Finit Automata (iirc).

👍 3
pez16:07:02

When I have been working with the scanner I have sometimes wished for it being a multiline scanner. Right now I can’t recall why I have wished that, but anyway. I think that would require a DFA approach since the matched text quickly gets quite huge.

pez16:07:17

It chokes on our regexes, though, but that might just be about regex flavour.

pez17:07:06

No, it is about: > Convert simple regular expressions to deterministic finite automaton But anyway, nice tool.

Tomas Brejla17:07:06

@U0ETXRFEW I tried to somehow hack the solution you suggested and it seems to generally work. When I originally intentionally crippled the rangeForDefun function, code evaluation using alt+enter stopped working. Which is expected, as it's being used to lookup the range to be executed, right? After implementing the strategy you suggested, I'm again able to evaluate using alt+enter both from my core.clj file, as well as from output.calva-repl. And it's FAST! So is appending to output.calva-repl . Even after 20k lines in repl editor, appending is still perfectly responsive (and thanks to previous batching effort, it's fast no matter if you're appending single line or 1000 of them). It will definitely need some more work, for example the comment must be handled somehow, as now alt-enter actually tries to execute the range including (comment) top-level form. And of course additional code cleanup and checking for regressions will be needed. But in general, it's starting to seem to work really nicely.

Tomas Brejla17:07:02

there'll definitely be some more bugs in that code I have right now. For example when I place cursor to any place in mapv, the threading macro gets evaluated, but I think (do should be returned as top level form and evaluated.

(do (->> (range 1000)
         (mapv #(println "printing line " %)))
    nil)

Tomas Brejla17:07:38

But these small things should hopefully be easily fixable

pez17:07:43

Awesome news! I don’t know how good the test coverage is for this, but generally it is important that the depth argument keeps working. 😃

Tomas Brejla17:07:18

yeah, I'll need to understand its place first 🙂 Will look into this later after dinner

pez17:07:34

Strange with he mapv example there, or exciting, or something 😃

pez17:07:41

Time to create dinner here as well.

Tomasz Pik07:07:21

Hi, is there a way/example of putting 'start calva' command (with project type) in project's tasks.json file?

pez07:07:51

I don’t think there is a way to do that. I could be wrong. What is the use case?

Tomasz Pik08:07:26

Simplify work :)

pez08:07:24

Then you might get away with using Custom Jack-in/Connect Sequences: https://calva.io/connect-sequences/

Daniel Slutsky17:07:49

Hi! Sorry that I'm asking something that has probably been asked in the past (couldn't find it though): If I understand correctly, VIM keybindings in vscode do not respect Paredit. For example, if I erase a paren with x in normal mode, then the corresponding other paren stays, and I get unbalanced parens. Did I get it right? 🙏

pez17:07:06

I haven't seen that particular question before. 😎 You'll need to bind x to the paredit delete command. There's a VSpaceCode config as well.

Daniel Slutsky17:07:50

Thank you so much!

gabo19:07:11

Hi 👋 I think that my vscode config (shared mutable state, amirite?) is messing with the jack-in command. Symptoms: 1. When I execute the jack-in command from a deps.edn file, I get the prompt to choose my repl type; when I choose Clojure CLI nothing happens and this message appears Running the contributed command: 'calva.jackIn' failed. 2. When I execute the jack-in command from a clojure file, nothing happens altogether I found this https://github.com/BetterThanTomorrow/calva/issues/1182 that looks similar so I tried to debug It like bpringe suggested

gabo19:07:58

I ran calva in debug mode and in the console I see for 1:

An error occurred while initializing project directory. TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type function ([Function (anonymous)])
when I executed the jack-in command, but I do get to the prompt to choose the project type then nothing happens (Something to note, I have an alias in the deps.edn file, and when I choose to use it it works!)

gabo19:07:36

And for 2 I see the same message:

An error occurred while initializing project directory. TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type function ([Function (anonymous)])
and the Running the contributed command: 'calva.jackIn' failed. prompt And nothing else happens

bringe04:07:43

If you do, and disabling it makes jack-in work again, then maybe we should look into what's going on there. If you don't have it installed, then if you can give us a reproducible project + vscode config, that would be helpful.

gabo13:07:08

Hi! I tried disabling and removing the python extension but I still couldn't jack in. (Removing my whole .vscode folder does indeed solved the issue 😛 ) Anyways, I'm going to try and get a minimal repro. Thanks!

bringe00:07:39

No problem and thank you