Winding back versions, the last version this worked in is v2.0.394

So something in v2.0.395 broke this.

That would be the changes to make it squint compatible, apparently.

Interesting. What’s the change log saying?

Ah.

See if the nrepl message log has anything interesting.

Where do I find that?

Found it -- I had to toggle nREPL messages on...

Here's part of that log:

1701153971073 <- received 83ms
{
  id: '4',
  'nrepl.middleware.print/error': "Couldn't resolve var cider.nrepl.pprint/pprint",
  session: '416d8be5-90c8-465f-b241-18eac567972c',
  status: [ 'nrepl.middleware.print/error' ]
}

1701153971138 ->| not sent! (not supported by the server) 
{
  op: 'debug-instrumented-defs',
  id: '5',
  session: '416d8be5-90c8-465f-b241-18eac567972c'
}

1701153971196 <- received 123ms
{
  id: '4',
  ns: 'user',
  session: '416d8be5-90c8-465f-b241-18eac567972c',
  value: 'nil'
}

1701153971196 <- received 0ms
{
  id: '4',
  session: '416d8be5-90c8-465f-b241-18eac567972c',
  status: [ 'done' ]
}

1701153971197 ->| not sent! (not supported by the server) 
{
  op: 'clojuredocs-refresh-cache',
  id: '6',
  session: '416d8be5-90c8-465f-b241-18eac567972c'
}

1701153971199 ->| not sent! (not supported by the server) 
{
  op: 'classpath',
  id: '7',
  session: '416d8be5-90c8-465f-b241-18eac567972c'
}

1701153971201 ->| not sent! (not supported by the server) 
{
  op: 'init-debugger',
  id: '8',
  session: '416d8be5-90c8-465f-b241-18eac567972c'
}

1701153971252 ->| not sent! (not supported by the server) 
{
  op: 'debug-instrumented-defs',
  id: '9',
  session: '416d8be5-90c8-465f-b241-18eac567972c'
}

1701153972206 -> sent 
{
  id: '10',
  op: 'eval',
  ns: undefined,
  session: '416d8be5-90c8-465f-b241-18eac567972c',
  code: '(spit ".portal/vs-code.edn" {:host "localhost", :port 37537})',
  stdout: [Function (anonymous)],
  stderr: [Function (anonymous)],
  pprint: false
}

And it doesn't write that file -- nothing comes back.

(and, no, the remote server does not have CIDER available by design)

That's with v2.0.395

When I go back to v2.0.394, the log output is the same except that last command returns (as expected):

1701154306771 -> sent 
{
  id: '10',
  op: 'eval',
  session: '9e986203-3d88-4cd2-809e-fe1842c7fc38',
  code: '(spit ".portal/vs-code.edn" {:host "localhost", :port 45401})',
  stdout: [Function (anonymous)],
  stderr: [Function (anonymous)],
  pprint: false
}

1701154306857 <- received 86ms
{
  id: '10',
  ns: 'user',
  session: '9e986203-3d88-4cd2-809e-fe1842c7fc38',
  value: 'nil'
}

1701154306942 <- received 85ms
{
  id: '10',
  session: '9e986203-3d88-4cd2-809e-fe1842c7fc38',
  status: [ 'done' ]
}

Hmm, the difference is not sending ns: undefined perhaps? 394 does not send that, 395 does.

(it's 11 pm here so I'm off to bed -- hopefully the above is helpful and we can pick this up again in my morning!)

Thanks for the investigation! Looks like you found it. I’ll hopefully have a VSIX to test by your morning.

Is the spitting of that file part of some connect sequence configuration, @U04V70XH6?

I don’t know how to reproduce this so I will chill a bit until and try understand what’s going on together with you. Issue welcome of course. There is clearly a bug uncovered here.

Yes, spitting that file is what allows Portal (on the remote server) to work with the extension inside VS Code on my local machine. But that part is not relevant: if I #_ it out, I get the same problem: I cannot eval into the remote server. The difference is that ns: undefined which seems to be something introduced in v2.0.395 with the squint stuff?

Since ns: is normally either omitted or is a valid string, I'm fairly sure that's the problem.

It gets important when I try to figure out why it’s undefined. The old behavior of not passing ns was a mistake, so we can’t go there.

I won't be in a position to test with more recent versions of Calva until I'm done with work really so I won't be able to dig into it today I expect (going out later).

Thanks for heads-up. Which version of nrepl are you using in the project?

I've tried both 1.0.0 and 1.1.0. The former is what Calva injects. I've tried both on the server-side. lein connect locally works with both on the server.

Thanks. I can rule out “ancient nrepl” as a possible cause, then. 😃

Heh, you should know me by now -- I'm the one usually on the bleeding edge of releases! 😆

My dev tools are nearly all {:mvn/version "RELEASE"} 🙂

Re: ns: undefined -- couldn't you just send ns: 'user' in that case? (for any context where Calva can't figure out ns)

I plan to do that. But I also would like to understand why it gets to be undefined. Because that ’s closer to the root cause.

(later, if I get time at work, I'll do the nREPL logging thing again without the eval from JoyRide and just a simple ctrl+enter from the editor and see what it sends, since that also seems to hang)

Could it be something Calva is relying on CIDER for in the initial handshaking? We deliberately don't have CIDER in our remote server processes -- and we've never needed it for basic execution before.

Eval from Joyride? I have missed that part of this.

That was the spit part you asked me about (which I commented out and got the same behavior when trying to just eval a form)

I can try without cider here, but no, it shouldn’t rely on that.

I thought it was part of a connect sequence (as in calva.replConnectSequences). Are you using Calva API to evaluate things with Joyride?

I can’t reproduce with non-cider repl either. I do wonder what’s going on…

It only happens with the REPL server where my Joyride script is like this:

(ns remote-repl
  (:require ["$v0" :as calva]
            ["fs" :as fs]
            ["path" :as path]
            ["vscode" :as vscode]
            [clojure.edn :as edn]
            [promesa.core :as p]))

(defn- start-tunnel [nrepl-port portal-port extension-port label remote-server]
  (let [terminal (vscode/window.createTerminal #js {:isTransient true
                                                    :name label
                                                    :message (str label " Remote REPL...")})]
    (.show terminal)
    (.sendText terminal (str "ssh -C -N"
                             " -L " nrepl-port ":localhost:" nrepl-port
                             " -L " portal-port ":localhost:" portal-port
                             " -R " extension-port ":localhost:" extension-port
                             " " remote-server))))

(defn- connect-repl [nrepl-port]
  (vscode/commands.executeCommand "calva.disconnect")
  (vscode/commands.executeCommand "calva.connect"
                                  #js {:connectSequence "Generic"
                                       :port nrepl-port}))

(defn- portal-config []
  (->  vscode/workspace.workspaceFolders
       first
       .-uri
       .-fsPath
       (path/join ".portal/vs-code.edn")
       (fs/readFileSync #js {:encoding "utf8"})
       edn/read-string))

(defn repl-setup [nrepl-port portal-port label remote-server]
  (let [config (portal-config)]
    (start-tunnel nrepl-port portal-port (:port config) label remote-server)
    (p/do
      (p/delay 3000)
      (connect-repl nrepl-port)
      (p/delay 1000)
      (calva/repl.evaluateCode "clj" (pr-str (list 'spit ".portal/vs-code.edn" config))))))

If I comment out the evaluateCode call, I can't eval from the editor (it hangs). If I run the code as above with evaluateCode, that call hangs.

Later I'll try to get the nREPL message sequence from v2.0.395 with evaluateCode commented out and a simple ctrl+enter eval from the editor of an ns form at the top of a flie.

Unrelated question. Aren’t calva.disconnect and calva.connect returning promises? If not, we should fix that.

No idea. If they return promises, I'll need to update my code to wait on the results, right?

I mean, you could. Is usually better than delaying for some arbitrary time.

OK, I've found some time to do more exploration... It does appear that it's only the evaluateCode in Joyride that ends up with ns: undefined. I forgot that changing a Joyride script needs a reload to pick up the change (yes?) so my earlier tests were not quite testing what I thought.

And I can't put (ns user) in the evaluateCode call because that still sends ns: undefined...

If I comment out that evaluateCode call in my Joyride script (and reload) then I can eval basic stuff into the remote nREPL server as expected (but I can't start Portal because the config file hasn't been written out on the remote server).

So... I suspect you are only getting ns: undefined in the case where there is literally no applicable namespace to send because the code is not part of a file -- it's purely in Joyride. So defaulting to ns: 'user' in that case would be a reasonable thing to do.

So at this point, I'll update to v2.0.400 but keep that Portal config spit commented out until you have a solution. I can survive without remote Portal for a while (barely! 🙂 )

> I forgot that changing a Joyride script needs a reload to pick up the change (yes?) The script needs to be re-evaluated. So it depends on the code what it takes to do that. :reload in some require might do it and then re-running the some script? Reloading the VS Code window is sort of a nuke option.

Can you try sending along {:ns "user"} as options to the evaluation? Something like so:

(calva/repl.evaluateCode "clj" 
                         (pr-str (list 'spit ".portal/vs-code.edn" config)) 
                         {} 
                         #js {:ns "user"})

Yes, that solves the problem -- thank you!

One issue I ran into even with that -- and it doesn't seem entirely consistent -- is a lot of Couldn't resolve var cider.nrepl.pprint/pprint errors from nREPL and the inline results not showing. So I think you have a deeper dependency on CIDER than you used to have and I'd really like to see that adjusted so plain nREPL servers are sufficient to print results.

There’s no change there really. We’ve always depended on cider-nrepl. You might get away with disabling pretty printing.

Ah, OK.

Thank you for the code fix. I need to re-read the docs for evaluateCode to see what those four arguments are all about 🙂

A reason you don’t recall reading about the options argument may be that it is not documented 😃 https://calva.io/api/#replevaluatecode I don’t know why it’s exposed, it’s rather internal. It came in handy now as a workaround, but I’ll have to figure what makes sense to expose and how.

It turns out this affects a bunch of my Joyride scripts... My javadoc script was failing too because nREPL/CIDER tried to call symbol on the value passed in as ns which wasn't valid there... so I've ended up adding {} #js {:ns "user"} to all my evaluateCode calls for now...

Did it also mess up any further evbaluations for those other script?

Not sure what you're asking?

(I have several scripts that all call evaluateCode once and none of them worked until I added {} #js {:ns "user"})

I also have some joyride scripts calling evaluateCode and started seeing this issue as well. I added the extra options and can verify it fixed it for me 👍

@U04V70XH6 in the original problem I got the impression that no evaluations worked?

Because that initial evaluateCode call seemed to bork subsequent manual evals.

Once evaluateCode was fixed, everything started working again.

Exactly. So my question is if that seems to be the case with your other scripts?

That using them borks the repl.

I hadn't tested the other scripts (beyond noticing that my javadoc script didn't work today). And I've been reloading Calva so often to fix those evaluateCode calls that I can't say for sure whether subsequent manual evals were borked.

I can’t test myself, since I can’t repro the borking.

But it doesn’t much matter. I need to fix it asap regardless.

My sense from workflows today is that although the evaluateCode calls did not work, subsequent manual evals did work, into a local nREPL + CIDER connection. But for a remote (over SSH tunnel) nREPL-only server, it did bork subsequent manual evals.

But I've also turned OFF pprint mode today, to avoid the errors from CIDER pprint not being available in the remote server... so maybe that factored into it too...

There's a lot of moving parts here!

I hadn't noticed that some of my Joyride scripts weren't working (in the 2? weeks since v2.0.395 came out) because I've been doing a lot of infra-related stuff rather than clj coding, and no remote server debugging (clearly).

Yes, many moving parts! For pprint, if disabling it avoids the errors, you could probably use the calva setting for the pretty printer. That’ll do the pretty printing client side.

But since you use portal maybe it doesn’t matter to you… I’m wondering if turning the pretty printer off might give you hangs if you evaluate infinite (or just very long) seqs.

I currently have:

"calva.prettyPrintingOptions": {
      "printEngine": "pprint",
      "enabled": false,
      "width": 120,
      "maxLength": 50
    }

printEngine => calva.

Oh, OK.

But calva or disabled would both have the infinite seq problem. FYI.

So it's now:

"calva.prettyPrintingOptions": {
      "printEngine": "calva",
      "enabled": true,
      "width": 120,
      "maxLength": 50
    }

iirc calva means using zprint client side, and knowing that there are very few ppprint options exposed by Calva 😃

The JSON schema should help you with which options are available and some info about them, btw.

Yeah, that was what I meant about "so many pprint options" -- a lot of different engines.

This will make my "evaluate ns form without moving the cursor" Joyride script a lot simpler. Thank you! Will try this out later today.

I finally got it back by cutting / pasting a piece of text with an ansi reset code in it.

is there a calva ansi color reset thinger ?

you could end the calva repl prompt string with an ansi reset code, likely

I don’t know what you mean with the window’s default color. Also there’s nothing in Calva dealing with ansi except that we strip it from the results/output. Maybe there’s something in VS Code that cares about ansi codes…

I can show you if you want

dm me

I forced it into a bad state. did a normal thing to reload my tests. all the output is red for everything. terrible state. no way to reset it unless you know this hack. I forced it back to normal and works fine. Calva can trivially avoid this from ever happening.

I don't know what originally put me into the red state. But it felt impossible to get out of it. I can show you how to repro it if you want.

Paired with @U0ETXRFEW on this and the solution wasn't quite like I thought. And I only got into this state once. So we decided nothing needs doing. And @U0ETXRFEW had a better way to get out which was entering a " because apparently the window was in a string state.

@U9A1RLFNV this was the syntax highlighter that rendered an unclosed string. I think it’s something we want to make sure whatever new output window implementation isn’t as vulnerable to. Also we could consider honoring ansi escape codes in the new implementation. Instead of just stripping it, like we do today.

Ha, @U05H8N9V0HZ, so that’s why we couldn’t evaluate ansi codes to the output window. Calva stripped it. Why I forgot that while we were trying is interesting, because I had just mentioned the stripping to you. 😃

ah

Thanks for looking into it with me.

> Also we could consider honoring ansi escape codes in the new implementation. I'm not super familiar with this, but how might it apply to a webview?

ansi escape codes is what is colorizing your terminal. It can also do things like move the cursor and render UI:s, e.g. with libraries such as curses: https://en.wikipedia.org/wiki/Curses_(programming_library) There are Clojure libraries that helps with this too: https://github.com/lambdaisland/ansi And I think I’ve seen a Babashka Pod for it as well. Some Clojure dev tools use this. E.g. Ultra https://github.com/venantius/ultra and so far we have handled it by simply stripping In the current output window it is impossible to support this. But in a Webview we can do whatever we want.