@pez No, nothing special. We’ve been polishing the Elisp bencode code for quite a while and it’s pretty reliable at this point.

Btw, https://github.com/clojure-emacs/cider-nrepl/releases/tag/v0.18.0

@pez just a quick theory, if your network callbacks aren't complete, then perhaps you're receiving chunks? e.g. di6e followed by i3ee

I've been wondering about whether this can happen.

I mean, it definitely can, that's just TCP.

@bozhidar does elisp bencode stream?

I don’t remember at this point to be honest. Haven’t touched this code since the early days of the project.

I just remember it was pretty buggy, we fixed some bugs and forgot about it. 😄

python-nrepl couldn't handle negative numbers or empty lists for a very long time.

Only a clojure programmer would have made the mistake, empty lists are falsey in python 😄

I do remember that the Emacs bencode also didn’t handle negative numbers in the beginning.

I think it's a common mistake to overlook

https://github.com/clojure-emacs/cider/pull/616/files I see something here for handling incomplete streams. @pez

t seems that when the result is large it is chunked up, yes, but the chunks are not encoded correctly, by what I can see. That is probably my error, when adding stuff to the buffer. Hmmm… Thanks for that question, @dominicm!

I wrote a lot of C code as a teenager, I now have a bit of a sixth sense for network problems and race conditions. It's a little trigger happy, but I like it that way 😄

In fact I get decode errors for some non-large result values as well, but I haven’t looked at that yet, I do think empty lists and negative numbers are handled, though. 😃

@pez I found generative testing amazing for this. I can highly recommend pairing your encode/decode functions in a generative test.

It ironed out all of the problems in python-nrepl-client (from what I can tell, anyway)

I’ve never looked at generative testing. It seems to be time to open that box now.

(In case it isn’t obvious I am still very much a Clojure (and functional) noob. I can even admit to not having groked functional fully yet. Still waiting for that “click”… But that is why I am rewriting Calva in ClojureScript, eventually it has to happen!)

I had been waiting a very long time for a chance to use it 😄 It doesn't come up so often.

Haha!

I suspect that's a function of the way the codebases I work on are written too though.

Most of it is side effecting, and very little of it is data pushing & business logic.

I am trying to factor the Calva code so that ClojureScript is doing almost only data pushing and business logic, and then keep the side effecting out in the TypeScript glue into VS Code.

that makes sense, that should be amenable to generative testing then 🙂

I am pretty sure it is. But I haven’t even looked at spec yet, so I have some learning to do. Guess I should be happy there is still so little of Calva ported to CLJS yet. 😃

BTW, I heard that @bozhidar hates spec and generative testing. 😃

o rly?

spec and generative testing do pair, but test.check can be used without spec.

I must find a way to test this programmatically anyway, the manual way I am doing it now is driving me mad.

I've just been trying to wrap tests around pack (uberjarring command line tool I wrote). It's surprisingly hard. I wanted something that ran the CLI and tested that the jar behaved correctly. I've found so many bugs in shell-based harnesses. It's really hard to integration test non-http things. Vim does have some harnesses about, but I haven't tried writing tests for replant yet, in the fullness of time I suppose I should though 🙂

What does testing VSCode look like?

> BTW, I heard that @bozhidar hates spec and generative testing. 😃

Really? 😄

I don’t like spec in its current form, that’s true, but I don’t hate idea in general. Although I do feel that it goes against the nature of dynamic programming languages to a large extent.

I don’t have anything against gen testing, but in my experience the touted benefits are rarely strongly pronounced.

It's either a great fit or a terrible fit

https://lispcast.com/testing-stateful-and-concurrent-systems-using-test-check/ That is a lot of effort for just k/v, I suspect the complexity here is exponential, so that's a lot of work for most systems.

I do hope that smiley did its work when I put words in your mouth, @bozhidar ❤️

@dominicm I haven’t looked much at testing in VS Code. For Calva Formatter (a separate extension) I have unit tests on the CLJS side, but I can’t leverage them fully because I have stubbornly stuck to keeping my tests together with the functions they are testing and cljs,test do not discover them. I might give up un that, but have so far now and then tried to figure out how to get the testrunner to see them… Sometimes I wish I was less stubborn about things.

I found Eric Normand’s talk about making composable abstractions very interesting, even though most of it flew over my head. He is specing his way forward in the process: https://youtu.be/jJIUoaIvD20

Btw, @pez, I think that probably you and other people who can work directly with JS or CS in their editor would probably benefit a lot from a native JSON/EDN transport in nREPL.

That should be pretty simple to setup and would save you from bencode completely.

transit even?

It’s as simple as implementing a function like https://github.com/nrepl/nREPL/blob/master/src/clojure/nrepl/transport.clj#L84

Well, we can have a transport in whatever format we want.

Bencode was chosen for the default transport with the assumption it would be easy for many editors to support it.

My experience with JSON and ClojureScript is that it can be slow, but EDN makes total sense.

I think that for the small messages that are typically encoded probably any format will do. 🙂

But yeah - obviously the more efficient the format, the better.

Going from JSON to Transit sped up the initialization of SPA I am working with (for food) magnitudes.

And that was not huge amounts of data, even though larger than typical nREPL results.

Anyway, what would it entail? Updating nREPL?

The problem is that ideally nREPL should not have 3rd party deps, as they can conflict with something in the project of the people using it.

Makes sense.

But yeah, we just need a different transport function that uses say EDN and that’s it.

The encoding/decoding will be trivial then.

Want me to pack a PR?

Yeah, that’d be great is you’re willing to tackle this.

Seems like a Good First Issue ™️

:thumbsup:

It will also be an opportunity to extend http://nrepl.readthedocs.io/en/latest/design/#transports 😄

It’d be nice if we added some small tutorial on the subject.

Anything I need to know about how to hack on this project?

it would be quite neat if there was a cli-friendly way to inject a transport :thinking_face:

OTOH, using bencode everywhere is super beneficial as it means you can connect to any arbitrary nrepl server.

@pez Just a normal Clojure project. Nothing special about it. There’s a section on hacking in the manual, but it’s pretty basic.

@dominicm Well, if there were more bundled transports that wouldn’t really change anything. After all each transport defines it’s own protocol, so you can use all of them depending on your preference.

(I might not understand your reservation, though)

@bozhidar as long as all are loaded, sure. But you end up with people saying "Oh, emacs uses bencode, I'll start that" and http/edn never gets loaded.

As for the injection of transports - that shouldn’t be hard, but I think something like an json or edn transport should probably be built-in.

json without a dependency is very unlikely

I think the JDK has something built-in.

huh, so there is

javax.json

I just remember that had something built-in for XML, so probably they caught on the json front as well. 😄

> But you end up with people saying “Oh, emacs uses bencode, I’ll start that” and http/edn never gets loaded.

Well, if the EDN parser for Emacs matures that might change. Time will tell. 🙂

Sure, but then vim hangs out on bencode and it's still true 😉

Choice is okay, except when you want to try and have a uniform interface. Then you get things like content negotiation. At that point we should use http 😄

Yeah, of course.

There’s another reason why more transports (json in particular) have been on the back of my mind - which such a transport it’d be easy to expand nREPL to an LSP implementation. 🙂

(although you can obviously just proxy an LSP server to nREPL)

I'm waiting for the latter to happen before I consider the former.

Yeah, baby steps.

🙂

Overall, bencode has been surprisingly good. It's got limited expressivity which sucks (transit over bencode anyone?) but writing a bencode decoder/encoder is difficult primarily because of handling streaming in the language, which is quite fundamental anyway, you'd have to do the same with JSON. I'm quite a simple programmer, and bencode is well within grasp for me to write a parser/encoder for. Json, not so much.

No argument here. But there’s also the point it’s unlikely you’ll have to write a json parser yourself. 🙂 Anyways, Let’s bench the json idea for now and only focus on edn.

I'm just thinking out loud really. I wonder if an json streaming parser exists for most languages.

Hmmm, there is no write in clojure.edn, right?

My computer goes super hot and then becomes quite unresponsive working with the nREPL project. Seen something like that, @bozhidar?

> Hmmm, there is no write in clojure.edn, right?

I’ve never used it, so I can’t answer that one. 🙂

> My computer goes super hot and then becomes quite unresponsive working with the nREPL project. Seen something like that, @bozhidar?

First time I hear this. Did you notice anything in particular that triggered this? Sounds like some infinite loop or something, but who knows…