Fork me on GitHub
#unrepl
<
2017-03-15
>
cgrand06:03:02

@cfleming I agree that most people (me included until this thread two years ago) don't see the difference between a top level interpreter and a true lisp repl. Even c2 or Wikipedia don't mention that.

thheller06:03:35

@cfleming I think its a definition problem. If all you have is a terminal a REPL is exactly what you want. text in/text out. If you have a full blown IDE you can do way more like inspector windows, trace windows, dedicated windows for output

thheller06:03:10

does the tool use a REPL to accomplish that? I do not think so.

thheller06:03:52

the lisp tools all seem to embed the VM into the IDE process. Giving them full access to the runtime, even running the IDE in that runtime.

thheller06:03:14

I couldn't find any mention that they accomplish that over a remote REPL connection

thheller06:03:54

I doubt they do as I couldn't even find a remote connect option in LispWorks for example

thheller06:03:59

notice how the above window does not contain the word REPL?

thheller06:03:47

they call it a "Listener"

thheller06:03:10

so if we started from scratch without the assumption that we must use a REPL for everything

thheller06:03:05

or used better words instead

thheller06:03:51

I think lighttable came up with InstaREPL ... which isn't even remotely related to a REPL. InstaEVAL ok ... but not even close to a REPL.

thheller07:03:37

in the IntelliJ debugger you have an "Evaluate expression" that takes exactly one expression and displays the result

thheller07:03:16

I want that ... just don't call it a REPL and heck even bind *in* to nil no one can even begin to use it as such

thheller07:03:03

whether or not the whole tooling stuff then gets started over a REPL in the first place or not

thheller07:03:36

is debatable ... not sure to be honest.

thheller07:03:21

point is that you could ... that would be a lot harder with nREPL to the point of impossible

thheller07:03:42

> But, we now have nice editors/IDEs often running on the same machine, with the ability to use sockets. Multiplexing their needs with those of a human REPL consumer over the same connection is going to make things bad for one or both of them.

thheller07:03:22

Multiplexing their needs just don't do that ... open another connection and everything is good again

cgrand08:03:04

@cfleming so while users don’t want a “true repl”, there may be value in having a true repl. It’s sort of an exegesis work to understand what makes a true REPL according to Rich (I’d like to find other sources). So my current interpretation is that a REPL allows: • a simple sequential evaluation model • upgrading to anything (taking over in and out) • extending it (because the components of the repl are available you can put an extended version together and “upgrade” to it)

cgrand08:03:43

@thheller @cfleming with unrepl I’m trying to define what a m2m-friendly repl could be. I’m not trying to provide a solution to all tooling needs.

cgrand08:03:30

In the past days I think I’ve drifted.

kotarak08:03:36

@cgrand that was my understanding to begin with: machine friendly repl

kotarak08:03:04

Doing tooling things in a repl is just a pain

thheller08:03:05

I think the question is whether the unrepl model of upgrading/downgrading a REPL is simpler for tools than just connecting tools to a different port than the user would?

cgrand08:03:51

@thheller: it depends if you had the opportunity to install the tool first

kotarak08:03:01

It is. Bye can provide for example a hint that the tooling has to provide further stdin. Useful for vim.

kotarak08:03:23

There you just don't have streaming.

thheller08:03:41

the issue is: if you don't control *out* precisely you are going to get into trouble

cgrand08:03:43

@kotarak if you just want to upgrade to an autocompletion endpoint, I wouldn’t even bother to upgrade to unrepl first

kotarak08:03:05

I would neither.

thheller08:03:13

(future (Thread/sleep 10000) (println "foooooo")) (unrepl.repl/upgrade)

thheller08:03:18

that is going to mess up your protocol hard

kotarak08:03:23

But the user may issue (read in).

kotarak08:03:47

Unrepl helps tremendously to implement that in vim.

cgrand08:03:07

@thheller don’t upgrade a dirty repl

thheller08:03:31

well how do you tell that it is clean?

kotarak08:03:42

Background output is general a problem with multiple connected repls.

kotarak08:03:03

Nailgun used a threadlocalinheritable thingy.

cgrand08:03:08

@thheller because you started a new one?

kotarak08:03:33

But that doesn't help much with send, async etc

thheller08:03:37

@cgrand yeah but how do you get into the CLJS REPL

thheller08:03:30

well to be honest I sort of have given up on "upgrading" into a CLJS REPL

thheller08:03:50

probably better to have that sitting on a different port altogether

cgrand08:03:10

I believe the CLJS REPL could be refactored in an upgradeable REPL

thheller08:03:33

the problem is with the upgrading

cgrand08:03:55

but the current impl is not upgradeable

thheller08:03:58

is how do you ensure that nothing bad goes over *out*?

thheller08:03:18

that only works if you upgrade as soon as you connect

thheller08:03:16

which then begs the question .. wouldn't it be easier if it was upgraded before you connected?

thheller08:03:28

it could then still do the :bye downgrade

thheller08:03:36

but never an upgrade

thheller08:03:46

so the first message you get when connecting is a :unrepl/hello

cgrand08:03:12

If the first thing you do is (set! *out* (wrapper *out*)) (and stash the original out somewhere) you can later mute all spurious outs

thheller08:03:03

well .. the first thing you get when you connect to a REPL is user=>

thheller08:03:17

so we already have to "unread" that

thheller08:03:08

(using Socket REPL here, assuming thats what we are going to upgrade)

thheller08:03:36

if we already started the process in the first place

thheller08:03:49

we can just start unrepl at port 5000 and socket repl at port 5001

cgrand09:03:21

So yes you have to consume what’s printed, then you upgrade and part of the upgrade is muting spurious outs so once upgrade is done you have a clean connection.

thheller09:03:47

but even consuming the prompt is not as easy as it sounds

cgrand09:03:53

having unrepl started by the process on a different port. Connecting to a plain repl to start an unrepl server. etc. are all different ways of getting to unrepl with different tradeoffs

thheller09:03:49

ClojureScript Node.js REPL server listening on 58310
To quit, type: :cljs/quit
cljs.user=>

thheller09:03:00

is the welcome message for CLJS

thheller09:03:13

who is to say that another socket REPL doesn't also print something other than the prompt

thheller09:03:50

heck if I ssh into a server I get Last login: Mon Mar 13 13:46:10 2017 from 2a02:8109:10bf:c004:bc02...

thheller09:03:58

and then the prompt

thheller09:03:16

again .. that is useful to a human ... not so useful for a tool

thheller09:03:48

the :bye is very valuable cause the tool can downgrade

cgrand09:03:51

just swallow everything that is sent. Do some regexes on it if you want

thheller09:03:02

how do you know when you are done?

thheller09:03:18

reading from *in* blocks

cgrand09:03:28

(.ready *in*)

kotarak09:03:00

Tada. Upgraded.

thheller09:03:44

that works until something in your bootstrap code started a thread with a rogue print

kotarak09:03:59

Well. It doesn't.

kotarak09:03:09

It's my bootstrap code.

thheller09:03:25

and you are not going to use any library at all?

kotarak09:03:38

I try to mimize this.

kotarak09:03:43

Works so far.

kotarak09:03:00

I only need data.json. Got a local copy of that.

thheller09:03:00

I'm just saying that things go wrong ... exceptions get thrown ...

kotarak09:03:16

That's a point.

thheller09:03:35

if you don't create the *out* you can't ensure that only stuff you want gets there

thheller09:03:06

and to be blunt here ... the socket REPL code is like 50 LoC ...

thheller09:03:00

the rest is the command line parsing and stuff

kotarak09:03:20

What does bootstrap do? It defines functions. The only exception there is syntax. That can be fixed statically. The upgrade is one function call. It can handle exceptions there.

kotarak09:03:30

BTW: tooling must never interfere with the project.

kotarak09:03:11

Library dependencies must be localized to the tool. (for hardcore: even tool version)

thheller09:03:42

so we assume Socket REPL as a baseline?

thheller09:03:13

and that the runtime was started with start in mind

kotarak09:03:56

Having unrepl available would make the initial bootstrap and upgrade nicer. Also the repl as in "user interface".

kotarak09:03:13

But I would not require it.

thheller09:03:34

so ... what does the user run?

thheller09:03:50

or the tool for that matter

kotarak09:03:00

:VimpireConnect localhost 5432

cgrand09:03:50

:VimpireBite localhost:5432

kotarak09:03:11

Ah. Nameing. The hardest problem there is! First improvement on the way. 😎

kotarak09:03:24

Done. From there on he can doc lookup, macroexpand or start a repl

thheller09:03:51

no ... that assumes there is something running on 5432

thheller09:03:56

how do you get THAT?

kotarak09:03:07

It's a socket repl.

kotarak09:03:17

Obviously you need another side.

thheller09:03:28

so who started that side

thheller09:03:42

lein run -m unrepl.repl/start?

kotarak09:03:00

Vim is not about task handling. This is a job for other tools. Lein boot gradle make whatever

thheller09:03:34

point is .. the user needs to run something (and make sure that something also exposes the REPL port)

kotarak09:03:02

You could run it automatically if that connection fails, but i wouldn't. I prefer Proper control over started processes.

thheller09:03:04

so it boils down to someone called lein run -m unrepl.repl/start .. whether that was a tool or the user doesn't matter

kotarak09:03:13

Yeah. Well. You need an environment. One thing I learned: implicit, hidden fairy dust is the root of all evil

thheller09:03:15

heck it could be lein unrepl start

thheller09:03:42

point is that all this could start a unrepl at one port .. plain socket repl on another and maybe even a tool thing on another

thheller09:03:10

sorry but I don't see the entire bootstrapping over a REPL as easier

kotarak09:03:15

I just do a lein repl in a terminal. Problem solved. And second repl always handy if need bem

thheller09:03:30

it just leads to a bunch of "bootstrap failed because ... a message nobody understands"

kotarak09:03:16

It solves several problems. One having a split in the both sides which become disconnected.

kotarak09:03:29

In terms of API changes for the tooling.

kotarak09:03:29

Having a running socket repl is less trouble than having a lein plugin which requires additional libraries only at dev time which setup there own socket....

kotarak09:03:36

Much more moving parts.

thheller09:03:03

sorry I may be too biased to CLJS here

thheller09:03:10

but you are going to add dependencies NO MATTER WHAT

cgrand09:03:55

(try (bootstrap code) (catch Throwable t (println “I’m sorry Dave I can’t do that”)))

thheller09:03:17

@cgrand EXACTLY! How do you help the user if that fails?

kotarak09:03:28

For cljs that might be. For clj I survived 4 years without. And reactivating that 5 year old code now seems to show it still works....

kotarak09:03:56

The same way you do when lein unrepl start fails. With a helpful message.

cgrand09:03:20

well if it fails you tell the user it failed and report to the nice benevolent maintainer

thheller09:03:30

yes exactly .. the process is the same. the user is most likely going to edit their project.clj and restart the process

thheller09:03:49

bootstrap sounds cool in theory, in practice things go wrong. I'd much rather fail to start the process altogether

thheller09:03:46

instead of "hey my process works" but my REPL doesn't ... putting the burden on every tool out there

cgrand09:03:07

it’s no different than a connection failure or a protocol mismatch (who put SMTP on 22?)

kotarak09:03:24

Life is hard. The tool is a tool. It must put as low a burden on the user as possible. If the user has to care for the tools boilerplate something's wrong. In particular it must not be noticeable unless if asked something. That means no conflicts with the user project. Making life easy for toolmakers on the back of users is a no-go. My opinion.

kotarak09:03:19

That doesn't mean it does everything. As I said: it's not vim's job to run background processes.

thheller09:03:55

@kotarak its not my point to make vim do any of that. you are correct that it shouldn't.

thheller09:03:26

it is my impression however that bootstrapping everything you want to do with a tool over an unknowing REPL is a fantasy

kotarak09:03:47

Just trying to pointing out a line between user experience and fairy dust.

thheller09:03:51

look at how many deps that has

thheller09:03:15

are you going to do everything that provides without any deps in the first place?

kotarak09:03:03

The deps are not the problem. I got data.json as well.

kotarak09:03:21

It's just the less the better.

kotarak09:03:41

The problem with deps is that they have deps.

thheller09:03:46

its always one to the user

thheller09:03:16

I totally agree with you .. but the user is going to add one no matter what ...

kotarak09:03:30

Tool devs are easy people. They shouldn't be.

thheller09:03:38

ensuring that that one in turn has as little other deps as well is our job then

kotarak09:03:44

??? The user?

thheller09:03:57

data.json? or unrepl

kotarak09:03:34

I added that. Not the user. And it will end up in vimpire.clojure.data.json.

thheller09:03:03

I feel like we are talking past each other

thheller09:03:10

you are making my point for me here ...

kotarak09:03:25

But the tooling deps aren't that easy.

thheller09:03:26

you are already assuming to inject a dependency

kotarak09:03:47

Yes. I do. 😊

kotarak09:03:57

It will not conflict.

kotarak09:03:09

It doesn't break my bootstrap.

thheller09:03:38

> It will not conflict.

thheller09:03:00

is fairy dust ... are you going to write your code against Clojure 1.2?

thheller09:03:16

or do you want to use modern things? do you want to support clojure.spec maybe?

thheller09:03:29

so you are going to set a minimum bar for support

kotarak09:03:42

The code as it was written against 1.2.

kotarak09:03:58

But I chucked that out of the window.

kotarak09:03:10

Socket repl is 1.8 anyway.

kotarak09:03:28

Having a robust discovery mechanism is also the tool's responsibility.

kotarak09:03:03

You can upgrade to minimal backend.

kotarak09:03:09

1.2 level.

kotarak09:03:33

Then you can query the version und inject further code.

kotarak09:03:25

What's not available afterwards, eg. clojure.spec, is deactivated for this connection.

thheller09:03:55

so in my view the user will start a process, if that process succeeds and displays a "Ready" message everything works

thheller09:03:10

that is the non-bootstrap view

thheller09:03:56

for bootstrap the process will start and appear normal .. whether or not the REPL will actually work is TBD

kotarak09:03:19

@thheller Why? When the bootstrapping fails the user might as well get an error message.

kotarak09:03:52

Sorry. For my typos. Phone....

thheller09:03:45

@kotarak yes the user will get an error message. in my view when the process starts, for bootstrap when the REPL starts

thheller09:03:12

and IMHO it is better to fail as early as possible

thheller10:03:38

@cgrand the CLJS things comes back to the same argument about bootstrap

thheller10:03:47

do you assume it started out as an unrepl or not

thheller10:03:21

you are going to need some cooperation no matter what

thheller10:03:23

> put the cljs env in a dynamic variable

cgrand10:03:45

Making cljs repls upgradeable would benefit every repl tinkerers

thheller10:03:25

but how do you do that in the first place, given that you can't just start another REPL

cgrand10:03:40

The idea would be to have this refactoring in CLJS proper

thheller10:03:58

I will go ahead and call this very unlikely

cgrand10:03:18

Is there a precedent?

thheller10:03:57

doubt that anyone ever attempted that so this is just my arguably pessimistic instinct

thheller10:03:17

but I can implement what you described easily in my REPL

thheller10:03:26

but again I don't feel switching is the best way to go

thheller10:03:50

just another cond branch for :switchable.repl/clj

thheller10:03:52

but that would drop down to a normal clojure.main/repl again probably

thheller10:03:57

which then would need to be upgraded again

thheller10:03:01

all that makes my head spin

thheller10:03:00

(embed/start-worker :my-build)

thheller10:03:21

(embed/repl :my-build) which I can quit any time without stopping the CLJS build worker in the background

thheller10:03:58

but thats about process mangement again, sometimes I just want the build running with auto reloading

thheller10:03:05

and only occasionally connect to the REPL

thheller10:03:29

but that is already "switchable"

thheller10:03:11

it just doesn't speak unrepl

thheller10:03:23

I gotta go shopping for some food, bbl

cgrand17:03:56

Ok. @kotarak @thheller either it's the headache talking or you won. I'm considering ditching escaping in favor of a second connection dedicated to process control.

thheller17:03:40

go relax and come back without the headache, all this needs more hammock-time I feel

thheller17:03:08

the second connection isn't perfect either .. now that I implemented some of the language server protocol I already hate it

thheller17:03:11

and the escape thing is very useful if you are actually using a terminal

thheller17:03:09

opening a second connection manually is a lot harder than hitting ^C

cgrand17:03:31

I'd love to read more about what you hate with LSP

thheller17:03:46

well its too early to call it hate, its the versioning thing Rich talked about recently

thheller17:03:33

it version 3 now .. which made breaking changes to v2

thheller17:03:04

so I used v3 as a reference .. and couldn't figure out why things didn't work in vscode

thheller17:03:13

not realizing that that is still running v2

thheller17:03:04

their versioning/capability management/negotiation could be improved

thheller17:03:45

ClientCapabilities is a typed thing

thheller17:03:48

I think it would be better if the clients just sends a set of methods (their names as strings)

thheller17:03:56

and the server replies with a its set of methods

thheller17:03:21

so you get a textDocument/didSave-v1 textDocument/didSave-v2

thheller17:03:02

instead of changing DidSaveTextDocumentParams

thheller17:03:22

(though they didn't change this example)

thheller17:03:06

but if I treat this as alpha for now its all good

thheller17:03:56

but I started out with LSP4J which is a java implementation of the protocol

thheller17:03:18

and ditched that very fast 😛

thheller17:03:07

just painful to work with from clojure since it expected annotations on functions

thheller17:03:23

but I think I can now use nREPL as a transport protocol as well

thheller17:03:54

in theory, nREPL doesn't seem to have a standard to differentiate requests from notifications

thheller17:03:41

its all hidden away in the session management stuff

thheller18:03:59

but now onto my first goal on textDocument/didSave process the file and report diagnostics (warnings, errors) back to the editor

thheller18:03:19

btw this is what got me into all this

thheller18:03:20

but the problem isn't actually in the build process ... its that no tool expects this information

thheller18:03:48

(except vscode via the language server)

thheller18:03:19

so I began looking at why this is so hard .. and eventually got the REPL

thheller18:03:29

oh and I should mention .. I can now fully embed this server into a REPL. all it needs is *in* and *out*

thheller18:03:15

what I haven't figured out is how the client would know when the protocol starts, given the user=> prompt issue I talked about earlier

thheller18:03:25

but it can be done

cgrand19:03:36

Re: escaping. The problem with framing/escaping etc. Is that at some point you have a buffer or a queue. And forms waiting to be evaluated may block interruption for example. You can push the boundaries and create separate queues etc. But it gets complex fast.

cgrand19:03:37

Right now I'm thinking of advertising in :hello a form to eval in another repl to turn it into a remote control.

thheller19:03:05

another issue comes with actually "reading" ahead when it comes to aliasing

thheller19:03:22

(in-ns 'foo) ::foo

thheller19:03:44

if you read that before eval'ing you get :user/foo instead :foo/foo

cgrand19:03:58

Yeah that's why I would need to .read ahead and just scan for the esc or sequence. I've come up with `"\uE5CA\n[)\"[)" as an escape sequence with low probability of buffering and not being valid input.

thheller19:03:56

wouldnt't \0 work?

thheller19:03:37

since this is text it will never contain that otherwise

cgrand19:03:39

Sadly no. \0 is valid in strings. And it has been reiterated in EDN. No general escape mechanism.

cgrand19:03:15

Copy pasting can bring a lot of surprising characters. Even when copying from the output.

thheller20:03:44

only problem is I don't want to use vscode @cfleming

cfleming22:03:17

@thheller I don’t blame you 🙂

cfleming22:03:17

So I’m having trouble following all of this, I come in each morning to a wall of text 🙂

cfleming22:03:42

And I’m not down in the weeds enough at the moment since I’m busy on other things. I’m hoping to start on REPL stuff next week.

thheller22:03:47

yeah, I sorta hijacked the discussion with my own agenda (sorry about that)

thheller22:03:23

the above screenshot is just there to demonstrate what is possible with a independent language server connection

thheller22:03:38

whether that is started over the REPL or not is up for debate

thheller22:03:15

but I think we agree that this connection connection should no longer be treated as a REPL if it was upgraded

cfleming22:03:21

@thheller Yes, I’ll be adding that soon.

thheller22:03:24

as that would probably be the baseline for anything else we discussed here too

thheller22:03:59

so far jetbrains seems kinda silent about the language server thing

cfleming22:03:46

They use it for TypeScript

cfleming22:03:53

And I think Racer uses it.

cfleming22:03:22

Rather, you can use it for TypeScript if you choose to (which everyone does these days, I think).

thheller22:03:47

ah cool, no idea how easy it would be to add stuff into that for you though

thheller22:03:10

my plan currently would be to just expose some more features

thheller22:03:13

cljs/list-builds -> [{:id :website :type :browser ...}]

thheller22:03:41

cljs/autobuild {:id :website} -> :ok

thheller22:03:45

and most importantly

thheller22:03:11

cljs/repl {:id :website} -> {:host "foo" :port 123}

thheller22:03:59

all of this is the language server of course, but the port you would connect to there is the Socket REPL

thheller22:03:24

we just need to find a good compromise protocol for that

thheller22:03:21

alternative would be to cljs/eval {:code "(prn :foo)"} -> {:result "nil" :out ":foo\n"}

thheller22:03:36

with an equiv to clj/eval

cfleming22:03:18

Ok, like I say, I don’t have enough room in my head for all this right now. I’m planning to look at your shadow stuff next week when I start on all this.

thheller22:03:32

don't look at my shadow stuff

thheller22:03:28

I follow the language server protocol for now but I could make all that work over nREPL as well

thheller22:03:09

so if you have nREPL infra already (and most tools do) it might be better for us to stick to that

thheller22:03:16

instead of really starting from scratch

thheller22:03:33

however some nREPL parts would need to be ironed out

cfleming22:03:35

The problem with nREPL is that you can’t access the protocol from user code, you have to use middleware.

thheller22:03:45

yeah exactly 😛

cfleming22:03:51

No middleware for me.

cfleming22:03:04

It relies on me starting the server process, which I can’t always

thheller22:03:06

however again: if you assume the language server model the nrepl protocol is flexible enough to use that

thheller22:03:09

I just wouldn't use any of the actual nREPL stuff, just the protocol

thheller22:03:39

so no middleware, but yes the server must be started somehow

cfleming22:03:13

My problem is that I don’t want to create two parallel tooling implementations, one over nREPL and one that is not.

thheller22:03:15

how much of nREPL are you using in Cursive? you don't use that for stuff like autocomplete right?

cfleming22:03:48

In the REPL editor I do autocomplete using the REPL connection, but it’s not using nREPL middleware, just eval

thheller22:03:49

yeah so if the server spoke the nREPL protocol that would work

cfleming22:03:16

Right. Currently I use a simple REPL abstraction that provides eval and load-file

thheller22:03:36

as nrepl :ops

cfleming22:03:05

But again, then you’re tied to nREPL, and it won’t work over a socket REPL.

thheller22:03:31

nono you are mixing again

thheller22:03:53

you are using the nREPL behind the scenes for everything cursive needs

thheller22:03:07

but the actual REPL the user is using is just the Socket REPL

thheller22:03:15

different connection

thheller22:03:33

I'm just saying to re-use the nREPL protocol for the tool stuff since you already have that (as does every other tool)

thheller22:03:12

JSON-RPC isn't better/worse than nREPL+bencode

cfleming22:03:01

Ok, like I say, I’m going to have to deal with this next week.

cfleming22:03:10

I don’t have the brain space for it right now.

thheller22:03:36

let me know if I can assist you in any way

cfleming22:03:59

Will do, thanks.

thheller22:03:25

already found so many things I will need to fix at the CLJS compiler level

thheller22:03:18

{:warning :invalid-arithmetic, :line 7, :column 1, :msg "cljs.core/+, all arguments must be numbers, got [string number] instead.", :extra {:js-op cljs.core/+, :types [string number]}} can't properly underline code based on that info

thheller22:03:20

needs more data 😛

thheller22:03:07

but the concept of the language server and editor talking to each other has me excited

thheller22:03:36

so many options 🙂