Fork me on GitHub
#unrepl
<
2017-03-14
>
thheller08:03:40

@cgrand I still have trouble wrapping my brain around the intended upgrade process with CLJS in mind

thheller08:03:11

once started the eval/`print` happens in the JS runtime

cgrand08:03:14

well eval is split in compile and run and only the run part happens in the JS runtime (self-hosted CLJS notwithstanding)

thheller08:03:40

but you need to override read (for interrupt etc) as well which happens on the JVM side

thheller08:03:46

starting another REPL while inside CLJS is kind of inlikely since it would start read on the JS side which doesn't exist

thheller08:03:03

Things can work if the CLJS REPL starts out with the protocol in mind, but upgrading it now as done is CLJ is really hard

cgrand08:03:32

@thheller upgrade process from or to a CLJS repl?

thheller08:03:03

upgrade to CLJS (assuming from CLJ)

thheller08:03:22

no sorry .. upgrading a running CLJS REPL

thheller08:03:47

the CLJ REPL was upgraded but returned to normal by calling (node-repl)

thheller08:03:55

only way I see so far would be to force the user to call (unrepl.cljs/repl the-actual-repl-fn) so it can inject the hooks it needs

thheller08:03:14

how that would transfer the necessary stuff for print/eval to the CLJS side I don't know though

thheller08:03:14

trouble is that there are many ways to get a CLJS repl

thheller08:03:44

figwheel, boot, shadow-devtools, cljs.repl.node, cljs.repl.nashorn, etc

thheller08:03:57

not just clojure.main/repl

cgrand08:03:57

The problem is the hybrid nature of CLJS.

cgrand09:03:37

If you set up a new loop the loop lives in the JS runtime, not on the JVM

thheller09:03:15

for extra kicks the JS runtime also doesn't have a blocking read from *in* or even an *in* when in a browser

thheller09:03:07

we could probably fake that somehow by forwarding the *in* of the JVM to JS but that just makes my head hurt

thheller09:03:07

the JS platform just has so many differences to the JVM that many things just can't be translated 1:1

cgrand09:03:47

It’s really the hybrid model and the current crop of repls giving access only to the CLJS side (no defmacro at the repl, right?)

thheller09:03:48

no defmacro no

thheller09:03:24

but you get that by just using one CLJ repl and one CLJS repl

thheller09:03:23

over one connection would require the user switch the intended target which sucks

cgrand09:03:35

atm repl* must be forked into unrepl* 😕

cgrand09:03:26

target switching sucks as a UI, less so at the connection level

cgrand09:03:05

and would allow takeover of the loop while in CLJ mode

cgrand09:03:53

short term I believe it’s easier to provide an unrepl replacement to repl/repl compatible with its backends.

cgrand09:03:15

About yesterday issues, they are not similar: • the classloading stuff suffered from concurrency between msgs sent by the server and the client • interruption/backgrounding is ok as long as the client knows when a form is complete.

cgrand09:03:45

The first case can’t be solved reasonably.

cgrand09:03:54

The second case can (thanks for questioning my asumptions @thheller).

thheller09:03:38

but if the client is assumed to wait for the result before sending more isn't that RPC?

thheller09:03:09

a reader for CLJ(S) is actually harder than it seems

thheller09:03:32

tagged literal and aliased namespaced keywords are especially complex

cgrand09:03:33

I don’t believe so (RPC) but I don’t want to play a definitions game, waiting is a meh solution

cgrand09:03:15

tagged literals are not complex as long as you preserve them, aliases, autoresolution, autogensyms, syntax quote, eval-read etc. are the pain

thheller09:03:47

but the reader doesn't do that

thheller09:03:22

I was referring to

thheller09:03:29

> as long as the client knows when a form is complete

thheller09:03:56

to know that it must have a basic reader

cgrand09:03:10

Hmmm that’s true we don’t need a full blown reader we just need to tell when a form is complete (and to cross finger to not disagree with the server – eg a new syntax in Clojure 1.10 )

thheller09:03:33

yeah I implemented a naive version of that a while ago

thheller09:03:03

only later to realize that I couldn't send #inst "2017" as it treated it as two forms

thheller09:03:40

granted it is not that hard to fix but still something every tool must be aware of

thheller09:03:50

might be harder to do in vim or emacs or whatever

kotarak09:03:56

Reading on the client is broken.

thheller09:03:29

@kotarak some clients will do it anyways though to provide syntax highlighting or auto-complete for the REPL input

thheller09:03:39

but I think it should be optional

kotarak09:03:15

That doesnt mean it's actually reading. There is no reading in Vim for syntax highlighting. There were bugs in lein because it didn't have necesaary custom readers available.

thheller09:03:13

yeah you are correct. thinking too much in terms of Cursive

cgrand09:03:10

So what I have in mind is to have one (1) escape sequence (preferrably just one char). From an impl point of view, at the Reader level (big R: java Reader, before the lisp reader) the escape sequence is detected once detected further characters are sent to an edn reader, once 1 edn form is read.

cgrand09:03:02

it means that the input (from the user) is still free form and commands can happen anywhere in the middle.

cgrand09:03:20

but no commands in commands

kotarak09:03:32

Didn't you say that eval runs un separate thread? Why isn't the command side-channeling possible then?

thheller09:03:54

I still think all this is much simpler in the shadow.repl model where you'd send the interrupt command over another connection

thheller09:03:37

generally opening a new connection is underrated, trying to make everything work over one connection is really hard

cgrand09:03:08

independent connections are simple, dependent connections a bit less. (I’m trying to draw a line in the sand)

kotarak09:03:32

I understand @theller like the user sends "(long-command) (next-command)". Eval of first runs. User sends stop/background command. Unrepl applies it to runnung command. Since the eval is on different thread, unrepl can still respond.

thheller09:03:52

@kotarak currently unrepl only peeks at the next char in the stream. which isn't the command so it won't handle the command

cgrand09:03:19

@kotarak, read may be altered by previous evals so (except if you accept unbound buffering) you can’t accept ^C anywhere

cgrand09:03:02

@thheller was right in his understanding of what I had in mind

kotarak09:03:35

Grmpf. Slack on the phone sucks.

cgrand09:03:53

(long-running)^C was ok (long-running)(+ 2 2)^C or (long-running)(+^C weren't

kotarak10:03:54

You might escape ^C.

kotarak10:03:09

Escape is the wrong word.

cgrand10:03:11

my mental model was “one form after the other” with ^C being a sort of form (clojure reader happily returns a symbol for ^Cbtw)

kotarak10:03:27

Capture? What the terminal does.

thheller10:03:41

@kotarak the issue is (long-running)|(+ 2 2)^C where | is the current stream position (after read and while in eval)

kotarak10:03:54

It will hijack the ^C and not send it as input.

cgrand10:03:55

but it means the next layer must buffer

kotarak10:03:43

Unix shells do it this way since ages. And they got mostly right.

kotarak10:03:03

Control is control.

kotarak10:03:09

Not normal input.

thheller10:03:35

yeah but the user can't define new control structures?

kotarak10:03:04

Well. That's history. With unrepl he can.

kotarak10:03:15

Ssh as wells as telnet have escapes.

kotarak10:03:45

Define that as ^P(my-fancy-control goes here)

kotarak10:03:40

You want to send ^P? Use ^V^P. Escape the escape....

thheller10:03:01

all of this seems to be very terminal oriented though

thheller10:03:14

isn't the intent to make this for tools?

kotarak10:03:30

This not about terminals. It's about flow controlm

cgrand10:03:38

@kotarak how does a term work when you have a process (computing not reading) and a stdin which is ....(1 million input chars)^C where does the megabyte go? (ok one megabyte is not big)

kotarak10:03:00

Dunno. But as far as I now are unix pipelines lazy. :)

kotarak10:03:31

The megabyte is either on the disk or the memory of the killed process.

kotarak10:03:49

Modulo buffering somewhere inbetween.

kotarak10:03:34

Same thing for 100GB if that's bigger.

kotarak10:03:20

I think the point is: there is stdin and the terminal which accepts the ^C from the user.

kotarak10:03:48

It doesn't go through normal stdin. If that's your question.

kotarak10:03:51

So yes. There is sort of a layer inbetween which screens the input.

cgrand10:03:24

and another which buffers the screened input

cgrand10:03:51

what happens when the buffer is full?

cgrand10:03:04

we don’t care and we OoM?

cgrand10:03:23

we care and we do X?

kotarak10:03:56

Like output the input can be messages. [:stdin data-batch] and [:command ...].

cgrand10:03:44

It’s something I wanted to avoid

cgrand10:03:02

and it doesn’t solve the issue

cgrand10:03:38

[:stdin big-batch][:command ..] then the command may be buffered

cgrand10:03:10

(if we have a buffer)

cgrand10:03:25

maybe unbound buffer is ok for a REPL even if it’s bad design in general

kotarak10:03:10

No. It's not. The big-batch is entirely up to the client. It goes to the stdin queue. The command can be executed immediately.

cgrand10:03:24

I mean: the user can run arbitrary code so preventing her/him from hogging memory is useless

kotarak10:03:39

The question is: are we even in unrepl here?

cgrand10:03:48

and the queue is a form of buffer

kotarak10:03:03

Reading in triggers a bye, doesn't it?

cgrand10:03:21

from eval? yes

kotarak10:03:23

The server may provide backpressure.

kotarak10:03:01

The talk about reading before eval?

kotarak10:03:12

So we talk about....

thheller10:03:32

read while in eval

cgrand10:03:35

if you backpressure further commands won’t be sent so the end result is the same : command is ignored (except that back pressure allows the client to inform the user

cgrand10:03:41

but a command may be stuck in a client-side buffer so the client doesn’t know it’s ignored.

kotarak10:03:46

Backpressure stdin. Not commands.

kotarak10:03:13

The client must handle stdin and commands differently.

cgrand10:03:14

there’s only one connection! you can’t backpressure half of it

thheller10:03:26

@cgrand I think you could read from *in* but expose another *in* to the REPL

kotarak10:03:33

You can if you use coordination.

thheller10:03:53

so unrepl reads constantly and looks for commands and pipes everything into the other *in*

cgrand10:03:58

@kotarak are you banding with @thheller to push for two connections?

kotarak10:03:25

@thheller is able to name what I tried to say.

kotarak10:03:45

Only one connection not twom

thheller10:03:02

PipedReader/`PipedWriter` from java should make this simple

thheller10:03:20

PipedReader is what the REPL reads from (wrapped into the proper type)

thheller10:03:17

while unrepl reads from the real *in* and writes everything except commands to the PipedWriter

cgrand10:03:04

Let’s put names on things: socket-in is all the client send and we want to route things either to regular-input-out or to commands-out, ok?

thheller10:03:52

unrepl reads from socket-in always and creates unrepl-in, binds that to *in* so the REPL will read from that

cgrand10:03:33

An expression is evaluating (trying to find a SHA256 collision ;-)), so no ones read from regular-input-in

thheller10:03:42

unrepl then pipes everything except command sequences to unrepl-in

thheller10:03:29

coordinating the threads will be a bit tough maybe

cgrand10:03:57

@theller but eval is busy so no reading occurs on unrepl-in, at some point (if no unbound buffer), the “routing” loop is going to block because writing to unrepl-in blocks

thheller10:03:12

eval is running in another thread

thheller10:03:42

yeah you need some kind of buffer

thheller10:03:55

otherwise the commands get stuck like you said

cgrand10:03:42

and if the buffer has a max capacity, commands may always be stuck at some point

cgrand10:03:14

ok let’s do an unbounded buffer, it’s a repl the user may find more way to OoM

thheller10:03:02

yeah it is unlikely the user will type megabytes of data before trying to interrupt something

thheller10:03:19

still seems very term-ish to me but that might be cool if you actually use a terminal and not a tool

thheller10:03:15

for a tool I'm fairly sure the second connection is easier, all you need to know is the ID of the connection you want to interrupt

kotarak11:03:09

The repl client is a tool.

kotarak11:03:26

Unrepl is not meant for telnrt.

cgrand11:03:28

framing the input makes blind upgrade vaguely more difficult

thheller11:03:35

@kotarak from a VIM perspective: the user wants to interrupt the REPL, so I assume you have a :repl-interrupt command for that, or just map ^C to that command

thheller11:03:00

whether that then actually just sends a ^C over the socket is impl detail

kotarak11:03:01

That would be my plan.

thheller11:03:16

you could just as well curl

thheller11:03:36

(my second connection)

thheller11:03:43

doesn't need to be http

thheller11:03:47

just using that as an example

kotarak11:03:09

I could. I'd rather reuse the existing connection, though. But in general: yes. It would work.

thheller11:03:51

specifically the textDocument/didSave things.

thheller11:03:58

so assuming that this connection is already running

thheller11:03:16

extending that to REPL related things is extremely simple

thheller11:03:42

if you don't have that connection you get the default REPL experience

thheller11:03:08

I want the the editor to tell me didSave so I can skip watching the filesystem in my CLJS build process

thheller11:03:16

and just let it tell me instead

thheller11:03:41

I can also use publishDiagnotistics to tell the editor about warnings/errors

thheller11:03:43

so I assume I have this tool connection anyways whether I'm running a REPL or not

thheller11:03:07

the REPL just becomes a different part of the system that just has to solve REPL things

thheller11:03:11

not other tool things

kotarak11:03:31

Nothing of what you describe is related to the repl in any way.

kotarak11:03:58

The repl is a repl. A thing for the user to interface with.

thheller11:03:02

I'm trying to explain why way of thinking (and why I'm pushing for the second connection)

thheller11:03:14

> the REPL just becomes a different part of the system that just has to solve REPL things

kotarak11:03:42

For tooling it's better to have your own server loop. I did that back then in VimClojure. It worked well.

kotarak11:03:13

The were omni completion, doc lookup, macroexpand, repl.

kotarak11:03:31

All just different server commands. The repl just one of them.

thheller11:03:38

NO IT ISNT! 😛

thheller11:03:44

the repl is not a command

thheller11:03:04

the REPL is a streaming process

thheller11:03:34

that is ongoing and may produce output at any time

kotarak11:03:36

That is not contradicting.

thheller11:03:10

I don't know about VimClojure .. but how well does it work for CLJS?

thheller11:03:35

does it detect that the user started (cljs.repl.node/repl) and provide complete/doc lookup for CLJS?

thheller11:03:59

assuming the user did that in the REPL and not through some other command you created in VimClojure?

kotarak11:03:04

Obviously there would be some :ConnectCljs command.

kotarak11:03:19

The repl is user interaction. No maguc fairy dust.

thheller11:03:43

there is no magic fairy dust about any of that

thheller11:03:33

As Rich said: > REPL stands for something - Read, Eval, Print, Loop. > It does not stand for - Eval RPC Server/Window.

thheller11:03:00

sorry don't mean to sound aggressive

kotarak11:03:04

For tooling I want to evaluate a certain set of commands with defined input. And get a certain output back.

thheller11:03:10

I have the impression that the REPL is the only thing you are allowed to use and must use it for everything

kotarak11:03:20

For user interaction I want a repl.

thheller11:03:22

yes I know .. I want that too

thheller11:03:26

yes exactly

thheller11:03:30

two different things

thheller11:03:48

doesn't have to use the same protocol, doesn't even need to be related in any way

kotarak11:03:04

I don't care whether it's RPC. It's a protocol. It's extensible. Fine with me.

kotarak11:03:39

I'm confused nowm

thheller11:03:02

well it would be good to have a standard everyone could agree on

thheller11:03:09

so we only need one server implementation

thheller11:03:17

and emacs or vim or cursive could all use that

kotarak11:03:30

Hahahabhahaha

kotarak11:03:37

That standard will never happen.

thheller11:03:55

It has already happened

thheller11:03:00

sort of ...

thheller11:03:56

disregard that the protocol does not cover a REPL. the point is that it shouldn't!

thheller11:03:06

the REPL is independent from that

thheller11:03:27

I also hate that its JSON-RPC

thheller11:03:42

it is very very similar to nREPL too

thheller11:03:53

if you just removed the REPL from nREPL

cgrand11:03:24

and bencode

thheller11:03:51

hehe, thats not better or worse than JSON-RPC

kotarak11:03:04

Yeah. Shane on me. I know.

kotarak11:03:22

That's why i stay out of these discussions. Usually.

thheller11:03:13

I don't have an issue with bencode or even JSON-RPC really they work for what they are supposed to be

thheller11:03:50

sure I would prefer EDN but it doesn't matter really. once you implement the protocol you never see it again

kotarak11:03:58

I'm still confused. I will just silently work on vimpire and see where I get. Unrepl could give vim a real repl with stdin reading. And without python.

cgrand11:03:38

What is left when one removes “RPC eval server” from a REPL? the “upgrade” behaviour?

kotarak11:03:31

Do you mean me?

cgrand11:03:52

you, me, everybody!

kotarak11:03:05

You get a repl?

kotarak11:03:33

Everything must be stringified into valid clojure code.

thheller11:03:59

I'm not too sure about that

cgrand11:03:23

I mean: what is in a REPL that is not covered by a RPC eval?

cgrand11:03:41

taking over in & out?

cgrand11:03:07

so the cljs repl is not a repl

cgrand11:03:13

but an eval server

kotarak11:03:17

I believe we talk from to different sides here.

kotarak11:03:29

The repl is the more powerful thing.

thheller11:03:31

no from the clients perspective it is still just talking in/out

kotarak11:03:49

For the rpc server is a short cut for common commands.

kotarak11:03:53

For tooling.

cgrand11:03:26

@thheller who were you replying to with your “no”?

thheller11:03:11

@cgrand so the cljs repl is not a repl

cgrand11:03:31

if the user can’t overtake it, it’s not a repl

thheller11:03:32

it is a REPL from the clients perspective as it just talks to in/out

kotarak11:03:05

{"op": "doc-lookup", "sym": "condp"} vs. Printf("(doc-lookup %s)", hopefully-proper-escaped-string)

thheller11:03:22

well its a REPL with limited eval but still a read-eval-print-loop

thheller11:03:37

@kotarak yes the issue is framing print

thheller11:03:09

but if you assume that tools use a proper framed protocol

thheller11:03:35

the REPL can print to its hearts content without messing with your tool

thheller11:03:01

we don't have to make the REPL framed as well to get this

kotarak11:03:31

I never said that.

thheller11:03:49

didn't say you did. but nREPL is structured that way for example

kotarak11:03:08

I just think a repl is poor counter point for a protocol that is not user interaction.

thheller11:03:21

@cgrand I do agree however that CLJS could get away with RPC eval just fine as it can never read anyways

thheller11:03:55

also CLJS might actually complete a read (JVM-side) but not be able to eval when no browser is connected to a browser REPL

thheller11:03:19

or another issue is too many connections, do you eval everywhere?

thheller11:03:36

also what happens if the browser is reloaded

thheller11:03:48

so many things you just don't need to worry about for a CLJ REPL

thheller11:03:46

but all the basics more or less still apply

cgrand12:03:20

re-re-re-re-reading Rich on REPLs >>> we should be precise about what we call REPLs. While framing and RPC orientation might make things easier for someone who just wants to implement an eval window, it makes the resulting service strictly less powerful than a REPL. Let's look at some of that power: 'read' is a protocol. It works on character streams, one of the most widely available transports. It uses delimiters (spaces, [], (), {} etc) to separate forms/messages. Readers implement this protocol and return data upon encountering complete forms. read is shipped with Clojure and multiple implementations are available. It is available for use in programs, including programs you invoke in a REPL. A substantial subset of evaluable+read-able forms are supported by edn, which has even wider support. The reader format is extensible. So, when you get read, you are getting a lot. When embedded in a REPL, even more is implied by read. There must be some source (stream) of things read. That source is the same source that should be used should the code being evaluated call 'read'. If that source is a terminal with a human, they can continue their interaction in place. 'eval' implies the full power of Clojure. Take a form as data, evaluate it, return a result as data. 'print' takes the returned data and prints it on an output stream as characters. It is true that not all things might print read-ably, which can pose challenges for programs sitting on the other end of a REPL that would like to call 'read' on its output. But we have improved Clojure in this area - now the default print-method prints something that is read-able (with an appropriate tag handler installed). Another area where many REPLs presume a human consumer and print un-read-ably is when exceptions occur. This socket REPL will default to printing exceptions read-ably, but you can switch to a human oriented formatter of your choice. Again, as with read, when embedded in a REPL, even more is implied by print. There must be a destination (stream) of things printed. That destination is the same destination that should be used should the code being evaluated call print. If that source is a terminal with a human, they can leave their reading focus in place and see all output targeted towards them. 'loop' - do it again. Note however that a REPL will not attempt to read/print while evaluating. The stream binding and nestability is important. It means that you can, in the code you evaluate in your REPL, launch a nested interpreter, or Eliza-like thing, or a text-adventure etc, and those things will get input from the same source and print to the same target. This is what someone who knows what REPLs are expects and demands. You can, if you desire, even launch an eval RPC server in a REPL, but not vice-versa. REPLs work over the most primitive connections, terminals, and can be pipe, rerouted, etc with all tools. They accomplish a lot over a single connection, which will often be all you can get. But, we now have nice editors/IDEs often running on the same machine, with the ability to use sockets. Multiplexing their needs with those of a human REPL consumer over the same connection is going to make things bad for one or both of them. With the socket REPL, the recommendation is that tools will open more than one connection, and use a dedicated channel for most editor/IDE operations that might want to interact with the running process. With the improved read-ability of REPL output, I think an edn-based REPL makes a fine RPC server for program-to-program communication, especially since the set of 'verbs' supported is bounded only by what you can evaluate, and the types of things you might discover is open. These are the intentions of this feature.

thheller12:03:57

> But, we now have nice editors/IDEs often running on the same machine, with the ability to use sockets. Multiplexing their needs with those of a human REPL consumer over the same connection is going to make things bad for one or both of them.

cgrand12:03:03

A takeaway is that a repl is also about providing its very own building blocks to the user. That’s where a CLJS repl fails. That’s where unrepl fails by not providing access to a function for tagged printing

thheller13:03:16

@cgrand I want that too though, I really like the automatic limiting of the output

thheller13:03:10

my approach so far is "cap the output at X size" and if it goes over that don't print the result but print "Eval successful but result value too large too print."

thheller13:03:33

to prevent blowing up a REPL by accidentally printing hundreds of megabytes

thheller13:03:48

after seeing your elision concept I prefer that

thheller13:03:54

but also sometimes the large values are completely uninteresting to me

thheller13:03:41

so from a tools perspective I would prefer if I could eval but don't print sometimes

thheller13:03:09

the REPL solves that in a sense already by (def x (produce-something-large)) and just printing #'user/x

thheller13:03:30

also a def is not always what I want

cgrand13:03:41

(do X nil)

thheller13:03:13

but that breaks *1

thheller13:03:43

I still might want the large value somewhere

thheller13:03:11

FWIW I have this issue in shadow-build where it would print the entire CLJS compiler-env after completing the build

thheller13:03:43

so all my fns that are invoked via the REPL looked like (defn foo [] (do-the-work) :done)

thheller13:03:19

but sometimes I forgot the :done .. so I had to restart my REPL

thheller13:03:49

unrepl would have solved that as well

cfleming23:03:31

FWIW I think Rich is wrong, or at least isn’t acknowledging the tradeoffs.

cfleming23:03:03

He argues that a REPL is only one strictly defined thing, and IMO most users don’t care.

cfleming23:03:18

I believe that most users think of their REPL as RPC.