Fork me on GitHub
#unrepl
<
2017-03-13
>
cgrand09:03:39

Demo of the “futurize current eval” command (here mapped to ^Z) I pushed hours ago:

[:prompt {:cmd false, clojure.core/*ns* <#C4C63FWP5|unrepl>/ns user, clojure.core/*warn-on-reflection* nil}]
(do (Thread/sleep 10000) 42)
Oh! It’s taking too long!
^Z
[:eval <#C4C63FWP5|unrepl>/object [#unrepl.java/class java.util.concurrent.FutureTask "0x68f1b17f" "[email protected]"] 4]
The current evaluation was backgrounded and a future returned. Wait a little and deref it:
(deref *1)
[:eval 42 5]

thheller09:03:40

@cgrand cool, but just like the linux equiv I usually open another tab and let the other one run undisturbed

thheller09:03:09

since that way I will be "notified" when the command finishes, opposed to having to check for it

thheller09:03:00

serial execution and all 😉

cgrand09:03:31

yup I have a heated inner argument going on 🙂

cgrand09:03:18

Sending (future …) (or a/go or a/thread) (as a user) is always a possibility. However we don’t too that too often. It would be cumbersome to do that constantly. Here the intent is really “fix my blunder” (same spirit as : interrupt) not generic bg-eval.

thheller09:03:44

well that assumes the blunder can be fixed, not everything is interruptible

cgrand09:03:47

So re:sequential exec I believe that the REPL should not add concurrency of its own, all concurrency should happen at the lang level (or put in other words: requested by the user)

thheller09:03:51

to be honest even the interruptible eval is too much for me

cgrand09:03:14

re:interruptible in imy impl I use the dreaded Thread/stop which is more potent

cgrand09:03:59

I perfectly understand and that’s why no command is mandatory.

thheller09:03:03

and also deprecated 😉

cgrand09:03:56

it has been deprecated for a long time

thheller09:03:35

yeah given the stance of backwards compatibility for the JVM it will probably never be removed

thheller09:03:56

still I never use deprecated things for new code I write

thheller09:03:02

just feels wrong

thheller09:03:54

but back to the subject, I think interrupt is a good thing

thheller09:03:35

but letting the REPL interrupt itself implies that everything must be evaled in its own thread

thheller09:03:57

instead of using the thread of the loop itself

thheller09:03:33

if you open another connection and somehow could access the other loop (hint shadow.repl) you could interrupt via the other connection

cgrand09:03:35

so 2 threads must pretend to be 1

thheller09:03:57

granted that is a lot harder on the user than ^Z

thheller09:03:04

be the 2 threads might probably be required anyways

thheller09:03:46

otherwise you can't really interrupt something when the user disconnects from the REPL while something is running

thheller09:03:03

well you could let that just finish

thheller09:03:02

that propbably needs to check for EOF

thheller09:03:42

might not always want to interrupt on EOF though

cgrand09:03:50

From a spec point of view it means going from a spec that assumes the existence of the connection to a spec that talks about setting up connections. Your modus operandi for interruption would broadens the scope of the spec.

thheller09:03:52

sockets die all the time after all

thheller09:03:44

yes, everything shadow.repl related assumes that the server was started in a shadow.repl aware fashion

thheller09:03:58

it does not assume that there is a plain socket REPL we are going to upgrade

cgrand09:03:04

setting the desired behavior on disconnect would be a good candidate for a command

cgrand10:03:53

with a pushbackreader you can unread eof

cgrand10:03:20

Speaking of EOF when stdin is connected to a term and you hit ^D (not escaped, handled by the term) then .read returns -1 but subsequent reads work.

thheller12:03:59

>> It follows that some tooling needs (e.g. autocompletion) may be better serviced by a separate connection which may not be a REPL (but may have started as a REPL upgraded to something else).

thheller12:03:01

amen to that 🙂

thheller12:03:58

not convinced that starting from a REPL and upgrading is the best course of action

thheller12:03:06

but it certainly is one way to go

cgrand13:03:21

@thheller that’s why I used MAY and not SHOULD

cgrand14:03:45

Here is a more unusual command: :unrepl.jvm/enable-sideloader

cgrand14:03:08

[:unrepl/hello {:commands {:interrupt <#C4C63FWP5|unrepl>/raw \u0003, :exit <#C4C63FWP5|unrepl>/raw \u0004, :background-current-eval <#C4C63FWP5|unrepl>/raw \u001a, :set-source <#C4C63FWP5|unrepl>/raw [\u0010 <#C4C63FWP5|unrepl>/edn (set-file-line-col <#C4C63FWP5|unrepl>/param :unrepl/sourcename <#C4C63FWP5|unrepl>/param :unrepl/line <#C4C63FWP5|unrepl>/param :unrepl/column)], :unrepl.jvm/enable-sideloader <#C4C63FWP5|unrepl>/raw "\u0010(enable-sideloader)"}}]
[:prompt {:cmd false, clojure.core/*ns* <#C4C63FWP5|unrepl>/ns user, clojure.core/*warn-on-reflection* nil}]
` Ask for a class:
unknown.Clazz
[:exception #error {:cause "unknown.Clazz", :via [{:type <#C4C63FWP5|unrepl>.java/class clojure.lang.Compiler$CompilerException, :message "java.lang.ClassNotFoundException: unknown.Clazz, compiling:(unrepl-session:1:165)”,
[.....] 
 <#C4C63FWP5|unrepl>/... {:get <#C4C63FWP5|unrepl>/raw "\u0010(... G__228)"}]} 1]
[:prompt {:cmd false, clojure.core/*ns* <#C4C63FWP5|unrepl>/ns user, clojure.core/*warn-on-reflection* nil}]
Now, let’s send the :unrepl.jvm/enable-sideloader command:
^P(enable-sideloader)
[:command :sideloader-enabled 2]
[:prompt {:cmd false, clojure.core/*ns* <#C4C63FWP5|unrepl>/ns user, clojure.core/*warn-on-reflection* nil}]
And try again asking for the class
unknown.Clazz
[:unrepl.jvm/find-class “unknown.Clazz”]
after this message the REPL waits for either nil or for the bytecode as a base64 encoded string.
nil
[:exception ... 3]

thheller14:03:07

why does that need to be a command?

thheller14:03:03

why not just eval (unrepl.jvm/find-class 'unknown.Clazz)?

thheller14:03:38

but that seems like something tooling vs REPL. is that something the user would ever type by hand?

cgrand14:03:48

The example is a bit too dry: it’s something that happens during eval

cgrand14:03:06

if the compiler (or the code itself) tries to load a new class

thheller14:03:07

not sure what your goals are here?

cgrand14:03:50

connecting to a process and extending its classpath

cgrand14:03:24

(require ‘namespace.available.on.the.client.but.not.the.target) would work

cgrand14:03:21

even if not top-level but a transitive dep etc.

thheller14:03:32

ah missed that part .. the server is asking the client for a class that the server cannot find

cgrand14:03:21

yes, in the middle of compilation or execution

thheller14:03:31

hmm but how would that work if you run into something that actually requires a bunch of classes?

thheller14:03:58

fail the first one, request it, retry, fail the next one, request that, retry, ....

cgrand14:03:30

Well it would be transparent to the user.

thheller14:03:55

I don't know ... I'm not convinced that bootstrapping the entire thing over a REPL is a good idea in the first place

thheller14:03:12

now shipping basically an entire additional classpath over it

thheller14:03:19

seems like that is asking for trouble

thheller14:03:43

how would you handle version conflicts? the server has guava v15 but the client expects guava v21 or so

cgrand14:03:27

The classloader is local to the repl session

thheller14:03:22

I assume its goal is to let tools ship "extensions" without adding that extension to the classpath of the server runtime?

cgrand14:03:24

It could but I was rather thinking about user code.

thheller14:03:50

but the user code should be on the classpath of the runtime no?

cgrand14:03:46

On a remote JVM because you are trying to figure out something happening in one environment but not locally? "Open a ssh tunnel for the repl and code like it’s local" <- that’s the intent

thheller14:03:58

I don't get it

thheller14:03:22

the assumption is that you connected to a socket, whether thats local or remote makes no difference

thheller14:03:32

the server has a classpath

thheller14:03:52

but you want to load things there that don't exist

cgrand14:03:22

yes and I believe it’s more convenient to have them shipped on demand by the repl client than scp-ing them on the server class path (and remembering to clean of after)

thheller14:03:45

yes I see that might be useful

thheller14:03:25

but you could just send (unrepl.jvm/load-this-jar-please "cool.jar" <jar-contents>)?

cgrand14:03:04

and build a jar every time you modify the file?

thheller14:03:27

(unrepl.jvm/load-this-class-please ...)

cgrand14:03:58

and transitive deps?

thheller14:03:18

solved by the tool

thheller14:03:29

and sending multiple load requests in order

thheller14:03:25

trying to think of something I would want to load this way

thheller14:03:46

the tool could even just process ClassNotFoundException and try to add them

cgrand14:03:02

so you’d rather reimplement the deps resolution mechanism on the tool side (with the risk of getting it slightly wrong or even totally wrong if the user code relies on reflection, explicit calls to class loader) rather than rely on the real thing.

cgrand14:03:26

but then you would respawn the evaluation until no ClassNotFoundException is thrown

cgrand14:03:45

(and hope none gets trapped/wrapped)

thheller14:03:02

how would you do that differently?

thheller14:03:51

ah the classloader can block

cgrand14:03:30

It’s about 20 loc: 10 for a stub ClassLoader in Java (which delegates findClass to a fn) and 10 for the fn

thheller14:03:54

hmm yeah thats better than ClassNotFoundExceptions

cgrand14:03:55

'enable-sideloader
                     (fn []
                       (var-set clojure.lang.Compiler/LOADER
                         (unrepl.ClassLoader. (var-get clojure.lang.Compiler/LOADER)
                           (fn [name]
                             (when @unrepl
                               (write [:unrepl.jvm/find-class name])
                               (with-bindings {in-eval false}
                                 (when-some [base64-encodedclass (read)]
                                   (base64-decode base64-encodedclass)))))))
                       :sideloader-enabled)

thheller15:03:13

but how would you handle version conflicts?

thheller15:03:52

you can't just inject random classes and potentially pass those into user code

thheller15:03:59

the server runtime has MyClass.someFn(a,b,c) loaded but the code you injected uses MyClass.someFn(a)?

cgrand15:03:13

Then your code throws

cgrand15:03:48

It’s for troubleshooting/dev, not for bulletproof hotpatching of a production server

thheller15:03:06

I remain skeptical, seems to me like it would be way too easy to shoot yourself in the foot

thheller15:03:57

its a cool idea though

cgrand15:03:59

The good thing is that no command is core to the protocol

cgrand16:03:32

This sideloader implementation is most certainly a bad idea. • The inversion of control between server and client may happens while the client as partially sent a form so the state of the PushbackReader may be dirty, which means more logic to track that. • if the repl has been upgraded it doesn’t work anymore.

cgrand16:03:07

All saner implementations seem to imply a repl connection and a request/response connection.

thheller16:03:08

not only side-loading suffers from partial input

cgrand16:03:09

what else?

thheller16:03:46

(some-long-running-task) (something-else) then ^Z while first one is running

thheller16:03:09

doesn't background the first one

thheller16:03:42

at least from looking at it the (.read *in*) would still have (something-else) left to read before getting to the ^Z

cgrand16:03:37

You conflate ui and protocol

thheller16:03:15

do I? I meant that I entered (some-long-running-task) (something-else) and it went over the wire, then I realize that it takes a long time and want to bg it

thheller16:03:47

not assuming any UI for the moment

thheller17:03:12

since its a streaming protocol, the ^Z will not be processed until after (some-long-running-task) finishes?

thheller17:03:52

you only peek at the next char in the loop and unread it

cgrand17:03:12

The client should not send an incomplete form.

thheller17:03:42

it didn't, the user was just impatient

thheller17:03:11

ie. I frequently load-file in-ns

thheller17:03:45

so I have queued two things

thheller17:03:12

can't bg or interrupt the load-file anymore

thheller17:03:51

it sucks when load-file fails because in-ns still goes through, but every repl has that issue

cgrand17:03:52

How ok but should the client send the second form before having acknowledged processing of the previous one?

thheller17:03:03

well that is the question .. is it streaming or RPC?

cgrand22:03:43

While I disagree with the question, it challenges me in the right way. Thanks

cgrand17:03:13

Why queue on the server (in the connection buffer) rather than on the client? Honest question

thheller17:03:11

it implies that the client knows how to separate the text the user is typing into tokens

thheller17:03:20

it can probably do that yes given that it upgraded the REPL in the first place

kotarak08:03:00

thheller: I would not expect that.

cgrand17:03:23

It reminds of two commands I wanted to explore: :enable-echo (to know how the reader cut the input and matching eval result with a chunk of input)

cgrand17:03:52

The other command is :reset-reader

cgrand17:03:47

And reset would effectively happens anywhere.

dominicm19:03:48

Thinking of things like inf clojure and neoterm, which send the current file/selection to a terminal naively, I can see the potential for confusion.

cgrand19:03:39

@dominicm are EDN out messages even reasonably parseable in vim script?

dominicm20:03:14

@cgrand no, but with python, no problem. Vim has good calling out to python, ruby and lua. Neovim can work with anything. Go has a good story around it for example.

kotarak09:03:51

dominicm: maybe soon they are. Whether reasonably is a different question. I always cringe on python, because it always fails on my windows machine here at workm