Fork me on GitHub
#clojure-dev
<
2015-09-11
>
danielcompton01:09:35

@alexmiller: the API docs for core.async seem to correspond to an unreleased version of core.async?

alexmiller02:09:02

Yes, they do. We have multi release set up for core but its rarely an issue on the contrib projects so we just have master there

alexmiller02:09:03

Unfortunately async has gone way too long without a release which weighs on me heavily but I need some oks from rich to move it forward

danielcompton02:09:35

@alexmiller: are contrib docs rebuilt from master on each commit?

alexmiller05:09:04

Yeah, commit hook

ragge12:09:34

socker server merged, exciting!

alexmiller13:09:00

should be a release today afaik

alexmiller13:09:05

alpha release that is

alexmiller13:09:32

andrewhr: I think you've got a patch in there!

andrewhr13:09:45

yep, my first one simple_smile

alexmiller13:09:50

hopefully more to come :)

andrewhr13:09:00

yep, me too!

andrewhr13:09:11

talking about it... there is something specific that you need someone to work on? If you haven't, I will look for myself for some random thing

alexmiller13:09:42

hard to say right now. I really need Rich to pull some things from the triaged list

cfleming14:09:19

@alexmiller: Socket REPL looks very nice

alexmiller14:09:36

@cfleming: would love your feedback in particular

alexmiller14:09:54

not sure if you've been following the prior patches or the session stuff I had in the prior rev (-12)

cfleming14:09:05

Ok, I’m pretty flat out right now but will try to read through it all soon

cfleming14:09:13

No I haven’t followed the gory details

cfleming14:09:06

As I understand it, there’s currently no support for publishing STAR__ns__STAR__ to other threads, is that correct?

alexmiller14:09:15

not in this version

alexmiller14:09:27

the -12 version had the ability to "attach" from one client session to another and eval in it's env

alexmiller14:09:46

Rich asked to pull that part out until we had more time to talk about it

cfleming14:09:10

That’s something I’d really like to see, prompt parsing is totally ghetto

alexmiller14:09:23

I'm not sure the "attach" metaphor is something Stu/Rich are comfortable with but that's really mechanism

alexmiller14:09:45

the key thing is the ability to capture the environment in one client and to eval-in that environment in the other

cfleming14:09:57

Right, and of course I can wrap executed forms to pass more data back, too, as nREPL does.

alexmiller14:09:59

and that's just a few lines of code in the -12 patch

alexmiller14:09:58

so I guess that's the question - what level of support do you need for this stuff and what would be useful

alexmiller14:09:34

in the comments for [[http://dev.clojure.org/jira/browse/CLJ-1671 near the end is a rescued bit of description from how the session attach thing worked

cfleming14:09:46

I haven’t looked at the details yet - I’ll need to open the user session (where the actual interaction goes) and a background session for tooling commands.

cfleming14:09:01

The second one needs some way to know the current namespace of the first.

cfleming14:09:33

I’m not sure yet whether I would configure the JVM to have two different servers listening or if both sessions could be open to the same server.

alexmiller14:09:59

no reason for two

cfleming14:09:55

Well, depends if they need different behaviour for some reason, not sure yet. Does the current REPL implementation print prompts, for example?

alexmiller14:09:29

it does, although could be configurable

cfleming14:09:48

Right, but then I’d need two servers with different config, right?

alexmiller14:09:10

I mean, in principle, it doesn't do this now

alexmiller14:09:22

but there could easily be a repl command like :repl/noprompt

alexmiller14:09:16

or it could be a server property and you'd run two servers, but that's twice as many threads and sockets so seems like that shouldn't be necessary

cfleming14:09:37

Yeah, or :repl/dataonly or something

cfleming14:09:58

It seems like having a data-only connection is going to be a common use case

cfleming14:09:15

I actually have a use case for that right now, but can’t mandate the Clojure version unfortunately

alexmiller14:09:42

in data only mode - what do you do with err ?

cfleming14:09:53

I don’t know

alexmiller14:09:12

well that's been a long lingering question

cfleming14:09:12

Or out, for that matter

cfleming14:09:25

I mean, any printing at all will break things

alexmiller14:09:33

well out is going onto the socket, but yeah that's the real issue

cfleming14:09:17

The problem is, as soon as out goes over the socket then you can’t have a data-only connection based on reading forms

alexmiller14:09:32

so that's do-able though

cfleming14:09:42

Which is what I was arguing in the original ML thread but I think it got lost in the noise.

alexmiller14:09:45

don't pipe out to the socket and only return eval results

cfleming14:09:04

Right, the question remains what to do with that output though.

alexmiller14:09:28

other than drop it on the floor? :)

cfleming14:09:32

Unless you return an nREPL style {:out “output here” :err “errors here” :values [values, here]}

cfleming14:09:42

By capturing when you evaluate

cfleming14:09:54

Hehe, I guess that’s always an option

alexmiller14:09:02

yeah, interesting

cfleming14:09:38

I think having a mechanism to do that would be very useful, it’s relatively simple to implement but I think that’s what a lot of people are going to want for RPC style systems.

cfleming14:09:02

It becomes nREPL-lite at that point

alexmiller14:09:03

for the tooling repl, what is the scope of things you want to be able to do?

cfleming14:09:37

Off the top of my head: completion, macroexpansion, things like core.typed type checking

alexmiller14:09:51

eval in their env covers a lot

cfleming14:09:22

Yeah - in general, I need to be able to eval in their environment, ideally without disturbing it

cfleming14:09:51

i.e. ideally an eval in a clone of their env, that’s tricky obviously

cfleming14:09:24

I’m playing around with ShimDandy to do some of this stuff, which may mean that I actually don’t need tooling connections at all, or perhaps only for completion

cfleming14:09:51

I’m also interested in trying to move away from completion coming over the REPL connection, so that may go away at some point too.

cfleming14:09:08

Not sure how feasible that will be though

alexmiller14:09:32

so the way the session attachment stuff works is that every time the user session evals, I clone their env and save it off

alexmiller14:09:50

the tooling session can then push that env on the binding stack and eval in their env

cfleming14:09:02

env being local bindings, I assume?

alexmiller14:09:03

and I was also going to the trouble of capturing *1, *2, *3

alexmiller14:09:13

their full dynamic binding set

cfleming14:09:11

That would work for most things I think - off the top of my head, I can’t think of anything I need except *ns* and *e*

alexmiller14:09:12

the nuts-and-bolts of this is in http://dev.clojure.org/jira/secure/attachment/15046/clj-1671-12.patch if you search for repl-eval

cfleming14:09:31

Great, I’ll take a look, thanks.

cfleming14:09:57

(add “print last exception” to the list of things I need to do)

alexmiller14:09:00

and nicely if you set! a dynvar, you're only in your copy of the bindings so there is some isolation

alexmiller14:09:07

yeah, * e will work

cfleming14:09:58

Things like test running etc also happen in the REPL, but I’m unsure about whether I should open a hidden REPL session for that anyway.

alexmiller14:09:19

this is all based on (get-thread-bindings) in the session thread and push-thread-bindings / pop-thread-bindings in the tooling thread

cfleming14:09:24

Ok, I’ll take a look - I’d be interested to see how the details work.

alexmiller14:09:45

that's the core of it - the rest is all just a mechanism to allow that to happen

alexmiller14:09:30

Stu thinks maybe just the ability to "eval-in" a remote session's env would be better than the "attachment" metaphor but either are really just trappings around the binding stuff

cfleming14:09:54

I think either would be fine. I agree that eval-in seems cleaner.

alexmiller14:09:37

heck, maybe that even exists somewhere already in core :)

alexmiller14:09:42

it's essentially what binding does

alexmiller14:09:06

or with-bindings / with-bindings*

cfleming14:09:20

Well, you need a way to get a reference to the bindings of the other context, right?

alexmiller14:09:23

the key is really just having the ability to look up another session's bindings

cfleming14:09:27

I’m not sure how that would work

alexmiller14:09:59

right now, every repl session has a server+client identifier

alexmiller14:09:21

and there is a global map where you can look up data stashed per session

alexmiller14:09:21

that's in clojure.core.server/servers

alexmiller14:09:35

although it's access is intentionally somewhat protected

cfleming14:09:52

So in the user session/tooling session case, how would I get the ID of the user session from the tooling session?

alexmiller14:09:05

that is the crux of it :)

alexmiller14:09:25

I had in -12 a command :repl/sessions that gives you a vector of the ids for all sessions

cfleming14:09:41

the issue is identifying the one you want, I guess.

alexmiller14:09:47

and there is a dyn var clojure.core.server/session that is set in every client session

alexmiller14:09:14

so the user session can say who it is - but I wasn't sure if you have some non-intrusive way to ask that

alexmiller14:09:29

could also use the wrapping technique though

alexmiller14:09:14

stepping way back for a second...

cfleming14:09:25

In general, anything I do to interrogate the user session will be visible to the user.

alexmiller14:09:35

if you have the ability to run your own socket server with it's own client accept function

alexmiller15:09:26

you can define your own custom repl protocol or have it do whatever you want

alexmiller15:09:55

like you could insert code to always update a global var with the ns of the current repl

cfleming15:09:55

Right, which I can trivially do by attaching a jar to the classpath

cfleming15:09:28

Yeah, I thought about that. I can also do that from the client. It feels very dirty, though simple_smile

alexmiller15:09:36

just like I'm inserting code to steal the thread bindings

alexmiller15:09:00

but what if that was actually supported by the server socket repl as a function you'd set

cfleming15:09:20

i.e.:

(let [result (eval-the-user-code)]
  (set! my-atom *ns*)
  result)

cfleming15:09:59

I can wrap evaluations in a form like that from the client too, without needing to run a different server

alexmiller15:09:14

yeah, although the set! location could be just a generic function call the repl client function invokes that's provided as an option to the socket repl accept fn

alexmiller15:09:24

it's built-in that is

cfleming15:09:36

I do much more complicated things like that for macroexpansion right now. I prefer it to middleware like CIDER uses.

alexmiller15:09:01

you could also call get-thread-bindings and save that off if you desired

alexmiller15:09:13

then you could do anything you like

cfleming15:09:56

Right, that would probably be the best option, then create a var containing an atom named with a UUID representing the session to store it in.

alexmiller15:09:16

I like this as a direction - I don't even need the session gunk at that point

cfleming15:09:21

In general, I think I prefer this approach to building too much into the server.

alexmiller15:09:12

ok, well I'm going to at least work up a patch over alpha5 along these lines

cfleming15:09:26

I thought about doing that for the clojure.main REPL in Cursive, but opted for prompt parsing for reasons which escape me right now.

alexmiller15:09:49

well, you couldn't get back into the same server to get the data, right?

alexmiller15:09:10

it's having multiple connections that helps here

alexmiller15:09:23

and then the other useful thing we discussed is the data-only connection

alexmiller15:09:35

which I'd like to think about a bit more

alexmiller15:09:54

but could easily be repl config options or something

alexmiller15:09:04

and then you would start a 2nd server

alexmiller15:09:23

thanks for your time on this - this has been hugely helpful

cfleming15:09:23

I think a data-only RPC connection is going to be a common use case.

cfleming15:09:35

No problem, it’s looking very nice.

alexmiller15:09:52

how we deal with out/err is the hardest part of the data version

cfleming15:09:08

It might be interesting to think about a simple framing mechanism for that case, although I’m not sure if it’s required.

alexmiller15:09:09

and something Rich may have thoughts about

cfleming15:09:37

For example, imagine you have two executions, both of which return symbols

cfleming15:09:02

If you just read repeatedly, you’ll get the symbols concatenated together as a single reply without framing of some kind.

cfleming15:09:51

Same for two numbers etc. Repeated reading only really works for delimited things (strings, collections)

cfleming15:09:23

Ok, scratch my previous wondering - framing is definitely required simple_smile

alexmiller15:09:59

yeah it seems to me like the stream-oriented cases are primarily human-oriented right?

cfleming15:09:12

Well that depends on who you ask 😉

cfleming15:09:16

But I think so, yeah

cfleming15:09:24

Logging might be another use case

alexmiller15:09:27

I've only asked humans so far

alexmiller15:09:00

I need to go back and read those messages from Rich on the ml thread

alexmiller15:09:06

re-read that is

cfleming15:09:20

You’ll need a cup of coffee

cfleming15:09:29

To get through the whole thread

alexmiller15:09:51

again, with a custom client accept function though - it's up to you

alexmiller15:09:10

we don't have to a priori make any decisions on the language side

cfleming15:09:29

Right. I like the idea of keeping the server as minimal as possible.

alexmiller15:09:45

so you could definitely provide a framing data repl server accept fn

cfleming15:09:46

After this discussion I don’t think the session hopping stuff belongs there.

cfleming15:09:13

Right, I’m not sure that’s something you need to add explicit support for

alexmiller15:09:36

which I am happy about

cfleming15:09:49

I’ll think about it. I like the flexibility of the minimal core, what I’m not sure of is how much it would be useful to provide to the user as example or built-in implementations.

alexmiller15:09:58

this is the best discussion I've had all week :)

cfleming15:09:01

I mean, it can all be libraries and maybe that’s a better idea.

alexmiller15:09:37

at a bare minimum I will build some examples to test it and maybe they become options that are available, or maybe libs, or maybe just documentation

cfleming15:09:59

Yeah, it might even be a good candidate for a core library.

cfleming15:09:07

Not sure about that.

alexmiller15:09:27

well I won't decide that now :)

cfleming15:09:47

Ok, bedtime for me. Finally got the jetlagged toddler off to bed.

alexmiller15:09:05

later! again, big thanks for your time.

cfleming15:09:26

No worries, I’ll try to do a client implementation for Cursive sometime soon, and let you know how it goes.

bronsa17:09:15

@alexmiller: by any chance, do you have a set of benchmarks for protocol method calls that you use?

alexmiller18:09:01

nothing official. I've created various things at different times