Fork me on GitHub
#sci
<
2021-08-18
>
victorb09:08:54

@borkdude I created a minimal example here: https://github.com/victorb/sci-reagent-with-let, tried a bunch of different things suggested and also guessed but to no avail

victorb10:08:53

Yeah, sorry about that, I've tried wrapping :require'd reagent.core/with-let in a .clj macro and function, tried adding it directly via :bindings, via :namespaces and a bunch of other versions but none are working (with different errors) I'll try adding all the different things I've tried in the same repo

borkdude10:08:44

Can you try exactly what I posted above this thread? I gave two working examples. It might be good to start from there.

victorb10:08:27

Yup, I did try those as well, will add everything to the same file

borkdude11:08:50

I will look at this closer tonight but with with-let1 the issue is that your fn expands into a call to reagent.core/with-let which you haven’t provided in your SCI config. You have to give an implementation of your macro, you can’t wrap a macro from the .clj code as this is not available at runtime

borkdude11:08:29

Just look at the promesa config in nbb and you might understand what I mean

borkdude11:08:40

On the phone in a forest right now

borkdude14:08:14

ok, I'm back now. let me just add the with-let macro to nbb, then you can just copy the config

victorb15:08:53

Back here too now. "the issue is that your fn expands into a call to reagent.core/with-let" Hmm, yeah, that seems about right, gonna see if I can correct the SCI config

borkdude15:08:22

I'm looking into it now as well

❤️ 3
victorb16:08:54

Nice! I was close, but yours is way better : ) Just as Ithought then, just a user error. Got it all working now, thanks a lot for the help, really appreciate it!

borkdude16:08:48

I actually think there needs to be 2 more improvements

victorb16:08:29

Oh. Haven't gotten as far to integrate the PoC into the project itself yet, anything obviously broken?

borkdude16:08:32

Add reagent.ratom/reactive? and synchronizing the reagent.ratom/*ratom-context*properly, I don't think that currently works

borkdude16:08:44

The first one is for the finally part

borkdude16:08:09

I'll have a stab at this later tonight

victorb16:08:28

I see. Thanks, there is no rush and you've helped plenty already. I'll give it a shot and see if I can get it to work, otherwise ask a bit more targeted questions

borkdude19:08:49

@UEJ5FMR6K Pushed the final fix, now finally also properly works with with-let

victorb15:08:20

Random note, just noticed SCI has 1024 commits just yesterday! Thanks for the awesome work on it borkydude, saved lots of time and always been a joy to use! 🎉

🎉 9
deactivateduser18:08:54

Kind of a follow up to yesterday’s discussion regarding macros; I’m trying to expose the clojure.repl/doc macro in sci (ideally emitting output for the actual ClojureJVM symbol that’s referred to, not the sci-equivalent) and have gotten a bit stuck. Here’s the closest I’ve gotten: (sci/eval-string "(doc println)" {:namespaces {'user {'doc (sci/copy-var clojure.repl/doc (sci/create-ns 'user nil))}}}) Which throws a ClassCastException from . Any tips?

borkdude18:08:12

@deactivateduser10790 babashka itself exposes clojure.repl/doc as well. if you want to enable printing then you can do (sci/alter-var-root sci/out (constantly *out*))

borkdude18:08:43

but if you want to look outside of the SCI ctx for what Clojure vars expose, then you could rewrite the clojure.repl/doc impl to do so.

borkdude18:08:08

Note that SCI already has clojure.repl/doc

borkdude18:08:23

you just need to enable the println using the sci/alter-var-root thing

deactivateduser18:08:45

Ah ok. I guess I’ll need to bind sci/out to a StringWriter though, as I need to send the output to somewhere other than local stdout.

borkdude18:08:05

@deactivateduser10790 you can also use sci/with-out-str for this, if that works for your use case

borkdude18:08:26

but I guess you also want the return value, in general

deactivateduser18:08:45

Yeah correct. Basically I’m using sci as a repl within a chat server.

deactivateduser18:08:07

Right now I just dump the result of evaluating a chat message, but capturing stdout would also make sense.

borkdude18:08:28

makes sense, sounds like a fun project!

deactivateduser18:08:41

Don’t laugh too hard at the code: https://github.com/pmonks/for-science 😉

deactivateduser18:08:28

It’s in the Clojure Discord server, if you’re interested in taking it for a spin: https://discord.gg/discljord

borkdude19:08:10

that's a different discord than the clojure one right?

deactivateduser20:08:05

It’s the replacement, yes. The clojure one’s admin left without anointing a new admin, so that one gets spammed pretty regularly, and no one has the ability to block the spammers.

deactivateduser20:08:44

As a result the admins of discljord have requested that links to the old one be removed from other Clojure resources and replaced with the discljord one, with mixed success…

deactivateduser20:08:10

Back to my bot though - I implemented what you described above, approximately as follows:

(let [sw     (java.io.StringWriter.)
      result (print-str (sci/binding [sci/out sw
                                      sci/err sw]
               (sci/eval-string (str "(use 'clojure.repl)\n" code)))]
Where “code” is the content of the chat message typed by a human. However when “code” is (doc println) , the StringWriter is empty (and result is “nil”, as expected).

deactivateduser20:08:42

It gets weirder too: (println "hello, world!") gives sw = “hello, world!“, result = nil (correct) (range 10) gives sw = “”, result = (0 1 2 3 4 5 6 7 8 9) (correct) (map println (range 10)) throws a ClassCastException (why???) and (doall (map println (range 10))) works as expected. Am I right in thinking that laziness “escapes” the call to the interpreter, but then fails for “I/O-ish” things, because the binding is no longer in effect? If so, is there any way to force de-laziness at that call site, short of wrapping “code” with a (doall) ?

borkdude20:08:44

you could try forcing the result by stringifying it?

deactivateduser20:08:57

I am - that’s what print-str does. But that’s outside the sci/binding so any laziness with side effects that rely on those bindings will fail.

borkdude20:08:19

then perhaps do that within the binding?

deactivateduser20:08:38

Yeah exactly. But I’m still surprised / concerned that sci lets laziness “escape” the interpreter. That doesn’t seem right.

deactivateduser20:08:54

And of course this also doesn’t explain why doc isn’t working (is not producing any output).

deactivateduser20:08:30

Should I just raise an issue on that in the repo?

borkdude20:08:39

what's the issue with doc again?

borkdude20:08:25

about laziness: there isn't a single right answer there, sci has different use cases, protecting the user against laziness isn't something that should be a choice inside the lib imo

borkdude20:08:39

in earlier versions, sci had something like "realize-max" which threw if you realized a lazy seq with more than n elements, this was to protect you from evaluating (doall (range)), but this came with performance trade-offs which in the end weren't really worth it

borkdude20:08:11

so it's hard to say: always do x because my use case is y for all y

borkdude20:08:09

user=> (let [sw (java.io.StringWriter.)] (sci/binding [sci/out sw] (sci/eval-string "(clojure.repl/doc inc)")) (str sw))
"-------------------------\nclojure.core/inc\n([x])\n  Returns a number one greater than num. Does not auto-promote\n  longs, will throw on overflow. See also: inc'\n"

deactivateduser21:08:04

Conversely:

(let [sw (java.io.StringWriter.)] (sci/binding [sci/out sw] (sci/eval-string "(clojure.repl/doc println)")) (str sw))
""

deactivateduser21:08:20

(let [sw (java.io.StringWriter.)] (sci/binding [sci/out sw] (sci/eval-string "(clojure.repl/doc if)")) (str sw))
""

deactivateduser21:08:46

user=> (let [sw (java.io.StringWriter.)] (sci/binding [sci/out sw] (sci/eval-string "(clojure.repl/doc let)")) (str sw))
""

deactivateduser21:08:04

All of which produce output when run with “real” Clojure.

borkdude21:08:10

try a more recent version of sci, a lot of these have been fixed

deactivateduser21:08:43

This is with v0.2.6, which is the latest AFAIK?

borkdude21:08:59

hmm yeah, perhaps not, special forms, and println are special, they might not have the proper docstrings

borkdude21:08:07

on SCI master I mean

borkdude21:08:11

use as git dep

deactivateduser21:08:29

Is master stable?

borkdude21:08:32

but you mentioned you were going to pick the docstrings from "real" Clojure anyway, if you're running in a JVM then this is easy.

borkdude21:08:50

master is usually better than the latest release

deactivateduser21:08:17

Ok - I’ll use that. And regarding “Clojure doc” vs “sci doc”, I don’t have a strong preference. Whatever is less code for me to write & maintain. 😉

borkdude21:08:38

@deactivateduser10790 rough version:

user=> (let [sw (java.io.StringWriter.)] (sci/binding [sci/out sw] (sci/eval-string "(clojure.repl/doc println)" {:namespaces {'clojure.repl {'doc ^:sci/macro (fn [_ _ sym] (binding [*out* @sci/out] (clojure.repl/find-doc (re-pattern (str sym)))))}}})) (str sw))

💯 3
deactivateduser21:08:40

Back to the earlier topic of laziness, it seems like that should be scoped to a sci context. In my case I’m not maintaining a context across calls to eval-string, so I would expect any interpreter “state” to go away as soon as that call is ended.

deactivateduser21:08:07

(and laziness, by virtue of the possibility of side-effects via bindings, I’d consider as part of that “state”)

deactivateduser21:08:29

(in effect I want a self-contained, one-shot interpreter)

borkdude21:08:50

well, as I said, that's just one use case. there are use cases where this escaping is desirable, e.g. when returning a function from the interpeter which should be callable outside of the context

borkdude21:08:57

e.g. SCI usage in malli

deactivateduser21:08:04

And in those cases is it onerous for the consumer to maintain a context?

deactivateduser21:08:12

Given that that provides them with “long lived” scope?

borkdude21:08:33

About laziness escaping: I'm very caution of adding features around protecting users from this as this will vary from use case to use case. Write your own protection layer for your use case, for now.

borkdude21:08:21

I have tried protecting users from other stuff and eventually I removed it because it wasn't working 100% correctly or it degraded performance.

deactivateduser21:08:50

Right, and I’ve already done that, including limiting sci’s execution time (right now to 2 seconds max). I’m just suggesting that this is a violation of the principle of least surprise. Sci already has contexts (which I’d understood to be a form of interpreter scoping), and because my use case didn’t need that, I didn’t use them. Now I discover that some of sci’s “scope” is indeed escaping the interpreter context (which just happens to not work for side-effecty code).

deactivateduser21:08:46

Take it as general feedback of the “is this design consistent?” form, rather than feedback of the “I’ve found a bug!” form. 😉

deactivateduser21:08:56

I hope you haven’t misunderstood it as the latter - that wasn’t my intent. [edits because words are hard]

borkdude22:08:33

OK, it might be good to add some docs around this.

borkdude22:08:54

The idea of the context is mostly to remember config + side effects like defined vars/functions over multiple invocations

borkdude22:08:12

That's about it really

deactivateduser22:08:18

Right. Seems like “state of laziness” could be considered as part of the same thing.

deactivateduser22:08:33

(since it includes state that has to survive across calls to eval-string)

deactivateduser22:08:06

BTW I’m trying to find how clojail used to do this…

deactivateduser22:08:16

(it was what I’ve used in the past for this kind of thing…)

borkdude22:08:18

every call to eval-string or eval-string* has a context, but with eval-string* that context is explicit while with eval-string the context is created for you every time

deactivateduser22:08:49

Sure, at the implementation level. But at the conceptual level eval-string only has a context within that one call.

deactivateduser22:08:59

As a consumer I never need to know about it.

deactivateduser22:08:31

I just fire a string value at eval-string and (in theory) everything that string value does is encapsulated within that call - nothing leaks out.

deactivateduser22:08:58

(except the final result of evaluating the string, and any bindings, of course)

borkdude22:08:11

that was never the intent of the context. e.g. with malli and clj-kondo, it's quite the reverse because the returned functions are basically closures over the context which can be invoked many times

deactivateduser22:08:32

In which case those consumers can (and should) create a persistent context and keep it around, no?

deactivateduser22:08:46

Since they need to explicitly manage a long-lived scope?

borkdude22:08:58

can but not necessarily should

deactivateduser22:08:26

Again, take this feedback as “is this design consistent?“, not “I’ve found a bug!” 😉

borkdude22:08:59

e.g. (eval-string "(def x 1) (fn [] x)"), what is the principle of least surprise here when invoking the result of eval-string?

deactivateduser22:08:38

user=> (def x 1)
#'user/x
user=> (fn [] x)
#object[user$eval146$fn__147 0x1be59f28 "user$eval146$fn__147@1be59f28"]

deactivateduser22:08:41

Something like that

deactivateduser22:08:59

Though of course the result of the first line gets thrown away.

borkdude22:08:43

I meant, invoking the result, like (*1) I think it's better to document various use cases of SCI and how to leverage it e.g. for your one shot interpeter use case than to make these choices inside the lib. So if you want to contribute to the docs, feel free to do so.

deactivateduser22:08:45

And, in my case, the var x gets thrown away too.

deactivateduser22:08:58

Since I don’t want / need long-lived state, so haven’t provided a context.

borkdude22:08:41

I'm calling it a night, it's getting late here

borkdude22:08:45

past midnight

deactivateduser22:08:45

Again, is this design consistent? That the “one shot interpreter” use case has to do extra work to meet its need? Work that isn’t necessarily if lazy results are de-lazied within eval-string, and by doing so other use cases (that make use of explicit long-lived contexts) are not impacted?

deactivateduser22:08:22

These are not rhetorical questions btw - there are different answers with varying trade-offs.

deactivateduser20:08:50

To answer my second question (the weird behaviour with map and println) - wrapping the call to sci/eval-string in a doall seems to have solved that issue, which confirms that laziness “escapes” sci.

deactivateduser20:08:00

Is that intended though? That doesn’t seem safe to me…