Clojurians Log v2

I'm trying to map my namespaces to sci namespaces automatically, but I'm messing something up: (def sci-opts {:namespaces {'user (copy-ns 'user (sci/create-ns 'user))}}) Any pointers? I'm sure I'm doing something obviously stupid.

you don't need the quote in 'user in the first arg of copy-ns

Allright, that should have been clear to me from the assert! Thank you.

although we could allow that in the macro to just take the second element if the argument is (quote user)

Well, I don't think there's necessarily any problem with the existing API. I think it's just that I don't have an example to reference, and I'm porting some existing code from using the self-hosted cljs to sci, so honestly things are a bit chaotic. But I'm hopefully almost there!

@borkdude So I see that sci doesn't (yet) support deftype - what will happen if you load a namespace that directly or indirectly uses deftype ?

@pmooser That's not a problem. You just cannot (at the moment) execute deftype from source

But SCI can re-use anything that's being brought in via functions in the SCI config

because it just calls those functions, it's not any magic :)

So I'm still on my journey of playing with copy-ns and see if I can replace my use of self-hosted cljs. I'm noticing that with a specific namespace only, when I try to use copy-ns, I get an error, and I was wondering if that could be caused by trying to copy a namespace containing a deftype.

So

(deftype Foo [])
(defn foo [] (->Foo))

(sci/eval-string "(foo)" {:namespaces {'user {'foo foo}}})

Encountered error when macroexpanding sci.core/copy-ns.
NullPointerException: 

(I realize that's not a very helpful stack or error I just pasted)

@pmooser I've had some of these issues too, it usually happens when there are protocols in that namespace.

you can try to :exclude those by name

I also know what the issue is

Ah yes, this namespace in particular contains a number of protocols.

but haven't gotten around to fixing that yet

So if I can ask, how does your foo example you pasted above work, if sci doesn't support deftype?

you can also just use copy-var - it's a bit more boilerplate, but should work more robustly

yes, because SCI has a reference to the foo function, it can just call that and then the rest within that function call isn't interpreted anymore, it's compiled code, so SCI isn't involved in that

Well, in this particular case, this namespace exports relatively few things, and the useful ones are things that return deftype instances, so that sounds like maybe the only real solution is to omit the namespaces, unless I'm not understanding what is happening.

Ahhhh ok.

So even though it doesn't support it, it might be able to call it, but not in some kind of dynamic fashion.

With "not supported in source" I mean:

(sci/eval-string "(deftype Foo [])")

Sure, ok. I apologize if my questions are a bit repetitive - I'm just trying to wrap my head around it.

no problem

I'd be interested in a repro of copy-ns + protocol behavior, so it can be fixed though

@pmooser which version of SCI are you using?

I can't reproduce it like this:

foo=> (defprotocol Foo (foo [this]))
foo=> (require '[sci.core :as sci])
nil
foo=> (sci/copy-ns foo (sci/create-ns 'foo))
{Foo #'Foo, foo #'foo}

Ah:

foo=> (deftype Dude [])
foo/Dude
foo=> (sci/copy-ns foo (sci/create-ns 'foo))
Unexpected error (NullPointerException) macroexpanding sci.core/copy-ns at (<cljs repl>:1:1).
null

@borkdude 0.3.3 which looks like I'm out of date by one revision.

I'll update to 0.3.4 and see what I can do.

I didn't have any luck with :exclude and copy-ns yet - I tried just listing all my protocols and types but it didn't seem to work.

Sorry, I'm wrong - I'm already on 0.3.4.

I'm trying to figure out how to reproduce it, since I can't just start commenting out pieces of it, since it's a vital part of my program. Ah, let me try physically copying that ns to another file, seeing if I can reproduce it there, and just start nuking parts of the file until the error goes away.

@pmooser Should be fixed with 03819bad08547a426c041e4031010f4ef75dfc0b

It was a deftype "var" in CLJS which has no :meta field

and when copying vars SCI expected that to be there

and this is why there was an NPE

@pmooser could you try that commit? (assuming you use deps.edn)

@borkdude Here is a minimal reproduction:

(ns my.ns)

(defprotocol PFoo
  (do-foo [this]))

(defrecord Foo [arg]
    PFoo
    (do-foo [this] 100))

It doesn't like the defrecord there and I'm not sure the protocol has anything to do with it.

Ok, I can see if I can that try that new SHA ...

ok, now works:

ClojureScript 1.10.914
cljs.user=> (ns my.ns)

(defprotocol PFoo
  (do-foo [this]))

(defrecord Foo [arg]
    PFoo
    (do-foo [this] 100))
nil
my.ns=> my.ns=> false
my.ns=> my.ns=> my.ns/Foo
my.ns=> (require '[sci.core :as sci])
nil
my.ns=> (sci/copy-ns my.ns (sci/create-ns 'my.ns))
{PFoo #'my.ns/PFoo, do-foo #'my.ns/do-foo, ->Foo #'my.ns/->Foo, map->Foo #'my.ns/map->Foo}

Great. Trying the new sha, just gotta remember the syntax for deps

{:git/url "" :git/sha "03819bad08547a426c041e4031010f4ef75dfc0b"}

perfect thanks

Sorry, I'm having some kind of issue (not related to the new version, just related to getting it):

Cloning: 
Checking out:  at 03819bad08547a426c041e4031010f4ef75dfc0b
Error building classpath. Unable to compare versions for org.babashka/sci: {:mvn/version "0.3.4", :deps/manifest :mvn} and {:git/url "", :git/sha "03819bad08547a426c041e4031010f4ef75dfc0b", :deps/manifest :deps, :deps/root "/Users/pmooser/.gitlibs/libs/org.babashka/sci/03819bad08547a426c041e4031010f4ef75dfc0b"}

it seems you have both SCI versions on the classpath

Yeah I'm working on fixing that ...

Ok, now moving on to testing the new version ..

@borkdude Nice job, it looks like it is fixed.

🎉

Do you have any advice for using this in the context of interactive development, like with shadow-cljs and hot code reloading? Is there a way to keep the existing sci environment but maybe update the namespace mappings?

@pmooser Yes: • create a sci context with sci/init using defonce and store that context in a mutable ref (atom or volatile) • evaluate using eval-string* + that context • in the REPL, update mapping using sci/merge-opts and swap! the new context into the mutable ref

So something like:

(defonce sci-ctx (atom (sci/init {...initial options ...}))
(eval-string* @sci-ctx "(+ 1 2 3)")
(swap! sci-ctx sci/merge-opts {:namespaces {'foo {'bar (fn ...)}}})

Ok, great - merge-opts is what I was missing. Thanks again!

@pmooser can I ask what type of app you're working on? closed or open source? domain? just curious

thanks for responding in private

I think this kind of project is important for the clojure ecosystem. Rich used to talk about clojure-in-clojure, but I think that's kind of never happening, and the self-hosted cljs is such a marginalized experience that any time something weird happens with it and shadow-cljs, immediately all fingers always point to the self-hosted cljs code.

So anyway, thanks for working on it ... it looks good so far.

im looking at SCI as a drop-in replacement for clojail. in the past i used clojail to run expressions from a large collection of #clojure IRC logs in a sandbox with a java security policy in case there was anything malicious. one thing i immediately noticed is that SCI doesn’t have a notion of timeout when eval’ing a string. given a lot of my inputs are things like (map inc (range)) and similar expressions which will run forever, i need some way of short-circuiting if it hasn’t returned in say 10sec. is that something SCI can accommodate?

i guess maybe I could just run things in a thread and set a timeout

@devn That's correct, SCI does not have a timeout, you should use a host-specific wrapper for this, e.g. in a JVM use a Thread with .stop or so

In Node.js there is "vm" which you can use for this

related: capturing *out* in addition to value is possible right? for example: (time (inc 1)) ’s value and output

@devn yes, sci has its own *out* which you can bind to clojure's *out* if you want:

(sci/binding [sci/out *out*] ....)
(sci/with-out-string ....)

great, thanks

clojail does seem quite a bit more permissive, but im going to give SCI a try on the input and see how it works out

for instance, saw immediately that time is not in there

time is something we can probably just add. it is available in babashka which uses SCI. you can copy it from there for now. https://github.com/babashka/babashka/blob/9727cabaf3499c7129657cb91b097c92152fe144/src/babashka/impl/clojure/core.clj#L17

This also works:

user=> (sci/binding [sci/out *out*] (sci/eval-string "(time (inc 1))" {:classes {'java.lang.System System} :namespaces {'clojure.core {'time (sci/copy-var time (sci/create-ns 'clojure.core))}}}))
"Elapsed time: 0.262606 msecs"
2

Ah now I remember why time wasn't added, since it requires java.lang.System ;)

🙂

i jumped in here like “ok, time to reboot this old bit-rotted project” and then got flung into “oh right, what i allow and don’t is actually half the project”

@devn what's the project, just out of curiosity?

https://github.com/devn/getclojure

ok one more question for ya: in your example above where you bind out — i want to produce a map containing the input expr, the value of the expr, and the out value

in the past i did something like bind #'*out* to a StringWriter.

(sci/binding [sci/out (new StringWriter)] ...)

d’oh, right, thank you

err i guess i tried that — i expected to be able to call (str sci/out) but that doesn’t seem to be it

nvm got it

(let [sw (StringWriter.)] ... (str sw))

(with-open [w (StringWriter.)] (sci/binding [sci/out writer] {:output (str w)}))

err after eval’ing it of course…

user=> (with-open [w (java.io.StringWriter.)] (sci/binding [sci/out w] (sci/eval-string "(prn :hello)") {:output (str w)}))
{:output ":hello\n"}