This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-06-10
Channels
- # announcements (1)
- # babashka (178)
- # beginners (216)
- # bootstrapped-cljs (1)
- # brompton (5)
- # calva (3)
- # chlorine-clover (1)
- # clj-kondo (2)
- # cljdoc (37)
- # cljfx (4)
- # cljs-dev (2)
- # clojure (360)
- # clojure-chile (8)
- # clojure-europe (3)
- # clojure-italy (5)
- # clojure-nl (9)
- # clojure-spec (2)
- # clojure-sweden (1)
- # clojure-uk (61)
- # clojuredesign-podcast (1)
- # clojurescript (83)
- # clr (2)
- # conjure (4)
- # core-async (14)
- # cursive (20)
- # data-science (2)
- # datomic (15)
- # docker (11)
- # emotion-cljs (1)
- # figwheel-main (28)
- # find-my-lib (1)
- # fulcro (46)
- # helix (16)
- # honeysql (14)
- # jobs (10)
- # jobs-discuss (17)
- # joker (1)
- # juxt (9)
- # kaocha (8)
- # leiningen (3)
- # meander (3)
- # news-and-articles (1)
- # off-topic (110)
- # pathom (7)
- # pedestal (4)
- # protojure (2)
- # re-frame (12)
- # reagent (25)
- # ring (4)
- # shadow-cljs (109)
- # spacemacs (9)
- # specter (1)
- # sql (3)
- # tools-deps (23)
Hi! I'm trying to compile the babashka on OS X from sources, and I'm getting this
maharj:babashka git:(master) $ script/uberjar
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM GraalVM CE 20.1.0 (build 25.252-b09-jvmci-20.1-b02, mixed mode)
Compiling babashka.curl
Compiling babashka.impl.async
Compiling babashka.impl.bencode
Syntax error macroexpanding at (interop.cljc:1:1).
Execution error (UnsupportedClassVersionError) at java.lang.ClassLoader/defineClass1 (ClassLoader.java:-2).
sci/impl/Reflector has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
It seems that something is picking up a different compiler at some stage, but I'm quite baffled what and who? Has someone seen this before?This is such an awesome tool – I want to do some JDBC-related stuff so hence the need to do a custom build. Been using the brew-version for quite a while now to do things. Thanks so much for making this 🙂
Bb now requires GraalVM java 11. If you want to do sql stuff check out babashka-sql-pods
Ah! 😄 https://github.com/borkdude/babashka/blob/master/doc/build.md that doc says to use the java8?
Postgres
Sounds good, where can I find details on that?
OK! Duckduckgo was not that helpful.
Found it – thanks!
Works like a dream. Thank you again 🙂
I updated the build docs to reflect the required java11 version. Hopefully this helps!
Yes, the pod works The only quirk I could find when I implemented the thing I wanted was that date handling should be done via next.jdbc.date-time
but that was not importable. I circumvented that by just using timestamps.
Sure, no problem 🙂 I could do that PR for you.
Question. I need to require [next.jdbc.date-time]
in the pod.babashka.sql
namespace. Does it matter that it's there even for HSQLDB, or do I conditionally require it? Tests seem to be passing for hsqldb as well with that in place.
I'll put the MR in and you can comment on it there.
well, https://cljdoc.org/d/seancorfield/next.jdbc/1.0.462/api/next.jdbc.date-time
OK. MR is in. You can make additional checks if you need? I only checked that running the HSQLDB suite did not fail after my modification.
As I understand from the doc it should be OK to have it there all the time.
does aero work with babashka? i am trying to figure out to have a way of having a edn file with #env and use babashka to substitute it 🙂
now that i think of it i might not even need aero and just do it purely with clojure.edn 😛 and custom readers
@helios I don't know if aero works with bb, but you can use (clojure.edn/read-string {:readers ...} ...)
Ahaha. One question though: Since bb without any flag already assumes *input*
is edn, how can i make it use custom data readers?
echo "{:foo #env BAR}" | bb -e '(prn (assoc *input* :bar "baz"))'
complains because it can't directly parse the input
i can probably work around it by using -i
and having the input as text, or by using a cli and reading directly with io/file
echo "{:foo #env BAR}" | bb -e "(prn (assoc (edn/read {:readers {'env identity}} *in*) :bar \"baz\"))"
{:foo BAR, :bar "baz"}
right now the :readers
for *input*
aren't configurable, it's just a convenience thing
@borkdude ping me if there's an equivalent change that needs making, happy to support bb.
well, this interop form has taken me a few hours now... https://github.com/juxt/aero/blob/1749e71f327d94805fdbb50d0cbbd692f696997d/src/aero/alpha/core.cljc#L6 since bb expects that call to be a method call and not a static field access. but I guess I should just fix that in bb
https://stackoverflow.com/questions/43150765/how-to-use-clojures-persistentqueue#43151037 I think I got that from here
I thought I could be clever and resolve static Java fields at analysis time instead of "interpretation" time. But now I realize, these fields could be mutable 😕
$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {aero {:mvn/version "1.1.6"}}}')
$ rlwrap lein bb
user=> (require '[aero.core :refer (read-config)])
nil
user=> (slurp "/tmp/config.edn")
"{:classpath #env \"BABASHKA_CLASSPATH\"}\n"
user=> (-> (read-config "/tmp/config.edn") :classpath (subs 0 50))
"src:feature-xml:feature-core-async:feature-yaml:fe"
This is with commit cc4ac98795c1134efc48dc1b381124e4937fd7c7
. New binary should arrive soon in #babashka_circleci_builds for testing.
$ time bb -e "(require '[aero.core :refer [read-config]]) (-> (read-config \"/tmp/config.edn\") :classpath (subs 0 100))"
"src:feature-xml:feature-core-async:feature-yaml:feature-csv:feature-transit:feature-java-time:featur"
bb -e 0.03s user 0.02s system 93% cpu 0.060 total
@dominicm Small update: I've got a metadata issue with bb + aero in alpha.core/reassemble
...
This works:
$ clj -A:bb-local -cp src -e "(require '[aero.core :as a]) (a/read-config (java.io.StringReader. \"{:x 1 :y #or [#ref [:x] \\\"bar\\\"]}\"))"
{:x 1, :y 1}
but this doesn't:
$ clj -A:bb-local -cp src -e "(require '[aero.core :as a]) (a/read-config (java.io.StringReader. \"{:y #or [#ref [:x] \\\"bar\\\"]}\"))"
WARNING: Unable to resolve "#ref [:x]" at [:y :form 0]
java.lang.Exception: Cannot call nil as a function. [at line 16, column 3]
https://bin.disroot.org/?8ae034a83a1d024d#2gaRkSgd9LPKXoS3sPYdcFULh9g4XABwVqxHEkaWgsHK
interesting, I didn't know you could actually bypass the limit using unofficial clients
so the issue I'm having in bb is that the metadata on something is nil while it shouldn't be:
(defn reassemble
[this queue]
((get
;; this is null while it should not be!
(meta this) `reassemble)
this
queue))
repro:
$ rlwrap bb -e '(ns foo) (def x) (ns bar (:require [foo :refer [x]])) `x'
bar/x
it should be foo/x
well, the metadata included a function like
{`reassemble (fn [...])}
and this symbol was wrongly resolvedI’ve found a counter example of where Sci’s :termination-safe true
doesn’t (really) protect you. It’s using recursion:
((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1))))) 20)
With n = 100
I had to kill my jvm. Is this within the scope of Sci?Or with bb:
time bb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1))))) 30)'
1346269
bb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1))))) 30)' 6.82s user 0.11s system 99% cpu 6.944 total
40
doesn’t finish with 5 minutes:
time bb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1))))) 40)'
^Cbb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1))))) 40)' 287.09s user 1.22s system 99% cpu 4:49.90 total
I’m guessing protection against situations like this would require a counting mechanism at the interpreter level https://github.com/borkdude/sci/blob/master/src/sci/impl/interpreter.cljc#L537 Not sure what other consequences this would have
@jeroenvandijk well, it does terminate, so it doesn't fall outside the category of termination-safe 🙂
I did consider examples like this but most of them fail already because of a stackoverflow: $ lein run -m sci.impl.main "((fn a [n] (a (inc n))) 100)"
so the question is: why does that function not crash because of a stack depth problem
Not sure, maybe because the stack is not filling up fast enough? I think the interpreter needs to do some work for every iteration
would you be happy to accept a patch to protect against this? I will give it a try. I think it’s a risk to rely on the stackoverflow error if you really want to use sci in places where it could be abused
timeouts don’t really work for endless loops i think
you would have to call thread/interupt every X iteration or something to make this work
i’ll look up the reference. I’ve looked into this before
Ah I see
just like a counter you could check the time I guess
why wouldn’t the counter work?
you call interpret every time?
is this a real problem you were having or a theoretical one? how would a counter protect you from @(promise)
?
For now it’s theoretic, but since it’s a possibility it will limit the scope of what Sci can do
I would not use Sci for editable pages by others
Otherwise I would like to do that for demo purposes
yes, so people could be “hacked” by others
no big harm, but still annoying
And I want to build a server that runs scripts made by others. One script could bring down the whole server (theoretically)
multitenant would become much harder
So all future use cases 🙂
you can always run in a Thread and then kill the Thread, I don't think sci can protect you from all cases
yeah I guess that’s the safest approach
Maybe the termination-safe option should also just be deprecated, since it's not entirely fool proof
would it be possible to stop recursion by not allowing the symbol of an fn
? e.g. the option of not allowing (fn a []…)
vs (fn [] )
I think it would be very powerful to have Sci become termination-safe. I would like to try and see if that’s possible
(given certain constraints, e.g. not allowing third party functions that are not termination safe etc)
It’s not important now. I’ll come with some cases and see if I can fix it
I was considering forbidding recursive functions when I created termination-safe. After some experiments it seemed that only loop
, recur
etc were sufficient. but maybe the case you've shown here should also lead to forbidding self-recursive functions or up to a certain depth. Maybe we should also forbid promise
, which I hadn't considered before.
Yeah I guess that’s a good one
I'd not like to put a lot of checks in the middle of every evaluation since I found this conflicts with performance. My test for this was:
$ multitime -n2 -s0 ./sci "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"
ah yeah makes sense. For most use cases performance is more important
I think not allowing promise is good way to help a naive user using termination-safe. a more advanced user can create their own blacklist maybe
This is an example where timeouts don’t work:
(deref (future (take 1 (drop 1000000000 (range)))) 1000 :timeout)
The computation should stop after a second, but it takes much longer. Things like (deref (promise) 1000 :timeout)
because they are not actually computing somethingAlso stopping a thread is a bit complicated https://www.baeldung.com/java-thread-stop
I’ll give it some hammock time. Maybe there is a clean solution like how you did realize-max
(deref (future (take 1 (drop 1000000000 (range)))) 1000 :timeout)
this will probably work with realize-max already? have you tested that?stopping a thread is something which CIDER also does using the deprecated .stop
method
you can have a look at https://github.com/borkdude/clojail for more ideas for server usage.
I used that for running arbitrary code from http://clojuredocs.org
I guess it could work on the server indeed
possibly
regarding the future, it was more meant as an example where cancelling a thread (the future) doesn’t work
true, future cancelling only works if you constantly check for Thread/interrupted etc. yourself. Thread stop does work, but not in GraalVM
ahh hmm, makes sense i guess
Would it make sense to have different behaviour at analyzer level based on the :termination-safe setting? Performance sensitive code would not use the :termination-safe
setting, all :termination-safe
code generates slightly different code with fn’s that count the number of executions. This would limit the number a loop
or a recursive function can do.
I guess the expansion code could be adapted here:
https://github.com/borkdude/sci/blob/master/src/sci/impl/analyzer.cljc#L165
https://github.com/borkdude/sci/blob/master/src/sci/impl/analyzer.cljc#L353
If you think it could be a possibility, I’ll give this a try some time
Cool, I’ll try 🙂
ok i think i have a simple and performant solution. Will cleanup tomorrow and I guess it needs some discussion around the exact behaviour of options https://github.com/borkdude/sci/commit/1378bc1304d064de50667f381adaac17239902a1
just FYI: Python (also an interpreter) uses a settable stack size limit https://docs.python.org/3.8/library/sys.html#sys.setrecursionlimit
Ah good to know. I’ll test my approach in a bit. I’m expecting that it can be a flag that doesn’t have to slow things down when it’s not turned on
@jeroenvandijk what about emitting code in the analyzer(.clj) instead of putting the checks in the runtime fns
?
that will have almost no impact on the runtime if the option isn't used, whereas now it's still going through a (no op) function call each time
yeah I can remove the no op function
I’ll finish this first and then look at the analyzer option. Not sure where to put it
also, I think it might be good to back up a bit and come up with a good problem statement, before working on a patch.
what is the problem that we are solving here? things can take too long? what are the alternative solutions?
the :timeout
solution could also be an approach (instead of checking a counter, check the time passed since the start).
so the order of things:
1) come up with a problem statement, write an issue, describe alternatives
2) explore alternatives using code
3) discuss
4) write PR, then discuss details of that PR
sure, I’ll do that. Now it’s just to get it out of my head
Ok created an issue and a PR 🙂 I’ll do it in the opposite order next time 🙃 https://github.com/borkdude/sci/issues/348#
Thank you. No rush. I’ll also see if I can move it to the analyzer namespace
So Athens (and Roam) use Datascript as an underlying storage. This gives it a lot of flexibility, but it needs some post processing to be useable. This is where Sci comes in. Sci can do simple things like sort or formatting. It can however also be used to do more advanced things like math or controlling some kind of visualization of this data (no examples because it doesn’t exist yet). The power of Sci is that it can allow for arbitrary execution. This needs to be controlled in another way to make it reliable
@dominicm yes!
$ clj -A:bb-local -cp src -e "(require '[aero.core :as a]) (a/read-config (java.io.StringReader. \"{:y #or [#ref [:x] \\\"bar\\\"]}\"))"
WARNING: Unable to resolve "#ref [:x]" at [:y :form 0]
{:y "bar"}
Woohoo, all aero tests are now passing in bb except for one minor thing which I know how to fix.
This is really exciting… I'm wondering what crazy things I can do with this. One idea is a CLI migration tool that will read my config.edn without paying a startup penalty!
@dominicm for static analysis, you might also want to look at: https://github.com/babashka/pod-babashka-parcera
babashka v0.1.1: https://github.com/borkdude/babashka/blob/master/CHANGELOG.md#v011-2020-06-10 This release can now run clj-commons/camel-snake-kebab and juxt/aero as libraries.