Fork me on GitHub
#babashka
<
2020-06-10
>
mikko07:06:10

Hi! I'm trying to compile the babashka on OS X from sources, and I'm getting this

maharj:babashka git:(master) $ script/uberjar
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM GraalVM CE 20.1.0 (build 25.252-b09-jvmci-20.1-b02, mixed mode)
Compiling babashka.curl
Compiling babashka.impl.async
Compiling babashka.impl.bencode
Syntax error macroexpanding at (interop.cljc:1:1).
Execution error (UnsupportedClassVersionError) at java.lang.ClassLoader/defineClass1 (ClassLoader.java:-2).
sci/impl/Reflector has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
It seems that something is picking up a different compiler at some stage, but I'm quite baffled what and who? Has someone seen this before?

mikko07:06:36

This is such an awesome tool – I want to do some JDBC-related stuff so hence the need to do a custom build. Been using the brew-version for quite a while now to do things. Thanks so much for making this 🙂

borkdude07:06:52

Bb now requires GraalVM java 11. If you want to do sql stuff check out babashka-sql-pods

borkdude07:06:52

Bb now requires GraalVM java 11. If you want to do sql stuff check out babashka-sql-pods

borkdude07:06:22

Should be updated to java 11 20.1.0

borkdude07:06:23

Which db do you use ?

borkdude07:06:44

Using a pod you can just use the one from brew then

mikko07:06:15

Sounds good, where can I find details on that?

borkdude07:06:48

Google babashka-sql-pods. I’m on the phone

mikko07:06:02

OK! Duckduckgo was not that helpful.

borkdude07:06:28

Under the babashka org on Github

mikko07:06:38

Found it – thanks!

mikko07:06:45

Works like a dream. Thank you again 🙂

mikko07:06:36

I updated the build docs to reflect the required java11 version. Hopefully this helps!

borkdude08:06:41

Cool. the pod worked well for you?

borkdude08:06:39

Merged your doc changes. Thanks

mikko09:06:05

Yes, the pod works The only quirk I could find when I implemented the thing I wanted was that date handling should be done via next.jdbc.date-time but that was not importable. I circumvented that by just using timestamps.

borkdude09:06:36

I would add that to the pod if you need it.

borkdude09:06:55

Issue and/or PR welcome

borkdude09:06:08

(at babashka-sql-pods)

borkdude09:06:20

It's a pretty young project, I'm pretty sure that are some rough edges to polish

mikko09:06:53

Sure, no problem 🙂 I could do that PR for you.

mikko09:06:08

Question. I need to require [next.jdbc.date-time] in the pod.babashka.sql namespace. Does it matter that it's there even for HSQLDB, or do I conditionally require it? Tests seem to be passing for hsqldb as well with that in place.

mikko09:06:25

I'll put the MR in and you can comment on it there.

borkdude09:06:44

I think that namespace is relevant to all databases right?

borkdude09:06:20

I'm fine with including it

borkdude09:06:27

as long as it doesn't make things fail 🙂

mikko09:06:52

OK. MR is in. You can make additional checks if you need? I only checked that running the HSQLDB suite did not fail after my modification.

mikko09:06:46

As I understand from the doc it should be OK to have it there all the time.

borkdude09:06:13

yeah, looks good to me

helios08:06:50

does aero work with babashka? i am trying to figure out to have a way of having a edn file with #env and use babashka to substitute it 🙂

helios08:06:50

does aero work with babashka? i am trying to figure out to have a way of having a edn file with #env and use babashka to substitute it 🙂

helios08:06:22

now that i think of it i might not even need aero and just do it purely with clojure.edn 😛 and custom readers

borkdude08:06:30

@helios I don't know if aero works with bb, but you can use (clojure.edn/read-string {:readers ...} ...)

helios08:06:46

yeah exactly 😄

borkdude09:06:42

I'm looking into making aero working right now. Nice challenge 😉

helios09:06:05

Ahaha. One question though: Since bb without any flag already assumes *input* is edn, how can i make it use custom data readers?

helios09:06:10

echo "{:foo #env BAR}" | bb -e '(prn (assoc *input* :bar "baz"))' complains because it can't directly parse the input

helios09:06:48

i can probably work around it by using -i and having the input as text, or by using a cli and reading directly with io/file

borkdude09:06:27

echo "{:foo #env BAR}" | bb -e "(prn (assoc (edn/read {:readers {'env identity}} *in*) :bar \"baz\"))"
{:foo BAR, :bar "baz"}

helios09:06:47

Ah, so that's the difference between *input* and *in*. Gotcha

borkdude09:06:36

right now the :readers for *input* aren't configurable, it's just a convenience thing

helios09:06:15

yeah get it, to use *in* for this is fair enough

dominicm12:06:40

You won't get #profile working properly using :readers :)

dominicm12:06:57

@borkdude ping me if there's an equivalent change that needs making, happy to support bb.

borkdude12:06:28

well, this interop form has taken me a few hours now... https://github.com/juxt/aero/blob/1749e71f327d94805fdbb50d0cbbd692f696997d/src/aero/alpha/core.cljc#L6 since bb expects that call to be a method call and not a static field access. but I guess I should just fix that in bb

dominicm12:06:41

You're right, no reason for that to be a function call.

borkdude12:06:46

yeah, that syntax is supported, but a bit confusing

dominicm12:06:58

I had no idea it was 😁

borkdude12:06:05

I should fix it nonetheless to be compatible

dominicm12:06:03

fixed on master

borkdude11:06:38

I thought I could be clever and resolve static Java fields at analysis time instead of "interpretation" time. But now I realize, these fields could be mutable 😕

borkdude13:06:08

@helios @dominicm

$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {aero {:mvn/version "1.1.6"}}}')
$ rlwrap lein bb
user=> (require '[aero.core :refer (read-config)])
nil
user=> (slurp "/tmp/config.edn")
"{:classpath #env \"BABASHKA_CLASSPATH\"}\n"
user=> (-> (read-config "/tmp/config.edn") :classpath (subs 0 50))
"src:feature-xml:feature-core-async:feature-yaml:fe"

borkdude13:06:20

This is with commit cc4ac98795c1134efc48dc1b381124e4937fd7c7. New binary should arrive soon in #babashka_circleci_builds for testing.

borkdude13:06:24

$ time bb -e "(require '[aero.core :refer [read-config]]) (-> (read-config \"/tmp/config.edn\") :classpath (subs 0 100))"
"src:feature-xml:feature-core-async:feature-yaml:feature-csv:feature-transit:feature-java-time:featur"
bb -e    0.03s  user 0.02s system 93% cpu 0.060 total

borkdude13:06:03

There are probably still cases that don't work, so I'd appreciate some testing

jkrasnay15:06:55

Does anyone know how to create a self-running bb script on Windows 10?

borkdude15:06:33

do you mean with a shebang?

jkrasnay15:06:04

Yes. The problem is I don’t have WSL on my work laptop.

borkdude15:06:25

don't know, but if you find it out, it would be nice to document

borkdude16:06:32

@dominicm Small update: I've got a metadata issue with bb + aero in alpha.core/reassemble...

dominicm16:06:20

@borkdude what's the problem? :)

dominicm16:06:42

Oh, you probably don't allow dynamic evaluation like that right?

borkdude16:06:26

This works:

$ clj -A:bb-local -cp src -e "(require '[aero.core :as a]) (a/read-config (java.io.StringReader. \"{:x 1 :y #or [#ref [:x] \\\"bar\\\"]}\"))"
{:x 1, :y 1}
but this doesn't:
$ clj -A:bb-local -cp src -e "(require '[aero.core :as a]) (a/read-config (java.io.StringReader. \"{:y #or [#ref [:x] \\\"bar\\\"]}\"))"
WARNING: Unable to resolve "#ref [:x]" at [:y :form 0]
java.lang.Exception: Cannot call nil as a function. [at line 16, column 3]

borkdude16:06:51

so a dangling ref

dominicm16:06:52

Oh, well boo you slack

borkdude16:06:13

Upload as file is better for large code chunks

borkdude16:06:48

Could you remove the large blob you posted before?

borkdude16:06:19

I'm still seeing it even after refreshing the slack window

dominicm16:06:54

interesting, I didn't know you could actually bypass the limit using unofficial clients

borkdude16:06:45

so the issue I'm having in bb is that the metadata on something is nil while it shouldn't be:

(defn reassemble
  [this queue]
  ((get
    ;; this is null while it should not be!
    (meta this) `reassemble)
   this
   queue))

dominicm17:06:40

Hmm, that shouldn't be possible.

dominicm17:06:51

What is this?

borkdude18:06:21

found the bug

borkdude18:06:44

repro:

$ rlwrap bb -e '(ns foo) (def x) (ns bar (:require [foo :refer [x]])) `x'
bar/x
it should be foo/x

dominicm19:06:14

Whaat. I don't even know how that's related!

borkdude19:06:55

well, the metadata included a function like

{`reassemble (fn [...])}
and this symbol was wrongly resolved

jeroenvandijk17:06:54

I’ve found a counter example of where Sci’s :termination-safe true doesn’t (really) protect you. It’s using recursion:

((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1))))) 20)
With n = 100 I had to kill my jvm. Is this within the scope of Sci?

jeroenvandijk17:06:21

Or with bb:

time bb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1)))))  30)'
1346269
bb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1)))))  30)'  6.82s user 0.11s system 99% cpu 6.944 total

jeroenvandijk17:06:46

40 doesn’t finish with 5 minutes:

time bb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1)))))  40)'
^Cbb -e '((fn a [n] (if (#{0 1} n) 1 (+ (a (- n 2)) (a (- n 1)))))  40)'  287.09s user 1.22s system 99% cpu 4:49.90 total

jeroenvandijk17:06:07

I’m guessing protection against situations like this would require a counting mechanism at the interpreter level https://github.com/borkdude/sci/blob/master/src/sci/impl/interpreter.cljc#L537 Not sure what other consequences this would have

borkdude17:06:10

@jeroenvandijk well, it does terminate, so it doesn't fall outside the category of termination-safe 🙂

borkdude17:06:10

@jeroenvandijk well, it does terminate, so it doesn't fall outside the category of termination-safe 🙂

borkdude17:06:42

I did consider examples like this but most of them fail already because of a stackoverflow: $ lein run -m sci.impl.main "((fn a [n] (a (inc n))) 100)"

borkdude18:06:01

so the question is: why does that function not crash because of a stack depth problem

jeroenvandijk19:06:57

Not sure, maybe because the stack is not filling up fast enough? I think the interpreter needs to do some work for every iteration

jeroenvandijk19:06:04

would you be happy to accept a patch to protect against this? I will give it a try. I think it’s a risk to rely on the stackoverflow error if you really want to use sci in places where it could be abused

borkdude19:06:38

Maybe implementing a timeout is a better idea

jeroenvandijk19:06:01

timeouts don’t really work for endless loops i think

jeroenvandijk19:06:31

you would have to call thread/interupt every X iteration or something to make this work

borkdude19:06:10

I don't see why not.

jeroenvandijk19:06:48

i’ll look up the reference. I’ve looked into this before

borkdude20:06:07

it doesn't work for Clojure, but we have control over everything in the interpreter

borkdude20:06:29

hmm, maybe not, e.g. @(promise) doesn't work

borkdude20:06:47

but a counter would also not solve it

jeroenvandijk20:06:51

just like a counter you could check the time I guess

borkdude20:06:09

yes, but there are always cases to hack around it probably

jeroenvandijk20:06:18

why wouldn’t the counter work?

jeroenvandijk20:06:25

you call interpret every time?

borkdude20:06:36

is this a real problem you were having or a theoretical one? how would a counter protect you from @(promise)?

jeroenvandijk20:06:28

For now it’s theoretic, but since it’s a possibility it will limit the scope of what Sci can do

jeroenvandijk20:06:49

E.g. Athens would not be able to use it safely for public hosted pages

jeroenvandijk20:06:02

I would not use Sci for editable pages by others

jeroenvandijk20:06:14

Otherwise I would like to do that for demo purposes

borkdude20:06:18

if the code runs in the browser, it's the user's problem?

jeroenvandijk20:06:35

yes, so people could be “hacked” by others

jeroenvandijk20:06:45

no big harm, but still annoying

jeroenvandijk20:06:26

And I want to build a server that runs scripts made by others. One script could bring down the whole server (theoretically)

jeroenvandijk20:06:42

multitenant would become much harder

jeroenvandijk20:06:49

So all future use cases 🙂

borkdude20:06:59

you can always run in a Thread and then kill the Thread, I don't think sci can protect you from all cases

jeroenvandijk20:06:20

yeah I guess that’s the safest approach

borkdude20:06:46

Maybe the termination-safe option should also just be deprecated, since it's not entirely fool proof

jeroenvandijk20:06:00

would it be possible to stop recursion by not allowing the symbol of an fn ? e.g. the option of not allowing (fn a []…) vs (fn [] )

jeroenvandijk20:06:00

I think it would be very powerful to have Sci become termination-safe. I would like to try and see if that’s possible

jeroenvandijk20:06:52

(given certain constraints, e.g. not allowing third party functions that are not termination safe etc)

jeroenvandijk20:06:40

It’s not important now. I’ll come with some cases and see if I can fix it

borkdude20:06:12

I was considering forbidding recursive functions when I created termination-safe. After some experiments it seemed that only loop, recur etc were sufficient. but maybe the case you've shown here should also lead to forbidding self-recursive functions or up to a certain depth. Maybe we should also forbid promise, which I hadn't considered before.

jeroenvandijk20:06:11

Yeah I guess that’s a good one

borkdude20:06:36

I'd not like to put a lot of checks in the middle of every evaluation since I found this conflicts with performance. My test for this was:

$ multitime -n2 -s0 ./sci "(loop [val 0 cnt 1000000] (if (pos? cnt) (recur (inc val) (dec cnt)) val))"

borkdude20:06:54

which I brought down from 12 seconds to roughly 1.7 seconds

jeroenvandijk20:06:35

ah yeah makes sense. For most use cases performance is more important

jeroenvandijk20:06:12

I think not allowing promise is good way to help a naive user using termination-safe. a more advanced user can create their own blacklist maybe

jeroenvandijk20:06:44

This is an example where timeouts don’t work:

(deref (future (take 1 (drop 1000000000  (range)))) 1000 :timeout)
The computation should stop after a second, but it takes much longer. Things like (deref (promise) 1000 :timeout) because they are not actually computing something

jeroenvandijk20:06:08

Also stopping a thread is a bit complicated https://www.baeldung.com/java-thread-stop

jeroenvandijk20:06:00

I’ll give it some hammock time. Maybe there is a clean solution like how you did realize-max

borkdude20:06:44

(deref (future (take 1 (drop 1000000000  (range)))) 1000 :timeout)
this will probably work with realize-max already? have you tested that?

borkdude20:06:29

also future isn't part of the termination-safe thing, so that's not an issue

borkdude20:06:54

stopping a thread is something which CIDER also does using the deprecated .stop method

borkdude20:06:05

every time you press Ctrl-c

borkdude20:06:05

you can have a look at https://github.com/borkdude/clojail for more ideas for server usage.

borkdude20:06:41

I used that for running arbitrary code from http://clojuredocs.org

jeroenvandijk20:06:44

I guess it could work on the server indeed

borkdude20:06:49

maybe something similar can be done using webworkers and terminating those

jeroenvandijk20:06:04

regarding the future, it was more meant as an example where cancelling a thread (the future) doesn’t work

borkdude21:06:43

true, future cancelling only works if you constantly check for Thread/interrupted etc. yourself. Thread stop does work, but not in GraalVM

borkdude21:06:03

(they did not implement the deprecated method, because it was ... deprecated)

jeroenvandijk21:06:49

ahh hmm, makes sense i guess

jeroenvandijk21:06:22

Would it make sense to have different behaviour at analyzer level based on the :termination-safe setting? Performance sensitive code would not use the :termination-safe setting, all :termination-safe code generates slightly different code with fn’s that count the number of executions. This would limit the number a loop or a recursive function can do. I guess the expansion code could be adapted here: https://github.com/borkdude/sci/blob/master/src/sci/impl/analyzer.cljc#L165 https://github.com/borkdude/sci/blob/master/src/sci/impl/analyzer.cljc#L353 If you think it could be a possibility, I’ll give this a try some time

borkdude21:06:09

that also crossed my mind. worth a try at least

jeroenvandijk21:06:09

Cool, I’ll try 🙂

jeroenvandijk22:06:47

ok i think i have a simple and performant solution. Will cleanup tomorrow and I guess it needs some discussion around the exact behaviour of options https://github.com/borkdude/sci/commit/1378bc1304d064de50667f381adaac17239902a1

retrogradeorbit03:06:05

just FYI: Python (also an interpreter) uses a settable stack size limit https://docs.python.org/3.8/library/sys.html#sys.setrecursionlimit

jeroenvandijk08:06:16

Ah good to know. I’ll test my approach in a bit. I’m expecting that it can be a flag that doesn’t have to slow things down when it’s not turned on

borkdude08:06:41

@jeroenvandijk what about emitting code in the analyzer(.clj) instead of putting the checks in the runtime fns?

borkdude08:06:09

that will have almost no impact on the runtime if the option isn't used, whereas now it's still going through a (no op) function call each time

jeroenvandijk08:06:30

yeah I can remove the no op function

jeroenvandijk08:06:00

I’ll finish this first and then look at the analyzer option. Not sure where to put it

borkdude08:06:07

also, I think it might be good to back up a bit and come up with a good problem statement, before working on a patch. what is the problem that we are solving here? things can take too long? what are the alternative solutions? the :timeout solution could also be an approach (instead of checking a counter, check the time passed since the start). so the order of things: 1) come up with a problem statement, write an issue, describe alternatives 2) explore alternatives using code 3) discuss 4) write PR, then discuss details of that PR

jeroenvandijk08:06:59

sure, I’ll do that. Now it’s just to get it out of my head

borkdude09:06:40

👍 which is ok too

borkdude09:06:48

where to add: in the expansion of the fn body in analyzer.cljc

borkdude09:06:04

I mean, that's also one possible approach which needs experimentation.

jeroenvandijk10:06:13

Ok created an issue and a PR 🙂 I’ll do it in the opposite order next time 🙃 https://github.com/borkdude/sci/issues/348#

borkdude10:06:08

Thanks. I'll take a look soon

jeroenvandijk10:06:54

Thank you. No rush. I’ll also see if I can move it to the analyzer namespace

borkdude10:06:52

what's the use case in the roam clone to run other people's code btw?

jeroenvandijk10:06:33

So Athens (and Roam) use Datascript as an underlying storage. This gives it a lot of flexibility, but it needs some post processing to be useable. This is where Sci comes in. Sci can do simple things like sort or formatting. It can however also be used to do more advanced things like math or controlling some kind of visualization of this data (no examples because it doesn’t exist yet). The power of Sci is that it can allow for arbitrary execution. This needs to be controlled in another way to make it reliable

borkdude19:06:04

@dominicm yes!

$ clj -A:bb-local -cp src -e "(require '[aero.core :as a]) (a/read-config (java.io.StringReader. \"{:y #or [#ref [:x] \\\"bar\\\"]}\"))"
WARNING: Unable to resolve "#ref [:x]" at [:y :form 0]
{:y "bar"}

dominicm19:06:40

@borkdude what changed?!

dominicm19:06:43

I was eating

dominicm19:06:43

I was eating

borkdude20:06:17

Woohoo, all aero tests are now passing in bb except for one minor thing which I know how to fix.

dominicm20:06:01

This is really exciting… I'm wondering what crazy things I can do with this. One idea is a CLI migration tool that will read my config.edn without paying a startup penalty!

dominicm20:06:14

Or maybe static analysis of my config file for linting purposes… so many ideas!

borkdude22:06:53

@dominicm for static analysis, you might also want to look at: https://github.com/babashka/pod-babashka-parcera

borkdude22:06:41

babashka v0.1.1: https://github.com/borkdude/babashka/blob/master/CHANGELOG.md#v011-2020-06-10 This release can now run clj-commons/camel-snake-kebab and juxt/aero as libraries.