Fork me on GitHub
#babashka
<
2020-08-17
>
epransky06:08:56

i've got a question about expected behavior. lets say i have a jar with a static resource foo.edn in resources folder. when i add this jar to my classpath in babashka, i should be able to access this resource by simply (io/resource "foo.edn") , right? this works in a standard clojure context, but i keep getting nil when i try this with babashka. am i missing something?

borkdude06:08:09

You should add resources explicitly to the classpath

borkdude06:08:35

Oh it’s from a jar? That should work

borkdude06:08:19

I’ll look into it in an hour or so

epransky06:08:06

thanks ill try to put together a proper replication

borkdude07:08:50

I think it’s a bug that’s been fixed on master

borkdude07:08:14

Also see the recently closed issues

borkdude07:08:05

You can go to #babashka_circleci_builds and try a build from the master branch

epransky07:08:39

:man-facepalming:yup its right there in the issues. didnt check. thanks for the quick response. ill check out master branch and see if it solves mu usecase

jeroenvandijk10:08:52

What kind of use cases do you see? Is it being able to run something from both bb and java? Is it a replacement for depstar?

borkdude10:08:31

The use case is if you want to package up a project and distribute it over multiple servers for example. Or just distribute it to users: here's the uberjar, run it with bb, no need to clone a project.

borkdude10:08:44

Uberjar can contain other assets like images, etc. as well

jeroenvandijk10:08:58

ah yeah i understand now. Thanks!

borkdude10:08:36

It uses a fork of depstar, but it's not a replacement. Babashka won't be able to do AOT of clojure sources.

jeroenvandijk10:08:14

So it is some kind of uber-uberscript :)

borkdude10:08:48

It's an alternative to uberscript. E.g. when your project uses io/resource it will still work with uberjar, but not with uberscript. For simple scripts uberscript is still a nice option

jeroenvandijk10:08:04

I agree. Great addition 🙂

hugod11:08:37

@borkdude Thanks for fixing the uberscript namespace ordering.

hugod12:08:04

I have a project which runs fine under bb, and which i can build a graalvm image from, but I can’t get it to uberscript. When running bb --uberscript it throws a Could not resolve symbol exception when loading one of the namespaces. The namespace is being required by a ns form that is three levels deep. The symbol is aero/read-config, but I don’t think that is important, it is just the first external symbol referenced in the namespace. Does uberscript affect how require works?

borkdude12:08:57

Hmm, yes, it only processes top-level requires as of now

borkdude12:08:46

During the uberscript build, it only looks at the top-level requires and the rest is effectively a no-op

hugod12:08:57

ok, so I’ll stop trying then 🙂

borkdude12:08:32

Maybe it's something that can be fixed. Not sure. Repro would still be welcome

borkdude12:08:16

Or we can document it as a caveat / known limitation and recommend the uberjar approach as an alternative

borkdude12:08:06

I wonder what your ns form three levels deep looks like

borkdude12:08:13

@hugod I notice --uberscript ignores a (require ...) form that is not in an ns form, I think that can still be fixed, if that require uses a literal symbol and is top-level

hugod12:08:27

I’m just using ns forms. I added a prn in load-fn and see it loading all the required namespaces. I’m guessing it just doesn’t evaluate them.

borkdude12:08:48

That's right. uberscript doesn't evaluate things, since it would depend on the runtime script arguments which namespaces would be loaded, so it's kind of a form of static analysis. It still uses the interpreter to run through the ns forms, but turns off function evaluation basically

borkdude12:08:24

We can change that with a setting maybe

borkdude12:08:43

You can try to enable function evaluation, to experiment

hugod12:08:46

umm, so why is the defn form it contains getting evaluated? (and causing the unresolved symbol exception)

borkdude12:08:02

Is that during the uberscript build?

borkdude12:08:04

I need some kind of repro, else it's only a guess

hugod12:08:49

Would putting up the code somewhere suffice?

borkdude12:08:02

You can make a private Github repo if you want, or public. What's best for you

hugod12:08:45

Run deps.exe -m makejack.main uberscript in https://github.com/hugoduncan/makejack/tree/wip (the project is very much an outline at the moment)

hugod12:08:32

It will print the bb uberscript invocation

borkdude12:08:09

That's something that should not occur. I'll take a look at this later this week, if that's ok

hugod12:08:52

Thanks for looking at all 🙂

borkdude12:08:45

What should one put into mj.edn ?

borkdude12:08:55

Just so it will work when invoking with the main method?

hugod13:08:48

I’m not sure I understand the question

borkdude13:08:12

Say I turned on evaluation during the uberscript and I ran the invocation that was printed, I'm getting something like:

{:options {}, :arguments [], :summary   -h, --help, :errors nil}
Failed to read makejack file mj.edn:  clojure.lang.ExceptionInfo: Config error on line null {:line nil}
clojure.lang.ExceptionInfo: Config error on line null [at line 33, column 7]

hugod13:08:49

That would occur if aero failed to read mj.edn, or we failed to merge the defaults It should be fine with an empty map in mj.edn

borkdude13:08:17

Hmm, I had an empty map, but still got this error

hugod14:08:29

so it prints bb -cp … -m makejack.main --uberscript bin/mj - is that what you’re running in the makejack project root? makejack.main is makejack’s main namespace. which is what it tries to uberscript. makejack runs the command after printing it.

borkdude14:08:33

yes. that's what I tried to run. I got rid of a couple of checks to that prevents eval during uberscript, but then I got into this error

borkdude14:08:45

I created mj.edn with {}

hugod14:08:02

I seem to be missing something. It didn’t work with the mj.edn in the tree? https://github.com/hugoduncan/makejack/blob/wip/mj.edn

hugod14:08:54

or are you running it outside the project root?

borkdude14:08:49

Within the root. Never mind, I hadn't noticed that there was an mj.edn in the root already

borkdude16:08:19

So this is the root cause of the problem, I'll have to ponder a little bit about a solution. Suggestions welcome

borkdude16:08:02

I'll also post this in the issue

borkdude16:08:34

Maybe the best solution is to just ignore these kinds of errors during the uberscript build

borkdude16:08:46

Since what counts is the top level namespace mostly

borkdude16:08:42

@hugod You can try the uberscript-ignore-unresolved branch of babashka. This will ignore unresolved symbols during analysis.

hugod17:08:00

that branch seems to work nicely

borkdude17:08:42

Excellent. I'll make a repro in the unit tests and then push it to master

borkdude18:08:05

Thanks for your criterium lib btw.

borkdude18:08:42

I'm considering making uberscript even more "accepting": https://github.com/borkdude/babashka/issues/538 It doesn't need to do all the analysis, just skip over the irrelevant forms

borkdude20:08:07

@hugod A'ight. I now made uberscript as accepting as possible. It will only inspect ns forms and ignore all the rest in the code.

borkdude20:08:30

(So potentially this could even be used for uberscripting JVM Clojure libs that don't even run with bb, as a side effect).

borkdude20:08:36

This is on master now. Testing would be good.

borkdude21:08:54

OK, now also top-level require is supported. I added this to the README: > Caveat: building uberscripts works by running top-level ns and require > forms. The rest of the code is not evaluated. Code that relies on dynamic > requires may not work in an uberscript.

tim.kutscha19:08:04

Hello 🙂 I am relatively new here and playing around a little. I would like to use babashka to get some clojure into my work life. I want to build a cli that connects to a datahike db and to different web apis. Is it currently possible to use babashka to connect to datahike? I tested it by setting the classpath and it did not seem to work, but this could also be because of my inexperience. Currently there exists no pod for datahike, maybe I need to wait for it (or build it). I also appreciate any resources about cli development in babashka / clojure that you came across and found to be helpful 🙂 Any help is much appreciated 🙃

borkdude19:08:01

Hi! Datahike is a pretty complex project. I've tried to compile it with GraalVM once but that didn't work. So a pod won't work because of that reason.

borkdude19:08:40

Also the bb repo has an example directory with some examples.

borkdude19:08:33

If datahike exposes a REST/HTTP API you can talk to it that way maybe

borkdude19:08:07

Babashka does have a feature flag for DataScript which does work with GraalVM

tim.kutscha19:08:08

Wow that was fast, dankjewel 🙂

tim.kutscha19:08:48

I think I am going to play around a little with datascript. My data-persistence-needs are so minimal, that it would be okay if I could just write/read the whole db to a file and replace it after an update.

borkdude19:08:22

OK, in that case you'll need to compile babashka yourself with the appropriate feature flag: https://github.com/borkdude/babashka/blob/master/doc/build.md#feature-flags Datascript could alternatively be turned into a pod, which might be a nice project if you're in for it

tim.kutscha20:08:24

I am executing script/compile. I did not explicitly set the max heap size. The process is now running for about 15 minutes, is that normal, or should I have set the max heap size?

borkdude20:08:51

yes. DataScript requires a lot of memory, so setting BABASHKA_XMX to -J-Xmx8g or more is recommended

borkdude20:08:13

Usually with such dependencies there are a few specific things which trigger a lot of memory usage. It could be worthwhile figuring out which parts that are by starting small and adding more incrementally.

tim.kutscha21:08:53

It took some time and with the heap to 6500m it worked. Now I am getting an error in some of the datascript examples. When I try to execute the following with babashka:

(d/q '[:find ?k ?x
       :in [[?k [?min ?max]] ...] ?range
       :where [(?range ?min ?max) [?x ...]]
       [(even? ?x)]]
     {:a [1 7], :b [2 4]}
     range)
I get this error:
java.lang.IllegalArgumentException: Class clojure.lang.Keyword[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection [at...
While other examples work fine. Did you encounter similar errors before? (Google is not much of a help)

borkdude21:08:56

Does the GraalVM error message also point to some location in the source of DataScript?

tim.kutscha21:08:38

no, this is the full error message:

java.lang.IllegalArgumentException: Class clojure.lang.Keyword[] is instantiated reflectively but was never registered. Register the class by using org.graalvm.nativeimage.hosted.RuntimeReflection [at /Users/tim/projects/todo-2/src/my/core.clj, line 32, column 1]

tim.kutscha21:08:01

executed with: ~/projects/babashka/bb -f src/my/core.clj

tim.kutscha21:08:30

contents of the file core.clj:

(require '[datascript.core :as d])

(d/q '[:find ?k ?x
       :in [[?k [?min ?max]] ...] ?range
       :where [(?range ?min ?max) [?x ...]]
       [(even? ?x)]]
     {:a [1 7], :b [2 4]}
     range)
(and a bunch of commented code

tim.kutscha21:08:32

whereas this example works without error:

(require '[datascript.core :as d])

(let [schema {:aka {:db/cardinality :db.cardinality/many}}
      conn (d/create-conn schema)]
  (d/transact! conn [{:db/id -1
                      :name  "Maksim"
                      :age   45
                      :aka   ["Max Otto von Stierlitz", "Jack Ryan"]}])
  (d/q '[:find ?n ?a
         :where [?e :age 45]
         [?e :name ?n]
         [?e :aka ?a]]
       @conn))

borkdude21:08:01

The problem is not so much your code, but how GraalVM works. If Clojure code needs reflection then GraalVM can't analyze this ahead of time and you need to add some reflection config. See https://github.com/borkdude/babashka/blob/master/src/babashka/impl/classes.clj. So there are two solutions: Either find out where the reflection happens in DataScript and prevent it or add the appropriate config.

borkdude21:08:54

Preventing reflection would be the better solution as it results in smaller more efficient binaries.

borkdude21:08:12

Note that GraalVM wants config for the array, not just the keyword class. There are a few examples in that file for other array types.

tim.kutscha22:08:39

Could you give me a pointer on where I can find configuration in the file for other array types? If the keyword class is only inserted when features/datascript? -> true, then the standard binary size would not be affected.

borkdude09:08:56

The [L is Java's way of indicating an array type