In this Docker image, do you intend to run an application with dependencies in bb.edn? I think the best way to do this is to produce an uberjar first and then run the uberjar

such that no dependencies have to be fetched at runtime

you can do this with:

bb uberjar

Thanks. In this case I'm fetching the dependencies when building the docker image (the user triggers build), and using bb prepare. The problem is that the clojure tools setup always run when there are changes to the source code files of the user (triggered with COPY). I want this setup to happen only in the first build, thus a step before the COPY Made available the code in a branch for better context: https://github.com/ccfontes/faas-bb/blob/setup-clj-tools/template/bb/Dockerfile#L24 To be fair, this only happens when function/bb.edn changes, which is a bit more rare than other source files.

bb prepare should also do Clojure tools not yet in expected location: when you are building the docker image. But you have to preserve the ~/.deps.clj directory into the image for it not to happen again

sure, it's preserving

then what is the issue?

I don't see how the clojure tools setup would run multiple times, even with bb.edn changes

unless you're not preserving the clojure tools directory

because of COPY. it busts the cache

what cache?

docker's

It sounds like this is a docker problem, not a bb problem then?

if you can make a repro Dockerfile I could have another look, not sure what to do about it now

maybe it helps to know that the tools dir is configurable: DEPS_CLJ_TOOLS_DIR

It's here. https://github.com/ccfontes/faas-bb/blob/setup-clj-tools/template/bb/Dockerfile#L24 Thanks

If I could force the clj tools setup before COPY, then it would solve my problem.

running bb prepare would force that

I mean, I could some hacking I guess with a dummy project

do you mean you don't have a bb.edn?

yes, I don't have bb.edn earlier than COPY

then you can do it like this:

bb -Sdeps '{:deps {some-fake-deps}}` prepare

oh jeez of course. genious!

using babashka.process, you can do an exec call (which is a true OS exec call) which might help. not sure about the rest of the trade-offs

that's awesome and another question that I had, so thanks I was just going to use https://clojuredocs.org/clojure.java.shell/sh but how do I get the abs path of my binary program at runtime

clojure.java.shell does not let you do an exec call btw it will create another child process

of course

This is how you can get the binary name of the invoked process:

$ bb -e '(-> (java.lang.ProcessHandle/current) .info .command .get)'
"/Users/borkdude/dev/babashka/bb"

thanks! I'll try it.

it's preferable for me for the binary to call or exec the bash after it does the CLI option parsing, so I don't have to repeat that in bash

I'm getting

Method info on class java.lang.ProcessHandleImpl not allowed!

(sci/eval-string "(-> (java.lang.ProcessHandle/current) .info .command .get)"
  {:classes {'java.lang.ProcessHandle java.lang.ProcessHandle}})

in the repl, not with graal...

check the bb classes.clj file, it has a function called public-class which cases the Impl class back to the public class

why do you need to do this in SCI btw?

You're right. I don't need this in SCI for what I'm trying to do here. Thanks for noticing that!

I graal compiled ys with it in there and it gets the install path right. \o/

when not graal compiled it gives /usr/lib/jvm/java-17-openjdk-amd64/bin/java though

I'll probably just default to a PATH lookup in that case since I want to be able to test without a full native-image compile.

@U04V15CAJ I ended up going with babashka.process to get https://yamlscript.org/posts/advent-2023/dec-20/ machinery working. https://github.com/yaml/yamlscript/commit/9d0c2dac87aa013fdc12e0b1f7ab26376bf9824f#diff-320f19934fd7d03ab26a1a99137a67e759faa60c287dc3ef144ca56b24fb5781R171-R174

process is very nicely done.

None of the jaxax.xml classes are currently exposed in babashka, but you can use clojure.data.xml instead

which is already built-in

I'll have a look later if we can exposes those javax.xml classes, but I doubt that eximia will be faster when running in bb due to interpretation overhead, but it's worth testing. Do you have any examples of big XML files + clojure.data.xml + eximia example that compares speeds?

if you can prepare a repository that I can clone locally and just run with both, that would be a nice start

Yes, I do, I have tried it already with data.xml and it is slower parsing and specially updating. Currently they are company files, but I'll see if I can generate some similar ones. XML files are around several MB.

if you want to experiment yourself, you can add the javax.xml classes here: https://github.com/babashka/babashka/blob/master/src/babashka/impl/classes.clj and then compile bb locally

build instructions are here: https://github.com/babashka/babashka/blob/master/doc/build.md

OK, thanks, I'll try

Would it be as simple as adding it to https://github.com/babashka/babashka/blob/53d16eb9a02ee7dbc39243ff7d2b73049f7fda63/src/babashka/impl/classes.clj#L254? or does it involve more?

no, that's it

you could try to first run with the JVM-version of bb to see if it works:

clj -M:babashka/dev my_script.clj

Hi again, I added the javax.xml classes to babashka as you suggested. It compiled with no problems. Now, when I run the script to read an xml file I don't get the Unable to resolve classname error anymore, but now I get the following error:

Method setProperty on class com.sun.xml.internal.stream.XMLInputFactoryImpl not allowed!

yes, let me check

Add to the :public-fn casts:

(instance? javax.xml.stream.XMLInputFactory v)
javax.xml.stream.XMLInputFactory

:public-class I mean:

add it near the bottom

OK, I'll try

Yes!, the error message disappeared, but now I get the error:

Method close on class com.sun.org.apache.xerces.internal.impl.XMLStreamReaderImpl not allowed!

you got it

It worked! Thanks!! I will add and test the functions to write xml files, I might have to add some more classes to the :public-class.

cool

Trying to write an XML file, I got an error:

Type:     java.lang.NoSuchFieldException
Message:  close

I Does write the xml file but stops almost at the end, its incomplete and invalid xml, the file is not that big (541KB).

This is a reflection issue on something in a with-open form

I think

at least .close is called on something but the .close method isn't available in the native image . perhaps you can track down which class this is

It shows this in the error:

----- Context ------------------------------------------------------------------
25: (defn save-xml
26:   "Saves the xml to a file"
27:   [xml path]
28:   (let [opts {:tag-fn exml/keyword->qname}]
29:     (exml/write xml (io/output-stream path) opts)))
        ^--- close

https://github.com/nilern/Eximia/blob/d57c6268eb3ee4a5c3f5baad7336a02693662ae0/src/eximia/core.clj#L195

let's see if I can reproduce that

This does work:

$ bb -e '(with-open [out (java.io.StringWriter.)] (.write out "foo") (str out))'
"foo"

How about with a long file like more than 500KB?

don't know. I think it would be useful to clone eximia locally and insert some printlns etc to see where exactly this is happening and then make a repro to fix

I debugged eximia, and putting a (.close out) after https://github.com/nilern/Eximia/blob/d57c6268eb3ee4a5c3f5baad7336a02693662ae0/src/eximia/core.clj#L135 generates the whole xml file, but the error remains.

and what is the type of out here? can you print (prn (type out))?

Seams to be: com.sun.xml.internal.stream.writers.XMLStreamWriterImpl

and you have javax.xml.stream.XMLStreamWriter in the class map + public-class fn?

Try to put this before the Map conversion

Before `(instance? java.util.Map v) java.util.Map`

since .close will be called on a Map if you do the cast later

got it?

I think so, I'll try.

Yes, that worked! Again thanks! Reading and writing xml files with eximia!

🎉

Now, is it faster than clojure.xml now?

Unfortunately no :(, here are some quick benchmarks: There was no transformation to the xml it only wrote back the xml that was read from the file. reading eximia babashka file size: 541K "Elapsed time: 236.811919 msecs" reading clojure.data.xml babashka file size: 541K "Elapsed time: 0.562374 msecs" writing eximia babashka file size: 541K "Elapsed time: 374.571785 msecs" writing clojure.data.xml babashka file size: 541K "Elapsed time: 117.879349 msecs" --- reading eximia babashka file size: 14M "Elapsed time: 5233.275265 msecs" reading clojure.data.xml babashka file size: 14M "Elapsed time: 0.565281 msecs" writing eximia babashka file size: 14M "Elapsed time: 8450.353057 msecs" writing clojure.data.xml babashka file size: 14M "Elapsed time: 3803.034851 msecs" --- reading eximia clj file size: 541K "Elapsed time: 49.630511 msecs" reading clojure.data.xml clj file size: 541K "Elapsed time: 19.460798 msecs" writing eximia clj file size: 541K "Elapsed time: 87.266801 msecs" writing clojure.data.xml clj file size: 541K "Elapsed time: 432.58637 msecs" --- reading eximia clj file size: 14M "Elapsed time: 284.419659 msecs" reading clojure.data.xml clj file size: 14M "Elapsed time: 19.567271 msecs" writing eximia clj file size: 14M "Elapsed time: 497.282376 msecs" writing clojure.data.xml clj file size: 14M "Elapsed time: 2173.910879 msecs"

This is what I suspected, the interpretation overhead is simply too big

Well, it was a nice experiment. I think I'll try my scripts with babashka and clojure.data.xml. Thanks for your time!

👍

perhaps it's useful to make a PR with it, then we can close it, but at least we have the code how to add those xml classes in case we do need it at some point

Here is a git patch with the changes I made, if it's quicker:

diff --git a/src/babashka/impl/classes.clj b/src/babashka/impl/classes.clj
index 5c2e808d..25998d78 100644
--- a/src/babashka/impl/classes.clj
+++ b/src/babashka/impl/classes.clj
@@ -553,7 +553,16 @@
                 ~(symbol "[Ljava.util.regex.Pattern;")
                 ~(symbol "[Lclojure.core$range;")])
           ~@(when features/yaml? '[org.yaml.snakeyaml.error.YAMLException])
-          ~@(when features/hsqldb? '[org.hsqldb.jdbcDriver])]
+          ~@(when features/hsqldb? '[org.hsqldb.jdbcDriver])
+          ;; for eximia (xml parser)
+          javax.xml.stream.XMLInputFactory
+          javax.xml.stream.XMLStreamReader
+          javax.xml.stream.XMLStreamWriter
+          javax.xml.stream.XMLStreamConstants
+          javax.xml.stream.XMLOutputFactory
+          javax.xml.namespace.QName
+          javax.xml.XMLConstants
+          ]
     :constructors [clojure.lang.Delay
                    clojure.lang.LineNumberingPushbackReader
                    java.io.EOFException]
@@ -675,6 +684,15 @@
                          java.lang.ProcessHandle
                          (instance? java.lang.ProcessHandle$Info v)
                          java.lang.ProcessHandle$Info
+                          ;; for eximia (xml parser)
+                         (instance? javax.xml.stream.XMLInputFactory v)
+                         javax.xml.stream.XMLInputFactory
+                         (instance? javax.xml.stream.XMLStreamReader v)
+                         javax.xml.stream.XMLStreamReader
+                         (instance? javax.xml.stream.XMLOutputFactory v)
+                         javax.xml.stream.XMLOutputFactory
+                         (instance? javax.xml.stream.XMLStreamWriter v)
+                         javax.xml.stream.XMLStreamWriter
                          ;; added for calling .put on .environment from ProcessBuilder
                          (instance? java.util.Map v)
                          java.util.Map

sure. that will do, thanks :)

https://github.com/babashka/babashka/issues/1651