Fork me on GitHub
#babashka
<
2020-11-12
>
Michaël Salihi10:11:28

Hi babashkackerz ! On the README, I read "Interop with commonly used classes (System, File, java.time., java.nio.)", so apart from these Java classes we can't use any other like org.freedesktop.dbus.DBusInterface for example, right?

borkdude10:11:27

That is correct, unless it is explicitly added in the babashka binary

Michaël Salihi10:11:30

I was wondering if we could do without Python for this kind of script for example. http://blog.muhuk.com/2015/09/09/displaying_clojure_test_results_in_taskbar.html

#!/usr/bin/env python

import dbus
import sys


def notify(summary, body='', app_name='', app_icon='',
        timeout=5000, actions=[], hints=[], replaces_id=0):
    _bus_name = 'org.freedesktop.Notifications'
    _object_path = '/org/freedesktop/Notifications'
    _interface_name = _bus_name

    session_bus = dbus.SessionBus()
    obj = session_bus.get_object(_bus_name, _object_path)
    interface = dbus.Interface(obj, _interface_name)
    interface.Notify(app_name, replaces_id, app_icon,
            summary, body, actions, hints, timeout)


if __name__ == '__main__':
    notify(*sys.argv[1:])

Michaël Salihi10:11:08

Thank you for confirmation @borkdude

borkdude10:11:07

@admin055 The ways to interact with classes not inside babashka are currently: - shelling out to another CLI that can do it - via a pod that is designed for this use case (which is also kind of a CLI but nicer integration code-wise)

borkdude10:11:01

@admin055 So if this is a frequent use case, you could invest in building a pod for it

Michaël Salihi10:11:00

@borkdude Yes I will. Thanks the CLI solution could a simplified first approach, thanks for the tip.

Michaël Salihi10:11:24

Yes pods are excellent feature, I had toying a little bit with pod-jaydeesimon-jsoup and it was nice ! I also plan to take some time soon to install the Graalvm toolchains and attempt to contribute with pods etc.

borkdude10:11:15

I want to simplify installing pods so you can put some config in a babashka.edn and the pod will be automatically downloaded and resolved by babashka.pods/load-pod (or maybe (babashka.pods/require 'foo.bar) since a pod management system could also list which namespaces belong to which pods). I hope that this will improve the adoption of pods. I already made something like this: https://github.com/borkdude/glam, but I think I will make something specific for pods in babashka to download the pods into a cache, etc.

Michaël Salihi10:11:00

Yeah, great idea! :+1:

holyjak11:11:53

@borkdude docker build --target BASE -t bb-builder . on the latest master fails for me during RUN ./script/uberjar due to > Retrieving borkdude/sci.impl.reflector/0.0.1-java11/sci.impl.reflector-0.0.1-java11.jar from clojars > OpenJDK 64-Bit Server VM warning: forcing TieredStopAtLevel to full optimization because JVMCI is enabled > Syntax error (ClassNotFoundException) compiling at (/tmp/form-init12305127242809915869.clj:1:74). > org.yaml.snakeyaml.error.YAMLException > ... > Error encountered performing task 'do' with profile(s): 'base,system,user,provided,dev,transit,reflection' Building outside docker seems OK. Any idea why / how to fix or troubleshoot? Is BABASHKA_FEATURE_YAML actually required?

borkdude11:11:07

This seems an error related to the feature flag. It will only try to load the yaml class if BABASHKA_FEATURE_YAML=true, or at least, it should.

holyjak11:11:08

I should mentioned I changed Dockerfile, setting all BABASHKA_FEATURE_* to false

borkdude11:11:50

@holyjak You can use BABASHKA_LEAN to accomplish the same thing. But you should set the flag for both the uberjar and compilation step

holyjak11:11:20

OK, thanks! BTW I have noticed https://github.com/borkdude/babashka/blob/master/script/compile#L73 includes most features but datascript and lanterna., Is that intentional or a mistake?

borkdude11:11:00

lanterna and datascript aren't enabled by default, so they are already set to false

borkdude11:11:16

Btw, the Github action JVM tests aren't working anymore, while everything on CircleCI works flawlessly. My experience with Github actions so far has been that it can be pretty flakey... If someone knows a solution, I'm open to suggestions.

holyjak11:11:59

But the same is true for JDBC and postgres yet they are on the list 🙂

borkdude11:11:33

That's probably because postgres and JDBC were part of babashka during one release, but I removed it in the next release

holyjak11:11:34

What if I want to build bb as lean as possible but with some selected features (namely csv, jdbc, oracle)? Then I cannot use LEAN. I could set all the other default features to false but then I will run into the problem ☝️ that yaml somehow is required anyway... But it is no big deal, I suppose they do not add that much to the size anyway.

borkdude11:11:58

@holyjak That is a bug which I cannot reproduce locally

borkdude11:11:33

We could make LEAN take into account variables that are set explicitly

holyjak11:11:18

I added

ENV BABASHKA_FEATURE_CORE_ASYNC=false
ENV BABASHKA_FEATURE_CSV=false
ENV BABASHKA_FEATURE_JAVA_NIO=false
ENV BABASHKA_FEATURE_JAVA_TIME=false
ENV BABAHSKA_FEATURE_TRANSIT=false
ENV BABASHKA_FEATURE_XML=false
ENV BABASHKA_FEATURE_YAML=false
ENV BABASHKA_FEATURE_HTTPKIT_CLIENT=false
ENV BABASHKA_FEATURE_HTTPKIT_SERVER=false
ENV BABASHKA_FEATURE_JDBC=false
ENV BABASHKA_FEATURE_POSTGRESQL=false
ENV BABASHKA_FEATURE_HSQLDB=false
ENV BABASHKA_FEATURE_ORACLEDB=false
ENV BABASHKA_FEATURE_DATASCRIPT=false
ENV BABASHKA_FEATURE_LANTERNA=false
to the Dockerfile before uberjar and run docker build .

borkdude11:11:13

let me try that locally

borkdude11:11:33

export BABASHKA_FEATURE_CORE_ASYNC=false
export BABASHKA_FEATURE_CSV=false
export BABASHKA_FEATURE_JAVA_NIO=false
export BABASHKA_FEATURE_JAVA_TIME=false
export BABAHSKA_FEATURE_TRANSIT=false
export BABASHKA_FEATURE_XML=false
export BABASHKA_FEATURE_YAML=false
export BABASHKA_FEATURE_HTTPKIT_CLIENT=false
export BABASHKA_FEATURE_HTTPKIT_SERVER=false
export BABASHKA_FEATURE_JDBC=false
export BABASHKA_FEATURE_POSTGRESQL=false
export BABASHKA_FEATURE_HSQLDB=false
export BABASHKA_FEATURE_ORACLEDB=false
export BABASHKA_FEATURE_DATASCRIPT=false
export BABASHKA_FEATURE_LANTERNA=false
script/uberjar && script/compile
No problems

borkdude11:11:16

Can you maybe print the value of BABASHKA_FEATURE_YAML in the script

holyjak11:11:36

ok, nevermind then. Thanks for trying! I will get back if it turns out to be a problem again.

borkdude11:11:41

I'm open for suggestions and PRs in this area

borkdude11:11:15

I'm also considering moving to the clojure CLI for compilation at one point, but that takes some refactoring to aliases for features

holyjak12:11:16

Any tips for troubleshooting when compile inside docker fails with

...
[bb:58]     (clinit):  20,613.49 ms,  2.03 GB
[bb:58]   (typeflow): 223,789.23 ms,  2.03 GB
[bb:58]    (objects): 936,796.65 ms,  2.03 GB
[bb:58]   (features):  21,151.56 ms,  2.03 GB
[bb:58]     analysis: 1,262,818.93 ms,  2.03 GB
[bb:58]     universe:  71,729.57 ms,  2.03 GB
Error: Image build request failed with exit status 137
com.oracle.svm.driver.NativeImage$NativeImageError: Image build request failed with exit status 137
	at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1558)
	at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1308)
	at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1269)
	at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1228)
	at com.oracle.svm.driver.NativeImage$JDK9Plus.main(NativeImage.java:1740)
The command '/bin/sh -c ./script/compile' returned a non-zero code: 137
? There are no details that would say why it failed 😞

borkdude12:11:57

This is an out of memory error. Bump BABASHKA_XMX by setting it to -J-Xmx8g or whatever amount you have available. Could also be a Docker setting

holyjak12:11:23

thanks a lot!!!

holyjak12:11:11

I have now run with --build-arg BABASHKA_XMX=-J-Xmx8g and still getting the error 😢

borkdude12:11:49

@holyjak Could it be that your docker doesn't have enough memory?

borkdude12:11:12

What os are you on?

holyjak12:11:26

Could be. OSXm 16GB mem in total.

borkdude12:11:57

Go to preferences and then resources

borkdude12:11:15

in your Docker Desktop

holyjak12:11:18

According to Activity Monitor Docker uses under 4GB so it would seem to be the problem

borkdude12:11:00

Note that you should not set BABASHKA_XMX higher than the total available memory minus some overhead, else you will get the same error

borkdude12:11:32

so if you set Docker memory to let's say 5GB, set BABASHKA_XMX to 4500m max

holyjak12:11:08

thanks a lot!

holyjak14:11:23

I have made some progress with -Xmx8575549440 : > Exception in thread "native-image pid watcher" java.lang.OutOfMemoryError: GC overhead limit exceeded 😢 I suspect that too much docker memory was on swap and thus too slow.

borkdude14:11:52

To how much have you set your docker memory

borkdude14:11:40

How much is 8 575.54944. Isn't that 8 megabytes?

borkdude14:11:08

Maybe you should supply a unit

borkdude14:11:16

Also, how are you providing BABASHKA_XMX? Can you post the entire Dockerfile setup?

borkdude14:11:38

You can also peek at the babashka docker build on circleci to get some ideas

holyjak14:11:41

this was printed in the stdout, I did run it with 8g

borkdude14:11:16

Can you try to compile without the Oracle driver, to see if that is the culprit?

holyjak14:11:17

I was running docker build --build-arg BABASHKA_XMX=-J-Xmx3g and in the Dockerfile I had ARG BABASHKA_XMX="-J-Xmx8g" I will wait until the osx build of bb finishes, which takes forever, then try again with Docker max 5g and build max 4500m, making sure Docker has enough memory available not to be swapped out

borkdude14:11:07

3g is too little

borkdude15:11:56

Somewhere between 4 and 5 gb is probably the minimal requirement

holyjak17:11:41

@borkdude I have managed to replicate the missing YAML error. When you run uberjar from https://github.com/borkdude/babashka/pull/641 with

BABASHKA_FEATURE_CSV=true
BABASHKA_FEATURE_JDBC=true
BABASHKA_FEATURE_ORACLEDB=true
BABASHKA_LEAN=true
BABASHKA_FEATURE_YAML=true
then it works but if I remove YAML it fails with ClassNotFoundException org.yaml.snakeyaml.error.YAMLException

borkdude18:11:07

@holyjak Is 641 good for merge?

holyjak17:11:56

BTW the loooong compile times and OOM errors seem to have been caused by me adding a few more functions and namespaces from next.jdbc.

holyjak17:11:56

BTW the loooong compile times and OOM errors seem to have been caused by me adding a few more functions and namespaces from next.jdbc.

borkdude18:11:00

It would be worth checking into which function(s) cause this. Sometimes it's related to a run-time resolve. I have forked my share of code to make it well suited for GraalVM

borkdude18:11:24

Thanks for looking into it. I was just seeing the PR

holyjak18:11:55

yes, I plan to try to find out which function(s) cause this

borkdude18:11:59

fwiw, the babashka pods compile in minutes on CircleCI. I'm not sure if those have an up to date next.jdbc version

borkdude19:11:35

I compared the next.jdbc version in the pods to babashka, it's the same

holyjak19:11:06

I upgraded it from.. 10 to... 13 AND added more stuff from it. Running experiments...

borkdude20:11:38

The functions in the babashka-sql-pods should work fine, I have tested those with GraalVM. So maybe you could start with those, to confirm you get the same results. See the binary build here: https://app.circleci.com/pipelines/github/babashka/babashka-sql-pods/83/workflows/c154eb40-ad4f-4c05-8c8f-07be42b4895e/jobs/251 A little over 1 minute.

borkdude20:11:18

Keep me posted :)

holyjak20:11:48

FYI building latest master with only ..JDBC=true takes 451s, JDBC+Oracle takes 581s, with no extra features 421s, LEAN fails with ClassNotFoundException YAMLException

borkdude20:11:35

This is in Docker right?

holyjak20:11:57

no, on my 4y old maskin