Fork me on GitHub
#graalvm
<
2021-03-11
>
Stuart11:03:00

I'm trying to do the hello-world tutorial on clj-graal-docs on windows and hitting issues. I have my folder structure and the main.clj and deps.edn. It compiles when I do

clojure -M -e "(compile 'hello-world.main)"
Then it says verify it works on the JVM with
$ java -cp $(clojure -Spath):classes hello_world.main
Hello world!
When I run that java command I get an error.
Error: Could not find or load main class :classes
Caused by: java.lang.ClassNotFoundException: :classes
The output to the classes folder it asked me to create was this: I'm not familiar enough with the CLI tools to know if that command to compile is wrong, or I'm doing something else stupid.

borkdude11:03:21

@qmstuart $(clojure -Spath) is bash syntax

Stuart11:03:31

ah ok, doh!

borkdude11:03:58

I do have some windows examples

borkdude11:03:01

let me check

borkdude11:03:16

@qmstuart For Windows (and in general now) I tend to first build an uberjar and then feed that uberjar to graalvm. This also takes care of "too long" classpath issues on Windows

Stuart11:03:06

thank you!

borkdude11:03:15

^ @lee this is what I was getting at with the Windows classpath / uberjar problem, it's one of the two issues

lread11:03:09

thanks, I think I can work this into a tip

borkdude11:03:49

or we can add a Windows-friendly hello world in there

wombawomba11:03:20

So I'm compiling a command-line app using native-image, and the resulting binary weighs in at a whopping 94 MB. I ran native-image with -H:DashboardDump=dashboard -H:+DashboardCode to try to tell where all this weight is coming from (`-H:+DashboardAll` crashes with a message about deadlock), but looking at the result at https://www.graalvm.org/docs/tools/dashboard the total size adds up to only 37 MB. Any idea what's going on here?

borkdude11:03:45

@wombawomba There are a few things which can bloat the binary size in clojure projects. I have seen this happen mostly with runtime requires/requiring-resolve usages

borkdude11:03:57

Usually I find out about this by starting small and then adding more stuff, to find out what triggers it

wombawomba11:03:25

@borkdude any ideas on how to start if I want to try something like that? Currently my build command is native-image --allow-incomplete-classpath --enable-http --enable-https --initialize-at-build-time --initialize-at-run-time=org.jline.terminal.impl.jansi.win.JansiWinSysTerminal,org.jline.terminal.impl.jna.win.JnaWinSysTerminal --no-fallback --no-server --report-unsupported-elements-at-runtime --verbose (the --inititalize-at-run-time arg seem to be required for https://github.com/jline/jline3)

borkdude11:03:13

The first start is just print hello world from your main.

borkdude11:03:22

And then add one function to it at a time

Stuart11:03:18

@borkdude your video on graalvm native hello world is good and helpful. Thanks for making these little videos 🙂

🙏 1
Stuart12:03:55

Error: Native-image building on Windows currently only supports target architecture: AMD64 (x86 unsupported)
Error: To prevent native-toolchain checking provide command-line option -H:-CheckToolchain
com.oracle.svm.core.util.UserError$UserException: Native-image building on Windows currently only supports target architecture: AMD64 (x86 unsupported)
Where can I specify x86 v x64 ?

Stuart12:03:48

or maybe this a consequence of teh VS I have installed, and therefore the cl.exe ?

Stuart12:03:36

OK, you need to use the x64 developer command prompt to run graal-vm

Stuart12:03:23

it works, i have a native windows.exe 😄

Stuart12:03:48

thanks for all the help 👍

thegeez15:03:40

What kind of file sizes do you folks see for binaries created with native-image? I always end up with sizes between 75-90mb (for clj+jetty+postgres+jdbc).

thegeez16:03:01

This 75-90mb is from uberjars around 10mb 🙂

borkdude16:03:16

@thegeez Hello world is around 10mb. With sci (interpreter) not much more. Adding postgres to babashka adds around 3mb, not much. But I think the incremental size depends on what you were already using before.

borkdude16:03:33

Babashka itself is now 70mb on macos. I curate every lib that goes in to make sure the binary size doesn't blow up

borkdude16:03:15

There are a few things that can trigger binary size to blow up, such as runtime requires/resolve

thegeez16:03:01

Thanks, then it might be worth it to investigate getting lower sizes for my case. Thus far I always assumed 60mb as the lowest bar

borkdude16:03:02

E.g. clojure.pprint has some usage of find-var inside of it. I patched that to make the vars known at at compile time. This took care of a couple of dozen mbs.

borkdude16:03:53

There is also https://github.com/borkdude/dynaload which gives better binary size than the clojure.spec dynaload

borkdude16:03:48

@thegeez babashka has httpkit built in. this only added 1.5mb to the binary including the http client

borkdude16:03:06

But adding the full ring stack will probably bloat stuff again due to some require at runtime somewhere

borkdude16:03:24

This has already been patched by someone who wanted to add ring to bb

thegeez16:03:22

Alright I guess I'll need to do a day of trail-and-error compilation to try to get down to those numbers. Too bad each native-image compilation takes around 6 mins on my machine

borkdude16:03:06

Yes, this can be a painful process...

lukasz16:03:44

I'm genuinely curios about the concern around binaries size - if it's was 0.5GB I'd be worried, but 90MB feels like nothing nowadays. It kinda makes sense for bb in some way, then again node_modules in one of our apps is 650MB....

thegeez16:03:12

It is not a problem for my use-case, which is aws lambda, the zip size is 20mb. But if there's any further improvement possible I'll gladly take it. Because deploy is compilation (~6min) -> upload 20mb

borkdude16:03:26

@thegeez Yeah, maybe not worry about this. 90mb sounds fine for an "app". clojure-lsp is also this size. Thinking about it though: both projects use next.jdbc. After upgrading recently I saw the binary size of babashka pods grow to 50mb, so it may be explained from this lib usage as well. Anyway, I agree with @lukaszkorecki: if this is something you use in private, then it's no big deal, if it's used by the masses, maybe more, but maybe still not a big deal.

thegeez17:03:36

I'm already happy that I don't need to change to use for instance ClojureScript on nodejs to use lambda, so far native-image is great already.

thegeez17:03:22

But it would also be awesome if I could email around the final binaries 🙂

borkdude17:03:55

If you're using amazon, put it in a zipfile on s3? ;)

lukasz17:03:25

Ah, I was not aware of the lambda file limit - we deploy Clojure on JVM in lambda and the uberjar size is <10MB so we're well below that. That said, the newly released Lambda + container images effectively eradicates that issue.

thegeez17:03:43

It's zipfile on s3 right now, that won't change even with a smaller size

borkdude17:03:52

I don't think thegeez said that aws lambda has a binary size limit?

thegeez17:03:01

It's nice to have, rather than must have. Which is good enough already

borkdude17:03:56

@thegeez What is the use case of a webapp on aws lambda? afaik you don't need a webserver to process lambda invocations right?

thegeez17:03:53

It's API Gateway -> Lambda bootstrap custom runtime. The runtime handles http calls containing lambda events from the API Gateway which gets http calls from browsers

thegeez17:03:16

I think all lambda request handler libraries/frameworks expose a web handler to receive requests, but perhaps there are other ways

ghadi17:03:24

I’m working on a custom Lambda runtime at Cognitect, interested in your use cases

lukasz17:03:52

I prototyped bb+Lambda+container image, doesn't get easier than this: https://gist.github.com/lukaszkorecki/a1fe27bf08f9b98e9def9da4bcb3264e

lukasz17:03:00

no need for custom runtimes etc

thegeez17:03:16

My use case is web-app with cljs front-end for real-time collab over API Gateway websockets, web app is API Gateway HTTP api to lambda talking to Postgres rds database. This is a prototype atm

borkdude17:03:12

So the webserver is short-lived, as soon as the person disconnects, the lambda can go to sleep again, let's say?

thegeez17:03:35

A single lambda can handle consecutive reqs, but may disappear at any time (in my setup; there's no full webserver handling concurrent requests)

borkdude17:03:20

right, so if there are no requests for a long time, you don't pay amazon. that's nice for low traffic websites/apps

thegeez17:03:08

I saw a tweet that describes serverless as "pay per request" architecture, that fits I think

borkdude17:03:17

and also for very high traffic, since it scales automatically =)

thegeez17:03:42

My goal is more lowest devops/most convenient devops required. I don't do well managing hardware

borkdude17:03:06

You're still managing hardware though ;)

borkdude17:03:16

Thanks for sharing your setup.

thegeez17:03:35

Yes serverless is a lie 🙂

thegeez17:03:46

Thanks for sharing your numbers

borkdude17:03:07

FYI, we moved to AWS now at Doctor Evidence, no more manual hardware fixes ;)

thegeez18:03:11

Correction to what I said above, there's no webserver in the native-image. It's browser -req-> API Gateway <-req- native app. So the native app on lambda only needs an http client.

borkdude18:03:13

Then the image could be fairly small (depending on what else it does). Which client are you using? Clj-http is known to bloat the binary. Http-kit is much better there

borkdude18:03:37

Does the image still do the db stuff?

thegeez18:03:51

http through clj-http-lite, the jdbc.next+postgres is still there

borkdude19:03:14

yes, but don't use a wrapper like hato, it has even worse bloat due to all sorts of runtime requires

borkdude19:03:37

Perhaps https://github.com/exoscale/telex fairs better, since it's really minimal

lukasz19:03:25

Underneath your lambda never runs a web server - it GETs event data via a private metadata endpoint and POSTs results to it. So in Lambda runtime/SDK there usually is some http client included.

dangercoder21:03:03

has anyone tried to compile a project depending on https://github.com/igrishaev/etaoin to a native-executable?

dangercoder21:03:18

clj-http giving me some issues

borkdude21:03:12

This is a better example of the usage: https://github.com/babashka/pod-registry/blob/master/examples/etaoin.clj I should really update the README

borkdude21:03:58

@jarvinenemil I'm looking back now. I made a fork of etaoin: https://github.com/borkdude/etaoin-graal and published it to clojars: borkdude/etaoin-graal "0.3.7-alpha.1"

😍 1
dangercoder21:03:02

@borkdude I tried https://github.com/babashka/pod-babashka-etaoin but I will have to make pull-request filling in a few functions that I need (such as etaoin.api/get-element-attr . Working on it

borkdude21:03:00

@jarvinenemil I think we'll have to update it with etaoin proper as well

borkdude21:03:12

it's 100 commits or so behind I see

borkdude21:03:16

I mean etaoin-graal. I will update it now

dangercoder21:03:25

I figured, cool, thanks!

dangercoder21:03:53

Really awesome that you had forked etaoin and made a graal-vm compatible version. I was just about to do the same

borkdude22:03:38

@jarvinenemil Published upgraded version as: borkdude/etaoin-graal 0.4.2-deb92aaa21c59c52521f42c0177bbce9ac10a0e2

borkdude22:03:52

Now I'll update the version at pod-babashka-etaoin and I'll await your PR

👍 1
dangercoder22:03:00

Heads up if anyone is using --enable-all-security-services ,

The --enable-all-security-services option is now deprecated and it will be removed in a future release.
https://www.graalvm.org/reference-manual/native-image/JCASecurityServices/

borkdude22:03:45

It will be deprecated in favor of automatically detecting by static analysis