Fork me on GitHub
#holy-lambda
<
2021-10-19
>
steveb8n05:10:29

Q: I just added Timbre for logging and now my HL native:executable is exceeding CI memory limits. Any tricks for getting around this?

steveb8n05:10:16

When run on local machine it uses just over 6Gb. The CI system claims to provide 8Gb but I suppose there is some overhead

steveb8n05:10:41

probably because the Docker daemon needs the other 2Gb

Karol Wójcik05:10:20

Try setting maximum heap size up to 5-6gb.

Karol Wójcik05:10:53

Sorry. It should be -J-Xmx5G set in native-image-args

steveb8n05:10:27

ok. will do. looks like I can reduce the docker daemon mem also

steveb8n05:10:32

will try both

Karol Wójcik05:10:39

Don’t do it by limiting memory size of your daemon. Native image does not utilize full all the available memory. Better to set memory limit to native-image and check if the argument is there by using —verbose

steveb8n06:10:17

0.5.0 fixed it. used about 1Gb less memory

Karol Wójcik06:10:25

New version will be compatible with 0.5.0, but also slightly faster and more memory efficient. ;)

steveb8n05:10:44

I’m also curious if anyone has successfully used Timbre with HL? I suspect it uses reflection in some layers

steveb8n05:10:17

SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder”. SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. WARNING: Could not register reflection metadata for org.apache.commons.logging.impl.Log4JLogger. Reason: java.lang.NoClassDefFoundError: org/apache/log4j/Priority. Warning: class initialization of class org.apache.commons.logging.impl.Log4JLogger failed with exception java.lang.NoClassDefFoundError: org/apache/log4j/Priority. This class will be initialized at run time because option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=org.apache.commons.logging.impl.Log4JLogger to explicitly request delayed initialization of this class. [

Karol Wójcik05:10:05

Btw could you remove Log4JLogger from the reflection-config.json?

steveb8n05:10:57

I’ll look into that once mem is fixed

steveb8n23:10:52

hmm, I need a little help with this config. I think you mean that I should use the props from the native-image.properties file in that repo. but where do I add these props? somewhere in bb.edn?

Karol Wójcik06:10:22

You have to add the dependency. Readme.md of the project should help you understand what you should do.

Karol Wójcik06:10:08

Alternatively you can pass the params from native-image.properties to bb.edn native-image-args

Karol Wójcik06:10:31

Do you know how to do this? @U0510KXTU

steveb8n06:10:17

I’ve added the dep. and the lambdas are working and logging so it’s not a big deal

steveb8n06:10:40

I didn’t realise the readme at the root of the project was where the instructions exist

steveb8n06:10:21

I was looking in the example project of the link you added above

Karol Wójcik07:10:30

Nope. native-image automatically gathers all native-image.properties and merges them together. This is not a feature of HL :D

Karol Wójcik07:10:54

Basically you have to add a dependency to a classpath and everything is done automatically

Karol Wójcik07:10:22

Anyway. Everything works fine? @U0510KXTU

steveb8n07:10:56

yeah. it’s all working great

steveb8n07:10:32

I’ve added the simple dep but still get the warning. however logging works properly so I don’t really care about the warning

steveb8n07:10:38

I’ll leave it alone for now

Karol Wójcik07:10:19

What is the warning?

Karol Wójcik07:10:32

It's probably easy to fix with one line of configuration

steveb8n07:10:06

hmm. I can’t find it now. maybe it’s already fixed

steveb8n07:10:13

let’s forget it in that case. false alarm 🙂

steveb8n08:10:48

ah found it….

steveb8n08:10:51

+ bb native:executable --runtime native
Could not find /project/.holy-lambda/clojure-tools-1.10.3.967.jar
Attempting download from 
Checking out:  at 5eaccc18337b6d875cc81d5908432d4df57899bc
Downloading: clojure-term-colors/clojure-term-colors/0.1.0/clojure-term-colors-0.1.0.pom from clojars
Downloading: clojure-term-colors/clojure-term-colors/0.1.0/clojure-term-colors-0.1.0.jar from clojars
[holy-lambda] Executable /root/.graalvm/bin/java does not exists! Using fallback: java instead! Did you set up GRAALVM_HOME?
[holy-lambda] Executable /root/.graalvm/bin/native-image does not exists! Using fallback: native-image instead! Did you set up GRAALVM_HOME?
[holy-lambda] Command <native:executable>
Warning: Ignoring server-mode native-image argument --no-server.
[output:164]    classlist:   6,913.00 ms,  2.23 GB
--initialize-at-build-time without arguments has been deprecated and will be removed in GraalVM 22.0.
[output:164]        (cap):     651.52 ms,  2.23 GB
WARNING: Could not resolve fierycod.holy_lambda.core$entrypoint$fn__6578 for reflection configuration. Reason: java.lang.ClassNotFoundException: fierycod.holy_lambda.core$entrypoint$fn__6578.
[output:164]        setup:   2,515.74 ms,  2.23 GB
Warning: class initialization of class org.apache.commons.logging.impl.Log4JLogger failed with exception java.lang.NoClassDefFoundError: org/apache/log4j/Priority. This class will be initialized at run time because option --allow-incomplete-classpath is used for image building. Use the option --initialize-at-run-time=org.apache.commons.logging.impl.Log4JLogger to explicitly request delayed initialization of this class.
WARNING: Could not register reflection metadata for org.apache.commons.logging.impl.Log4JLogger. Reason: java.lang.NoClassDefFoundError: org/apache/log4j/Priority.
[output:164]     (clinit):   2,187.24 ms,  5.27 GB
[output:164]   (typeflow):  57,038.49 ms,  5.27 GB
[output:164]    (objects):  69,540.11 ms,  5.27 GB

steveb8n08:10:48

strange warning about HL entrypoint as well

Karol Wójcik08:10:12

Ah. Right. The first one regarding entrypoint will be fixed in the next release. Ignore it for now.

Karol Wójcik08:10:51

Could you remove this: org.apache.commons.logging.impl.Log4JLogger and fierycod.holy_lambda.core$entrypoint$fn__6578 from reflection configurations?

Karol Wójcik08:10:19

Also add this option --initialize-at-run-time=org.apache.commons.logging.impl.Log4JLogger to bb.edn 😄

steveb8n08:10:09

can add the option easily. removing those entries means extra CI complexity because I run native:conf in CI as well

steveb8n08:10:19

I’ll meet you halfway 🙂

Karol Wójcik08:10:58

This one should work just fine: --initialize-at-run-time=org.apache.commons.logging.impl.Log4JLogger

steveb8n08:10:15

yep. running that one through CI now

steveb8n08:10:03

WARNING: Could not register reflection metadata for org.apache.commons.logging.impl.Log4JLogger. Reason: java.lang.NoClassDefFoundError: org/apache/log4j/Priority.

steveb8n08:10:12

same for that class?

Karol Wójcik09:10:02

This warning you get, because :conf generates an entry for Log4JLogger, but the class is not in the classpath. You should remove the entry from reflect-configurations 😄

steveb8n09:10:09

Roger that. Thanks

Benjamin09:10:42

I'm trying to compile for graalvm using this https://github.com/http-kit/http-kit how do I add to reflection-config.json because it get's auto generated by hl:native:conf right?

Karol Wójcik09:10:41

It's auto generated via hl:native:conf. See https://fierycod.github.io/holy-lambda/#/native-backend Configuration is auto-added 😉 Regarding the --initialize-at-build-time. The commit with the configuration on http-kit side has not been yet released.

Karol Wójcik09:10:15

--initialize-at-build-time=org.httpkit.client.ClientSslEngineFactory\$SSLHolder
or
--initialize-at-build-time=org.httpkit.client.ClientSslEngineFactory$SSLHolder

Karol Wójcik09:10:25

to bb.edn native-image-args 🙂

Karol Wójcik09:10:40

Reflection != init flags

Benjamin09:10:03

tried with the initialize at build flag. It throws this:

Error: Detected an instance of Random/SplittableRandom class in the image heap. Instances created during image generation have cached seed values and don't behave as expected.  To see how this object got instantiated use --trace-object-instantiation=java.security.SecureRandom. The object was probably created by a class initializer and is reachable from a static field. You can request class initialization at image runtime by using the option --initialize-at-run-time=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Detailed message:
Trace: Object was reached by 
	reading field sun.security.ssl.SSLContextImpl.secureRandom of
		constant sun.security.ssl.SSLContextImpl$TLSContext@7f03e44d reached by 
	reading field javax.net.ssl.SSLContext.contextSpi of
		constant javax.net.ssl.SSLContext@49504750 reached by 
	scanning method org.httpkit.client.ClientSslEngineFactory.getContextInstance(ClientSslEngineFactory.java:29)
Call path from entry point to org.httpkit.client.ClientSslEngineFactory.getContextInstance(): 
	at org.httpkit.client.ClientSslEngineFactory.getContextInstance(ClientSslEngineFactory.java:29)
	at org.httpkit.client.ClientSslEngineFactory.trustAnybody(ClientSslEngineFactory.java:33)
	at org.httpkit.client$coerce_req.invokeStatic(client.clj:68)
	at org.httpkit.client$request.invokeStatic(client.clj:242)
	at org.httpkit.client$request.doInvoke(client.clj:180)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at org.singularity_group.bot_announce.core.main(Unknown Source)
	at com.oracle.svm.core.JavaMainWrapper.runCore(JavaMainWrapper.java:146)
	at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:182)
	at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)

Karol Wójcik09:10:45

Add tracing for Secure Random and show the trace

Karol Wójcik09:10:45

What GraalVM version do you use?

Benjamin09:10:21

graalvm-ce-java8-21.2.0

Benjamin09:10:26

ah maybe it's old

Benjamin09:10:47

Trace: Object was reached by 
	reading field sun.security.ssl.SSLContextImpl.secureRandom of
		constant sun.security.ssl.SSLContextImpl$TLSContext@1beff060 reached by 
	reading field javax.net.ssl.SSLContext.contextSpi of
		constant javax.net.ssl.SSLContext@1d491df0 reached by 
	scanning method org.httpkit.client.ClientSslEngineFactory.getContextInstance(ClientSslEngineFactory.java:29)
Call path from entry point to org.httpkit.client.ClientSslEngineFactory.getContextInstance(): 
	at org.httpkit.client.ClientSslEngineFactory.getContextInstance(ClientSslEngineFactory.java:29)
	at org.httpkit.client.ClientSslEngineFactory.trustAnybody(ClientSslEngineFactory.java:33)
	at org.httpkit.client$coerce_req.invokeStatic(client.clj:68)
	at org.httpkit.client$request.invokeStatic(client.clj:242)
	at org.httpkit.client$request.doInvoke(client.clj:180)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at org.singularity_group.bot_announce.core.main(Unknown Source)
	at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Lorg_singularity_1group_bot_1announce_core_2_0002emain_00028_3Ljava_lang_String_2_00029V(generated:0)

Karol Wójcik09:10:19

Which version of http-kit do you use? @U02CV2P4J6S

Benjamin09:10:45

I see. So I could try downgrading for now

Benjamin09:10:02

http-kit/http-kit {:mvn/version "2.5.3"}

borkdude09:10:34

No, this is wrong. http-kit does support graalvm and you should in fact be using the newest alpha which has built-in native image config

borkdude09:10:04

however you should probably not be creating http clients on the top level which will initialize this class at build time, which is never supported

borkdude09:10:15

and that might be the case in your code.

Karol Wójcik09:10:35

But still if @U04V15CAJ says you should upgrade, you should upgrade 😄

Benjamin09:10:52

to 2.6.0-alpha1 ? Toplevel means in def right but now inside functions @U04V15CAJ

Karol Wójcik09:10:16

Or use a delay

Benjamin10:10:37

huehue worked now 🙂