jdbc?

Sorry, I don't follow

In this case the data store is Dynamo ... so I don't think there is JDBC involved ... but I could definitely be mistaken

there's no datomic specific credentialing or jks

it's all storage-specific

I'm only aware of jdbc ssl needing client and/or server keystores

in the jdbc case it goes into the connection string

I believe this key is used for communication with the transactor, not the data store

did you make it? how did you come across it?

The last Datomic item in the stacktrace when it doesn't find the file is datomic.artemis-client/create-session-factory

I didn't make it. It's packaged with Datomic. I learned about it when I got an exception trying to start the peer library in an environment where the class path does not map to the filesystem in the way that the code seems to expect and it could not find the file.

I've been using datomic for 6 years and I've never come across this

the closest I can find is this sample code using hornet:

https://github.com/Datomic/datomic-java-examples/blob/master/src/java/hornet/samples/PingProducer.java

not datomic

im going to check a running app and see if it/s on the classpath

$ jar -tf datomic-pro-0.9.5661.jar | grep transactor-key
datomic/transactor-key.jks

ok then

I suspect you can't move it

changing it would mean providing a netty option

piercing through datomic->artemis->netty

how did this get messed up? uberjaring? strange repackaging?

it's always been a "just works" thing for me

This is on an AWS Lambda, which seems to put everything from the jar under /var/task

So this file lives at /var/task/datomic/transactor-key.jks but Artemis/Netty is still looking for /datomic/transactor-key.jks

but it should be a resource path not a literal path

Yes, but it looks like Artemis is bypassing the resource API

so if /var/task is on the classpath, it should find it

basing that on this part of the stacktrace I see:

Caused by: java.io.FileNotFoundException: /datomic/transactor-key.jks (No such file or directory) 
at java.io.FileInputStream.open0(Native Method) 
at java.io.FileInputStream.open(FileInputStream.java:195) 
at java.io.FileInputStream.<init>(FileInputStream.java:138) 
at java.io.FileInputStream.<init>(FileInputStream.java:93) 
at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) 
at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) 
at java.net.URL.openStream(URL.java:1045) 
at org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeystore(SSLSupport.java:104)

if that were true, wouldn't it never work?

transactor-key is always in a jar

Yes, that is something I've wondered

I mean, normally

Looking at code it does look like Artemis gets that URL by calling findResource ... so, yeah, I'm not sure what is going wrong on the lambda that that isn't working.

https://github.com/apache/activemq-artemis/blob/master/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/remoting/impl/ssl/SSLSupport.java#L283-L299

maybe artemis/netty version fighting?

datomic deps on an artimis "uberjar" which includes netty-all

actually that's imprecise

datomic deps on an older version of artemis which deps on netty-all, which includes every netty class

it's not a pure dependency pom

so getting two netty versions on the classpath was a common problem

perhaps that's what's happening?

Ah, interesting, I will look at the deps tree

This is what I kinda blindly do now:

[com.datomic/datomic-pro "0.9.5786"
                  :exclusions [org.slf4j/slf4j-nop org.slf4j/slf4j-log4j12
                               ;; netty-all is a netty "uberjar". When combined
                               ;; with other things that depend on netty sub-deps
                               ;; directly, we get duplicate netty classes of
                               ;; different versions on the classpath.

                               ;; In our case, artemis-client-core depends on
                               ;; netty-all 4.0; google-cloud-logging-logback
                               ;; transitively depends on specific netty packages
                               ;; (not netty-all); when combined, we get
                               ;; duplicate netty classes of different versions
                               ;; on the classpath.

                               ;; Solution: exclude netty-all and depend on what
                               ;; artemis *actually* requires. Then maven
                               ;; can resolve the dep tree correctly.
                               io.netty/netty-all]]

I got burned so hard

I have even considered repackaging datomic peer lib with a different pom that fixes this issue

elsewhere I will include this:

;; Source for artemis netty deps:
                 ;; 
                 ;; artemis-commons
                 [io.netty/netty-buffer "4.1.34.Final"]
                 [io.netty/netty-transport "4.1.34.Final"]
                 [io.netty/netty-handler "4.1.34.Final"]
                 ;; artemis-core-client
                 [io.netty/netty-transport-native-epoll "4.1.34.Final" :classifier "linux-x86_64"]
                 [io.netty/netty-codec-http "4.1.34.Final"]

I don't think I have anything else requiring netty

could lambda be doing it?

I guess this wouldn't fix the issue if it were

Looking more at that Artemis validateStoryURL function I am thinking that the DWIM nature of that might explain why it works someplaces and not others

ah

it tries to handle URLs, file paths, and resource paths

I think it's ok to escalate to a datomic support ticket at this point

Yeah, I have, just trying to dig a little while I wait for their response

I wonder if this is what's happening:

in normal environments, file.exists() in validateStoreURL returns false so it falls through to the call to findResource, which finds it in the jar, and all is well

Lambda unzips the jar into /var/task which may also be the working directory of the process (not sure) so maybe that causes file.exists() to return true, so it returns the result of file.toURI().toURL() instead of calling findResource

I don't have an explanation for why, if that's true, the subsequent call to java.net.URL.openStream fails

me neither

Thanks for your help. You were right, my buffer positions were incorrect