Fork me on GitHub
#holy-lambda
<
2023-05-02
>
bmcgavin22:05:33

Hello, I'm trying to use Faraday to connect to DynamoDB in a native lambda and get lots of confusing errors about log patterns. Are there packages I should exclude in deps.edn? Faraday is bringing in commons-logging.

Karol Wójcik02:05:31

Hey, Could you post the stacktraces? You can also try to get extensive help on #CAJN79WNT channel ;)

bmcgavin06:05:05

I think it might be something to do with the cognitect.aws classes that faraday uses, so I'm adding those to the reflect.json, but here's last night's stack trace and odd logging errors :

ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
ERROR StatusLogger Unrecognized format specifier [d]
ERROR StatusLogger Unrecognized conversion specifier [d] starting at position 16 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [thread]
ERROR StatusLogger Unrecognized conversion specifier [thread] starting at position 25 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [level]
ERROR StatusLogger Unrecognized conversion specifier [level] starting at position 35 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [logger]
ERROR StatusLogger Unrecognized conversion specifier [logger] starting at position 47 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [msg]
ERROR StatusLogger Unrecognized conversion specifier [msg] starting at position 54 in conversion pattern.
ERROR StatusLogger Unrecognized format specifier [n]
ERROR StatusLogger Unrecognized conversion specifier [n] starting at position 56 in conversion pattern.
Exception in thread "main" java.lang.ExceptionInInitializerError
at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:142)
at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:137)
at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:100)
at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:80)
at taoensso.faraday$client_params.invokeStatic(faraday.clj:152)
at taoensso.faraday$fn__6311.invokeStatic(faraday.clj:177)
at taoensso.faraday$fn__6311.invoke(faraday.clj:176)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:667)
at clojure.core$memoize$fn__6894.doInvoke(core.clj:6342)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at taoensso.faraday$db_client.invokeStatic(faraday.clj:208)
at taoensso.faraday$put_item.invokeStatic(faraday.clj:1013)
at taoensso.faraday$put_item.doInvoke(faraday.clj:994)
at clojure.lang.RestFn.invoke(RestFn.java:445)
at bmcgavin.birthday.db.dynamodb$db_put.invokeStatic(dynamodb.clj:29)
at bmcgavin.birthday.db$fn__7221.invokeStatic(db.clj:21)
at bmcgavin.birthday.db$fn__7221.invoke(db.clj:21)
at clojure.lang.MultiFn.invoke(MultiFn.java:229)
at bmcgavin.birthday.api$put.invokeStatic(api.clj:56)
at bmcgavin.birthday.api$put_handler.invokeStatic(api.clj:66)
at bmcgavin.birthday.api$put_handler.invoke(api.clj:60)
at compojure.response$fn__612.invokeStatic(response.clj:47)
at compojure.response$fn__612.invoke(response.clj:35)
at compojure.response$fn__562$G__557__569.invoke(response.clj:7)
at compojure.core$wrap_response$fn__2242.invoke(core.clj:158)
at compojure.core$wrap_route_middleware$fn__2226.invoke(core.clj:128)
at compojure.core$wrap_route_info$fn__2231.invoke(core.clj:137)
at compojure.core$wrap_route_matches$fn__2235.invoke(core.clj:146)
at compojure.core$routing$fn__2250.invoke(core.clj:185)
at clojure.core$some.invokeStatic(core.clj:2705)
at compojure.core$routing.invokeStatic(core.clj:182)
at compojure.core$routing.doInvoke(core.clj:182)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invokeStatic(core.clj:669)
at compojure.core$routes$fn__2254.invoke(core.clj:190)
at fierycod.holy_lambda_ring_adapter.core$ring_LT___GT_hl_middleware$fn__2699.invoke(core.cljc:150)
at clojure.lang.Var.invoke(Var.java:384)
at fierycod.holy_lambda.custom_runtime$next_iter.invokeStatic(custom_runtime.clj:80)
at fierycod.holy_lambda.custom_runtime$next_iter.invoke(custom_runtime.clj:60)
at clojure.lang.Var.invoke(Var.java:399)
at bmcgavin.birthday.api$_main.invokeStatic(api.clj:97)
at bmcgavin.birthday.api$_main.doInvoke(api.clj:97)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.RestFn.applyTo(RestFn.java:132)
at bmcgavin.birthday.api.main(Unknown Source)
Caused by: java.lang.IllegalArgumentException
at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:260)
at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:274)
at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:347)
... 48 more
RequestId: 56b5ae56-95d1-4e67-a7d4-76052816fcb0 Error: Runtime exited with error: exit status 1
Runtime.ExitError

bmcgavin09:05:36

I've changed from faraday to the cognitect-aws package and now get :

clojure.lang.ExceptionInfo: Cannot find resource cognitect/aws/dynamodb/service.edn. {}
	at cognitect.aws.service$service_description.invokeStatic(service.clj:31)
	at cognitect.aws.client.api$client.invokeStatic(api.clj:69)
	at bmcgavin.birthday.db.dynamodb$make_ddb.invokeStatic(dynamodb.clj:86)
	at bmcgavin.birthday.db.dynamodb$db_get.invokeStatic(dynamodb.clj:89)
	at bmcgavin.birthday.db$fn__13291.invokeStatic(db.clj:13)
	at bmcgavin.birthday.db$fn__13291.invoke(db.clj:13)
	at clojure.lang.MultiFn.invoke(MultiFn.java:229)
	at bmcgavin.birthday.api$get_handler.invokeStatic(api.clj:80)
	at bmcgavin.birthday.api$get_handler.invoke(api.clj:77)
	at compojure.response$fn__612.invokeStatic(response.clj:47)
	at compojure.response$fn__612.invoke(response.clj:35)
	at compojure.response$fn__562$G__557__569.invoke(response.clj:7)
	at compojure.core$wrap_response$fn__2242.invoke(core.clj:158)
	at compojure.core$wrap_route_middleware$fn__2226.invoke(core.clj:128)
	at compojure.core$wrap_route_info$fn__2231.invoke(core.clj:137)
	at compojure.core$wrap_route_matches$fn__2235.invoke(core.clj:146)
	at compojure.core$routing$fn__2250.invoke(core.clj:185)
	at clojure.core$some.invokeStatic(core.clj:2705)
	at compojure.core$routing.invokeStatic(core.clj:182)
	at compojure.core$routing.doInvoke(core.clj:182)
	at clojure.lang.RestFn.applyTo(RestFn.java:139)
	at clojure.core$apply.invokeStatic(core.clj:669)
	at compojure.core$routes$fn__2254.invoke(core.clj:190)
	at clojure.lang.Var.invoke(Var.java:384)
	at org.httpkit.server.HttpHandler.run(RingHandler.java:121)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:829)
	at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:597)
	at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:194)
I added an agent call but this didn't change anything :
(agent/in-context
 (let [dynamodb (delay (aws/client {:api :dynamodb
                                    :region "eu-west-1"}))
       response (aws/invoke dynamodb {:op :ListTables})]
   (println response)))

Karol Wójcik09:05:43

Exactly. You have to regenerate configs ;)

bmcgavin09:05:19

Excellent, thank you. I got a bit further. I managed to get an executable with using a dummy dynamodb query in the agent, but that gave me this error when calling the API :

"Could not locate cognitect/aws/http/cognitect__init.class, cognitect/aws/http/cognitect.clj or cognitect/aws/http/cognitect.cljc on classpath."
So I made a more true-to-life dynamodb query in the agent, and now the config generation shows this error (but completes) :
Exception in agent-context:  #error {
 :cause No implementation of method: :-invoke of protocol: #'cognitect.aws.client.protocol/Client found for class: nil
 :via
 [{:type java.lang.IllegalArgumentException
   :message No implementation of method: :-invoke of protocol: #'cognitect.aws.client.protocol/Client found for class: nil
   :at [clojure.core$_cache_protocol_fn invokeStatic core_deftype.clj 583]}]
 :trace
 [[clojure.core$_cache_protocol_fn invokeStatic core_deftype.clj 583]
  [cognitect.aws.client.protocol$fn__10680$G__10636__10687 invoke protocol.clj 7]
  [cognitect.aws.client.api$invoke invokeStatic api.clj 130]
  [bmcgavin.birthday.db.dynamodb$fn__13282 invokeStatic dynamodb.clj 112]
  [bmcgavin.birthday.db.dynamodb$fn__13282 invoke dynamodb.clj 112]
  [bmcgavin.birthday.db.dynamodb__init load nil 112]
  [bmcgavin.birthday.db.dynamodb__init <clinit> nil -1]
  [java.lang.Class forName0 Class.java -2]
  [java.lang.Class forName Class.java 398]
  [clojure.lang.RT classForName RT.java 2212]
  [clojure.lang.RT classForName RT.java 2221]
  [clojure.lang.RT loadClassForName RT.java 2240]
  [clojure.lang.RT load RT.java 449]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6856 invoke core.clj 6115]
  [clojure.core$load invokeStatic core.clj 6114]
  [clojure.core$load doInvoke core.clj 6098]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5897]
  [clojure.core$load_one invoke core.clj 5892]
  [clojure.core$load_lib$fn__6796 invoke core.clj 5937]
  [clojure.core$load_lib invokeStatic core.clj 5936]
  [clojure.core$load_lib doInvoke core.clj 5917]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_libs invokeStatic core.clj 5974]
  [clojure.core$load_libs doInvoke core.clj 5958]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$require invokeStatic core.clj 5996]
  [bmcgavin.birthday.db$loading__6737__auto____13264 invoke db.clj 1]
  [bmcgavin.birthday.db__init load nil 1]
  [bmcgavin.birthday.db__init <clinit> nil -1]
  [java.lang.Class forName0 Class.java -2]
  [java.lang.Class forName Class.java 398]
  [clojure.lang.RT classForName RT.java 2212]
  [clojure.lang.RT classForName RT.java 2221]
  [clojure.lang.RT loadClassForName RT.java 2240]
  [clojure.lang.RT load RT.java 449]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6856 invoke core.clj 6115]
  [clojure.core$load invokeStatic core.clj 6114]
  [clojure.core$load doInvoke core.clj 6098]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.core$load_one invokeStatic core.clj 5897]
  [clojure.core$load_one invoke core.clj 5892]
  [clojure.core$load_lib$fn__6796 invoke core.clj 5937]
  [clojure.core$load_lib invokeStatic core.clj 5936]
  [clojure.core$load_lib doInvoke core.clj 5917]
  [clojure.lang.RestFn applyTo RestFn.java 142]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$load_libs invokeStatic core.clj 5974]
  [clojure.core$load_libs doInvoke core.clj 5958]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.core$apply invokeStatic core.clj 669]
  [clojure.core$require invokeStatic core.clj 5996]
  [bmcgavin.birthday.api$loading__6737__auto____3 invoke api.clj 1]
  [bmcgavin.birthday.api__init load nil 1]
  [bmcgavin.birthday.api__init <clinit> nil -1]
  [java.lang.Class forName0 Class.java -2]
  [java.lang.Class forName Class.java 398]
  [clojure.lang.RT classForName RT.java 2212]
  [clojure.lang.RT classForName RT.java 2221]
  [clojure.lang.RT loadClassForName RT.java 2240]
  [clojure.lang.RT load RT.java 449]
  [clojure.lang.RT load RT.java 424]
  [clojure.core$load$fn__6856 invoke core.clj 6115]
  [clojure.core$load invokeStatic core.clj 6114]
  [clojure.core$load doInvoke core.clj 6098]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.lang.Var invoke Var.java 384]
  [clojure.lang.Util loadWithClass Util.java 251]
  [bmcgavin.birthday.api <clinit> nil -1]]}

Karol Wójcik09:05:27

Hmm, I recommend to ask this question on #CAJN79WNT channel. I didn’t touch graalvm for several months.

bmcgavin09:05:31

That seems like it might be a 'me' problem.

bmcgavin09:05:43

Okay! Thanks for the help!

Karol Wójcik11:05:17

Okey, you didn’t reference the client via @client.

bmcgavin12:05:54

Here's the agent call, is that what you mean? It's more complex than it needs to be I guess as it's using the non-agent code, let me simplify it and see if it's still broken.

(def ddb (atom nil))

(defn make-ddb [data]
  (reset! ddb (aws/client {:api                  :dynamodb
                           :region               (get-in data [:aws :aws-region])
                           :credentials-provider (credentials/basic-credentials-provider
                                                  {:access-key-id     (get-in data [:aws :aws-access-key-id])
                                                   :secret-access-key (get-in data [:aws :aws-secret-access-key])})
                           :endpoint-override {:protocol (keyword (:db-endpoint-protocol data))
                                               :hostname (:db-endpoint-host data)
                                               :port     (Integer/parseInt (:db-endpoint-port data))}})))

;; agent to aid with native image compilation
(agent/in-context
 (let [_ (delay (make-ddb {:aws {:aws-region "us-east-1"
                                 :aws-access-key-id "test"
                                 :aws-secret-access-key "test"}
                           :db-endpoint-protocol "http"
                           :db-endpoint-host "localhost"
                           :db-endpoint-port "4566"}))
       response (aws/invoke @ddb {:op :ListTables})]
   (println response)))

Karol Wójcik12:05:06

You are returning nil from make ddb.

Karol Wójcik12:05:49

As far as I can see. I’m on the phone btw.

bmcgavin12:05:49

It's setting the ddb atom which is references by aws/invoke

Karol Wójcik12:05:11

But you have it in delay.

Karol Wójcik12:05:29

You didn’t reference the delay yet.

Karol Wójcik12:05:02

Btw, you are complicating this code more than it’s necessary.

bmcgavin12:05:29

Yes, I've simplified it and now I get a connection refused.

Karol Wójcik12:05:18

Perfect. Now the agent runs in docker context, so maybe you have to give different connection parameters for it to work.

bmcgavin12:05:33

Wonderful, thank you.

Karol Wójcik12:05:17

I would first check in REPL if you can connect to dynamodb and then adjust it for agent ;)

bmcgavin12:05:58

I can, thanks! I'm connecting to localstack in a container so I'll have to make it a shared network. Thanks so much for your help.

bmcgavin12:05:48

> You are returning nil from make ddb. You nailed it with this, thanks. I've now got the agent to connect (by setting :docker :network "host" in bb.edn) and I've corrected my code to not call reset! twice. Now I get this :

localstack  | [holy-lambda] Runtime error:
localstack  | {:via [{:type java.io.FileNotFoundException, :message "Could not locate cognitect/aws/http/cognitect__init.class, cognitect/aws/http/cognitect.clj or cognitect/aws/http/cognitect.cljc on classpath.", :at [clojure.lang.RT load "RT.java" 462]}], :trace [[clojure.lang.RT load "RT.java" 462] [clojure.lang.RT load "RT.java" 424] [clojure.core$load$fn__6856 invoke "core.clj" 6115] [clojure.core$load invokeStatic "core.clj" 6114] [clojure.core$load doInvoke "core.clj" 6098] [clojure.lang.RestFn invoke "RestFn.java" 408] [clojure.core$load_one invokeStatic "core.clj" 5897] [clojure.core$load_one invoke "core.clj" 5892] [clojure.core$load_lib$fn__6796 invoke "core.clj" 5937] [clojure.core$load_lib invokeStatic "core.clj" 5936] [clojure.core$load_lib doInvoke "core.clj" 5917] [clojure.lang.RestFn applyTo "RestFn.java" 142] [clojure.core$apply invokeStatic "core.clj" 669] [clojure.core$load_libs invokeStatic "core.clj" 5974] [clojure.core$load_libs doInvoke "core.clj" 5958] [clojure.lang.RestFn applyTo "RestFn.java" 137] [clojure.core$apply invokeStatic "core.clj" 669] [clojure.core$require invokeStatic "core.clj" 5996] [cognitect.aws.dynaload$load_ns invokeStatic "dynaload.clj" 10] [cognitect.aws.dynaload$load_var invokeStatic "dynaload.clj" 17] [cognitect.aws.http$resolve_http_client invokeStatic "http.clj" 83] [cognitect.aws.client.shared$fn__12533 invokeStatic "shared.clj" 14] [cognitect.aws.client.shared$fn__12533 invoke "shared.clj" 14] [clojure.lang.Delay deref "Delay.java" 42] [clojure.core$deref invokeStatic "core.clj" 2324] [cognitect.aws.client.shared$http_client invokeStatic "shared.clj" 22] [cognitect.aws.client.api$client invokeStatic "api.clj" 74] [bmcgavin.birthday.db.dynamodb$make_ddb invokeStatic "dynamodb.clj" 16] [bmcgavin.birthday.db.dynamodb$db_get invokeStatic "dynamodb.clj" 19] [bmcgavin.birthday.db$fn__13291 invokeStatic "db.clj" 13] [bmcgavin.birthday.db$fn__13291 invoke "db.clj" 13] [clojure.lang.MultiFn invoke "MultiFn.java" 229] [bmcgavin.birthday.api$get_handler invokeStatic "api.clj" 84] [bmcgavin.birthday.api$get_handler invoke "api.clj" 81] [compojure.response$fn__612 invokeStatic "response.clj" 47] [compojure.response$fn__612 invoke "response.clj" 35] [compojure.response$fn__562$G__557__569 invoke "response.clj" 7] [compojure.core$wrap_response$fn__2242 invoke "core.clj" 158] [compojure.core$wrap_route_middleware$fn__2226 invoke "core.clj" 128] [compojure.core$wrap_route_info$fn__2231 invoke "core.clj" 137] [compojure.core$wrap_route_matches$fn__2235 invoke "core.clj" 146] [compojure.core$routing$fn__2250 invoke "core.clj" 185] [clojure.core$some invokeStatic "core.clj" 2705] [compojure.core$routing invokeStatic "core.clj" 182] [compojure.core$routing doInvoke "core.clj" 182] [clojure.lang.RestFn applyTo "RestFn.java" 139] [clojure.core$apply invokeStatic "core.clj" 669] [compojure.core$routes$fn__2254 invoke "core.clj" 190] [fierycod.holy_lambda_ring_adapter.core$ring_LT___GT_hl_middleware$fn__2903 invoke "core.cljc" 150] [clojure.lang.Var invoke "Var.java" 384] [fierycod.holy_lambda.custom_runtime$next_iter invokeStatic "custom_runtime.clj" 80] [fierycod.holy_lambda.custom_runtime$next_iter invoke "custom_runtime.clj" 60] [clojure.lang.Var invoke "Var.java" 399] [bmcgavin.birthday.api$_main invokeStatic "api.clj" 107] [bmcgavin.birthday.api$_main doInvoke "api.clj" 107] [clojure.lang.RestFn invoke "RestFn.java" 397] [clojure.lang.AFn applyToHelper "AFn.java" 152] [clojure.lang.RestFn applyTo "RestFn.java" 132] [bmcgavin.birthday.api main nil -1]], :cause "Could not locate cognitect/aws/http/cognitect__init.class, cognitect/aws/http/cognitect.clj or cognitect/aws/http/cognitect.cljc on classpath."}

bmcgavin12:05:47

I've just seen this from https://github.com/clj-easy/graalvm-clojure/tree/master/aws-api-s3 : > Finally, you need to pre-create a http-client for the same reason and pass it to all your aws clients. For this you can use Clojure's delay, to ensure runtime initialization.

Karol Wójcik15:05:31

Yes, it seems you also need to pre-create http client as well.

bmcgavin15:05:13

Yes, it's very nearly working, I now think I have the problem where the dynamodb client is hard-coded with the values passed to the agent invocation.

bmcgavin15:05:51

Is there a way to pass environment variables to the agent docker container?

Karol Wójcik15:05:48

Nope. You have to do „if” in the code.

👍 2
bmcgavin16:05:21

Got it all working, thank you very much!