Clojurians Log v2

Hello again @caio.cascaes. We have been able to reproduce this issue. I opened a ticket https://github.com/cognitect-labs/aws-api/issues/265 to track it further. We are working on testing a fix, but meanwhile the ticket describes some workarounds that you can try right away if you are interested. Thanks for your writeup here, it was helpful to us in investigating and understanding what was going on.

Great! catjam Of course, not great for having that bug 😅

Whoa! Thanks for the detailed issue report :) I suspected that some really gnarly thread classloader race issue is behind this, but didn't have the energy to try figure out what is happening. A good bug report is half the solution :)

Hello again @caio.cascaes and @viesti. I was just wondering if either of you had had a chance to verify, one way or another, whether the aws-api 0.8.730-beta01 release resolved this https://github.com/cognitect-labs/aws-api/issues/265 issue? Not urgent, I'm just looking for feedback before releasing a formal release to follow up this beta release, which we'd like to do within the next couple of weeks or so.

eek, don't have access to the environment where the problem occurred any longer, could maybe whip up a repro, but not promising too much though 😄 Took a look at the change, I think defining the classloader to use is probably ok, since it's used to load aws-api resources specifically 🙂

No worries @viesti 🙂

https://clojurians.slack.com/archives/C015AL9QYH1/p1738270954993799 this release should fix this https://github.com/cognitect-labs/aws-api/issues/265 issue. @caio.cascaes @viesti if either of you have opportunity to verify it would be super helpful!

I'm sure the maintainers would appreciate a tight, self-contained repro to track it down

So far, jar of lambdas and a elastic beanstalk is working fine with this beta version

You're welcome 🙂 Yes it was a doozy! 😅

@viesti and as it turns out, https://clojurians.slack.com/archives/C09N0H1RB/p1734765316094869?thread_ts=1734731335.002159&cid=C09N0H1RB was prescient!

The issue appears to be intermittent. I can provide access to one of the lambdas where it was consistently occurring. After downgrading to version 0.8.692, the error stopped reproducing. However, when testing again with the recent version 0.8.723, the error reappeared. Does it sound good for you, @scottbale?

Sad news: https://clojurians.slack.com/archives/C09N0H1RB/p1740433160473829?thread_ts=1738270992.143669&cid=C09N0H1RB

interesting, cognitect_aws_http.edn is a different file though (original problem was with endpoints.edn, didn't now look if it was renamed)

Let me try a code that could explicitly reproduce this bug. I'll also check your suggestion.

I don't remember any longer, but I might have seen this in async lambda invocation, which you can create via aws cli https://awscli.amazonaws.com/v2/documentation/api/latest/reference/lambda/invoke.html

aws lambda invoke --invocation-type Event --function-name <FUNC> out

Hi @caio.cascaes. I've still had no luck reproducing this problem. Are you still experiencing it? Maybe we can narrow down whether this is a problem with the uberjar being built, or some other runtime problem. When I build an uberjar, I see an entry for the endpoints resource inside my jar:

-rw-r--r--  536019  17-Dec-2024 12:09:26  cognitect/aws/endpoints.edn

@caio.cascaes thanks for the writeup. I have a few questions: • What version of com.cognitect.aws/endpoints are using, in both cases? • What can you tell us about what you are building and deploying to AWS lambdas? Is it a lein uberjar? We are wondering how the endpoints.edn file is packaged up in the artifact that you are deploying, and why the JVM wouldn't be able to locate it as a resource, at runtime. Any information you can supply along those lines would be helpful.

Thank you, @ghadi, for pointing that out. Apologies for the lack of detail in my earlier message. Here’s a simplified example to replicate the issue. I hope this clarifies the scenario and helps in tracking down the problem: `project.clj`:

(defproject aws-lambda-ssm "0.1.0-SNAPSHOT"
  :description "AWS Lambda with SSM"
  :url ""
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url  ""}
  :dependencies [[org.clojure/clojure "1.12.0"]
                 [org.clojure/data.json "2.5.1"]
                 [nubank/matcher-combinators "3.9.1"]
                 [ch.qos.logback/logback-classic "1.5.14"]
                 [org.slf4j/jul-to-slf4j "2.0.16"]
                 [com.amazonaws/aws-lambda-java-core "1.2.3"]
                 [com.amazonaws/aws-lambda-java-events "3.14.0"]
                 [com.cognitect.aws/api "0.8.711"]
                 [com.cognitect.aws/endpoints "871.2.29.35"]
                 [com.cognitect.aws/ssm "871.2.29.35"]
                 [clj-time "0.15.2"]
                 [clj-http "3.13.0"]
                 [dev.weavejester/medley "1.8.1"]
                 [camel-snake-kebab "0.4.3"]]
  :plugins [[lein-ancient "1.0.0-RC3"]
            [lein-environ "1.2.0"]
            [lein-shell "0.5.0"]
            [com.github.clojure-lsp/lein-clojure-lsp "1.3.17"]]
  :clojure-lsp {:settings {:clean {:ns-inner-blocks-indentation :same-line}}}
  :aliases {"clean-ns"     ["clojure-lsp" "clean-ns" "--dry"]
            "format"       ["clojure-lsp" "format" "--dry"]
            "diagnostics"  ["clojure-lsp" "diagnostics"]
            "lint"         ["do" ["clean-ns"] ["format"] ["diagnostics"]]
            "clean-ns-fix" ["clojure-lsp" "clean-ns"]
            "format-fix"   ["clojure-lsp" "format"]
            "lint-fix"     ["do" ["clean-ns-fix"] ["format-fix"]]}
  :test-paths ["test/unit" "test/integration" "test/resources"]
  :profiles {:uberjar {:aot :all}
             :test    {:prep-tasks   [["shell" "cp" "-r" "test/migrations/." "resources/migrations/"]]
                       :dependencies [[org.clojure/test.check "1.1.1"]]}})

(ns aws-lambda-ssm.core
  (:gen-class
   :implements [com.amazonaws.services.lambda.runtime.RequestStreamHandler])
  (:require [clojure.tools.logging :as log]
            [clojure.data.json :as json]
            [ :as io]
            [cognitect.aws.client.api :as aws]))

(defn stream-handler
  [event-body]
  (let [ssm-client (aws/client {:api :ssm})]
    (log/info "content of a secure string:"
              (-> (aws/invoke ssm-client
                              {:op      :GetParameter
                               :request {:Name           "/hey/you"
                                         :WithDecryption true}})
                  :Parameter
                  :Value))))

(defn -handleRequest
  [_this is os _context]
  (let [w (io/writer os)]
    (-> is
        io/reader
        json/read
        stream-handler
        (json/write w))
    (.flush w)))

[com.cognitect.aws/endpoints "871.2.29.35"]

(then I downgraded only the [com.cognitect.aws/api "0.8.711"] to 0.8.692 and solved as workaround)

It was not renamed, @viesti, they are different files, the cognitect_aws_http.edn problem that @caio.cascaes is having is not related to https://github.com/cognitect-labs/aws-api/issues/265. Details in thread he linked.

Ah right, my bad :)

Crime scene: AWS Lambdas, in VPC with private subnets

I've also seen same, but haven't been able to figure what causes this. I haven't had other classloader related problems in JVM/Lambda. Thinking that this is somehow related to the threadpool that core.async uses, since aws-api uses core.async async to talk to the http client