Clojurians Log v2

@adityaathalye RE: classes loaded dynamically, I don’t think so. I’ve verified that the clojure.core.cache jar is in the uberjar. I’m not doing any dynamic loading of namespaces or classes. I’m getting access to the customer machine this afternoon, so I’ll report back then. Thanks!

Here’s the stack trace:

This is an app built using Leiningen running on JDK 8, Oracle 12c. The app is dying at one client’s location, but it appears to be functioning fine at other locations (with similar machine/OS/container setups). I can reproduce a very similar stack trace on my local machine by starting up the JVM with a severely restricted stack size.

This happened earlier this month, then stopped after an Oracle update was installed. Since this doesn’t happen with other Java/Spring services on the host, I’m wondering if AOT might alleviate the issue. I believe I’d have to rewrite a few thing to remove some macros (using the mount library) in order to do this though.

Stack size limit seems like a plausible problem! Your local compilation - is that also in some jdk 8? I'm not that fond of mount, but I don't see anything about mount in the stacktrace. But yeah, increase the stacktrace, or even update the JVM on the installation site?

@oscarlinusericsson I do run the app locally on JDK 8 before submitting to CI. I mentioned mount as I believe that would prevent AOT for the app. There may be other barriers.

@oscarlinusericsson Did you mean increase the startup stack size? (i.e. -Xss) The full stacktrace is accessible via a link at the bottom

For brevity’s sake, the stacktrace above gets cut off before this line: Caused By: java.lang.Exception: namespace 'clojure.core.cache' not found. Which I’ve confirmed that this library is located within the uberwar. It just occurred to me that this dependency could be causing a conflict with pedestal’s core.cache dependency

Nope. I was wrong. Pedestal doesn’t have a dependency on core.cache that I can see…

Oh, and updating the JVM isn’t possible. This serves an Oracle Forms application which is tied to Java 8 😄

How does your project.clj or deps.edn look? It could also be that some clojure library or other dependency compiled for a more recent jvm (but the error should likely be different). another trick is to clear you .m2 folder, sometimes things are just broken

project.clj

(defproject com.fipco/aus "0.2.6"
  :description "FIXME: write description"
  :url ""
  :license {:name "Eclipse Public License"
            :url  ""}
  :repositories {"fipco-maven" ""}
  :dependencies [[org.clojure/clojure "1.10.1"]
                 [io.pedestal/pedestal.service "0.5.8"]
                 [io.pedestal/pedestal.jetty "0.5.8"]

                 [ch.qos.logback/logback-classic "1.2.3" :exclusions [org.slf4j/slf4j-api]]
                 [org.slf4j/jul-to-slf4j "1.7.26"]
                 [org.slf4j/jcl-over-slf4j "1.7.26"]
                 [org.slf4j/log4j-over-slf4j "1.7.26"]

                 [clj-http "3.12.1"]

                 [org.clojure/data.xml "0.2.0-alpha6"]
                 [org.clojure/data.zip "0.1.3"]
                 [org.clojure/data.json "1.0.0"]
                 [org.clojure/core.cache "1.0.207"]
                 [net.jodah/expiringmap "0.5.9"]
                 [com.sun.mail/javax.mail "1.5.5"]
                 [ring/ring-codec "1.1.2"]
                 [org.clojure/java.jdbc "0.7.10"]
                 [com.oracle/ojdbc8 "12.2.0.1"]
                 [conman "0.8.4"]

                 [mount "0.1.16"]

                 [tick/tick "0.4.31-alpha"]

                 [org.clojure/test.check "0.10.0"]]
  :min-lein-version "2.0.0"
  :resource-paths ["config", "resources"]
  ;; If you use HTTP/2 or ALPN, use the java-agent to pull in the correct alpn-boot dependency
                                        ;:java-agents [[org.mortbay.jetty.alpn/jetty-alpn-agent "2.0.5"]]
  :plugins [[ohpauleez/lein-pedestal "0.1.0-beta10"]
            [lein-nvd "1.4.1"]]
  :pedestal {:servlet-description  "FIPCO Automated Underwriting Service Interface"
             :servlet-display-name "FIPCO AUS"
             :server-ns            "com.fipco.aus.server"
             :url-pattern          "/*"
             :war-resources-paths  ["war-resources"]
             :manifest             {"Created-By" "FIPCO"}
             :war-exclusions       [#"config/losDataSource\.properties" #".DS_Store"]}
  :profiles {:dev
             {:aliases
              {"run-dev" ["trampoline" "run" "-m" "com.fipco.aus.server/run-dev"]}
              :source-paths ["src/" "siderail"]
              :dependencies [[io.pedestal/pedestal.service-tools "0.5.8"]
                             [org.clojure/tools.trace "0.7.10"]
                             [expound "0.8.4"]
                             [jonase/eastwood "0.3.5" :exclusions [org.clojure/clojure]]]
              :plugins [[lein-kibit "0.1.8"]
                        [com.billpiel/sayid "0.0.18"]]
              :jvm-opts ["-Dhttp.proxyHost=127.0.0.1"
                         "-Dhttp.proxyPort=8888"]}
             :uberjar {:aot [com.fipco.aus.server]}}
  :main ^{:skip-aot true} com.fipco.aus.server)

The service is built via Jenkins on a Windows machine (not local). I’ll make a note to clear out the maven cache on that machine.

there is nothing apparently mysterious in the project.clj (but jonase/eastwood is usually replaced with clj-kondo these days).

hmm, windows...

maybe you need to add a resonable not to new version of https://github.com/clojure/core.memoize

That’s a great thought. I’m looking into how much I even really need it. I may be able to remove that dependency entirely. I’m just not certain that’ll be sufficient.

On further investigation, core.memoize is a transitive dependency (pulled in by pedestal.service). I was thinking about the core.cache dependency.

well, increase the stack size (or just memory in general - Xmx and Xms when compiling, I guess). The stack trace is passing through core.async, I remember there where bugs/incompatibilites in core.async. It could be a conflicting dependency somewhere. lein deps :tree it is infuriating to debug, esp if you don't have close access to the build machine is not in reach

@oscarlinusericsson thanks, I appreciate you taking a look

Spitballing... First: lein deps :tree, like Linus said, to inspect any obvious mismatches. Alternate: is it possible some class(es) is(are) being loaded dynamically, depending on stack allocation at JVM startup? If yes, do the class files just happen to be on the class path in the build machine as a side effect of the build process, (and on your dev machine due to use)?