Different classpaths?

Hmmmm how could that happen? I don't really see how that could occur

Using different aliases in different environments. Or relying on snapshot dependencies. Or something else.

Not the case here. I even checked if the dependencies weren't being overwritten by the test profile or dev. But I don't have anything in place for that

To be completely sure, you can compare (System/getProperty "java.class.path"). Also, something can potentially override the function at run time.

`/users/iceman/.m2/repository/clojurewerkz/machine_head/1.0.0/machine_head-1.0.0.jar` is in the classpath

that is what I am expecting

also (-> mh/connect class .getDeclaredMethods first .getParameterTypes alength) returns 1 during runtime (debug) and 2 in the repl 🥲

Is there a chance your debugging runtime has some classes on its classpath that has outdated compiled stuff? Or maybe it has something else on the classpath that overrides that particular function in that namespace.

forget it… I am just flabbergasted at my stupidity right now, sorry to have wasted your time (with-redefs [mh/connect (fn [uri] uri)

nuff said 😄

Well, there you go. As I mentioned, "something can override the function at run time". :)

Yes you were right… Well, I guess it is not a very good idea do work when tired 😄

I build with the following build.clj

(ns build
  (:require [clojure.tools.build.api :as b]))

(def lib 'brian/api)
(def version (format "1.2.%s" (b/git-count-revs nil)))
(def class-dir "target/classes")
(def basis (b/create-basis {:project "deps.edn"}))
(def uber-file (format "target/uberjar/brian-api.jar"))

(defn clean [_]
  (b/delete {:path "target"}))

(defn uber [_]
  (clean nil)
  (b/copy-dir {:src-dirs ["src/main" "resources"]
               :target-dir class-dir})
  (b/compile-clj {:basis basis
                  :src-dirs ["src/main"]
                  :class-dir class-dir})
  (b/uber {:class-dir class-dir
           :uber-file uber-file
           :basis basis
           :main 'brian.server-components.http-server}))

I tried pinning com.google.code.gson/gson 2.9.0 because it's the version linked in the readme of stripe-java

it gets a different error:

[{:type java.lang.NoClassDefFoundError
   :message "com/google/gson/ReflectionAccessFilter"
   :at [com.stripe.net.Webhook constructEvent "Webhook.java" 49]}
  {:type java.lang.ClassNotFoundException
   :message "com.google.gson.ReflectionAccessFilter"
   :at [jdk.internal.loader.BuiltinClassLoader loadClass nil -1]}]

this time the error happens both in dev and ubjerjar

the original error happens as well when pinning the version of gson that stripe-java expects based on what I see in deps tree

anyway I was just curious if anyone had an idea on how to resolve that kind of issue, for now I just ditched the whole stripe-java library since I was only using it to validate the signature. So if anyone finds this thread looking for an answer regarding this error I have not found one yet. But you can easily replace this lib if you use it to validate a signature with the following snippet: [buddy.core.codecs :as codecs] [buddy.core.mac :as mac]

(defn compute-hmac-sha256 [secret message]
  (-> message
      (mac/hash {:key secret :alg :hmac+sha256})
      codecs/bytes->hex))

(defn valid-signature? [stripe-webhook-secret payload stripe-signature]
  (let [{:strs [v1 t]} (apply hash-map
                              (mapcat #(str/split % #"=")
                                      (str/split stripe-signature #",")))
        signed-payload (str t "." payload)]
    (= v1 (compute-hmac-sha256 stripe-webhook-secret signed-payload))))

(defn process-webhook-event
  "May throw errors that should be caught by the http stack."

(beware you should also compare t to the current timestamp if you want to prevent replay attacks)

you should check and make sure the same version of google's gson library is used in the repl and in your uberjar

you might also check that you rare using the same java version when you try and the uberjar and at the repl

@U03K8V573EC I would start the jvm with -verbose:class to see how it load classes, the order and where they are coming from (what jar and version) in both cases (specially the one that works and the one that fails). Also do a jar -tf target/uberjar/brian-api.jar to check the class is there on the uberjar and go from there

Very strange things can happen (along these very lines) if 2+ jars on the classpath contribute distinct versions of "the same" library or portions thereof.

@U0HG4EHMH that might just be the explanation yeah, I'll run the experiment when I have more time

the serialized data contains references to a class created at runtime by clojure (as a result of loading clojure code and compiling it to jvm bytecode)

that class is available through one of clojure's dynamicclassloaders, but is not available to the classloader that is being used while deserializing

is there a way to find out which class it is?

it is in the exception there

clojure.core$eval193$fn__194

I have looked through the structure that gets saved to the file, and there's nothing unknown in there, that's why I don't get how there's a reference to something generated in runtime

and this has worked before, it stopped working recently

are you doing this in the same jvm or across different jvms?

I mean is the same process serializing and deserializing, or different processes(either across time or space)?

it is the same jvm, let's say I serialize on termination

and deserialize on startup

this has worked before

so different processes

same executable code

are you using the same clojure version as the last time you serialized?

yes

are you sure?

everything is the same, as I said, same jdk, same clojure, same executable

I run the executable, when I stop it, it saves, and when I run it again, it deserializes the data

or tries to

the class name clojure.core$eval193$fn__194 corresponds to an anonymous function eval'ed while the value of *ns* (the current namespace) is clojure.core

that is kind of odd, since clojure is aot compiled, but you can generate something similar like:

% clj
Clojure 1.11.1
user=> (in-ns 'clojure.core)
#object[clojure.lang.Namespace 0x1e392345 "clojure.core"]
clojure.core=> (fn [])
#object[clojure.core$eval138$fn__139 0x7c9b78e3 "clojure.core$eval138$fn__139@7c9b78e3"]
clojure.core=> (class *1)
clojure.core$eval138$fn__139
clojure.core=>
%

makes sense

also depending on how you are launching your code, it is possible *ns* just remains at its default value of clojure.core, in which cause any usuage of say clojure.core/eval would result in classes like that

how would one avoid that issue? or at least minimize it?

don't serialize references to classes generated by the clojure compiler

in certain circumstances it will generate strongly named classes (classes whose name will stay the same across different compilations) but that is the exception rather than the rule

I found the issue, and it is related with an anonymous function

FWIW, no version of Clojure that I can find has core$eval193 in it so that seems like something created "on the fly" once Clojure is up and running?

is there a way to let's say, name the anonymous function on the fly, so that it doesn't end up with a name like clojure.core$eval193$fn__194?

(fn somename [..] ..) you mean?

that still won't get you a stable name though

somename will be part of the generated class name

but some part of it will be a number that will be effected by how many times nextId is called in the compiler

basically the whole point of anonymous function is it has no name, but it is compiled to a jvm class, which needs a name, so the compiler generates one as to avoid colliding with other generated names

I am dynamically loading a configuration from an edn file, that can contain anonymous functions or point to specific functions that are already defined. But some of these might end up in the final serialized file. So is there a way to receive an anonymous function but name it when evaluated?

and generated class names are a bad mix with java serialization

if I used predefined functions, I can deserialize the file no issues

I was using nippy

but I still get the same issue, because it needs to use the Serializable interface

I found a possible solution:

(defn gen-fn
  [n as b]
  (let [n        (symbol n)
        as       (vec (map symbol as))
        fn-value (eval `(fn ~n ~as ~b))]
    (intern *ns* n fn-value)))

calling eval to read a config file is kind of gross

the file is not read with eval

but so is using java serialization

it is read with aero

but there are parts that need to be evaled

that is gross

thanks for the input

as long as you are using eval you are going to get generated class names, and those generated class names are non-deterministic, so mixing them with java serialization will always be fragile, and you'll hit things like moving code around (changing the order of eval) might break deserialization, changing clojure versions might break it) and changing jvm versions might break it

I am aware of that, but all of that would break even if I wasn't using Java Serialization. Since there are no guarantees of same serialization/deserialization between jdk versions

if you bound *ns* around the call to eval and maybe eval'ed deftypes instead of fns, you would have better control over the names which depending on how you create a name for each thing, might be less fragile

yeah, makes sense

thanks for all your help

https://github.com/clojure/data.int-map I guess I could just enumerate the key space using data.int-map and round any decimal keys, but I'm also curious if there's a name for this kind of thing

https://en.m.wikipedia.org/wiki/Interval_tree ?

So is the intention that keys are specified with ranges when inserted and with exact values when looked up?

https://github.com/dco-dev/interval-tree And as soon as I knew what to google, it is there! Thanks @U05092LD5

nice!

assuming the intervals don't overlap, B-tree is the classic d.s. here.

https://docs.oracle.com/javase/8/docs/api/java/util/NavigableMap.html would probably work. (Two implementing classes.) It keeps keys in a sorted order. • If there are no gaps, you could just use the end of the interval with floorEntry(). • If there are known gaps, you could use a placeholder value. • If there are no overlaps, you could store both ends and make sure your key is between them by using both floorEntry() and higherEntry() . • If there are overlaps, this might be the wrong data structure, but I'm not sure how you'd want it to work.

https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ConcurrentSkipListMap.html, https://docs.oracle.com/javase/8/docs/api/java/util/TreeMap.html.

you can enable it with the env var CLOJURE_CLI_ALLOW_HTTP_REPO

Excellent ! I was really dreading starting discussing with my IT department about configuring https

Looking at the commit history, the name predates the warning issued when an earmuffed var is not marked dynamic. I think this might have been done to avoid a breaking change. But perhaps others here know more

(ie, wanting to preserve the name with earmuffs before that meant it was dynamic. and it was marked dynamic to not print a warning on startup each time)

Thank you! Yes, I had a similar suspicion about the earmuffs and being marked dynamic

we use jetty with ring on top. recently switched to

info.sunng/ring-jetty9-adapter            {:mvn/version "0.18.5"}

https://github.com/ring-clojure/ring/issues/439 is an issue to update the “regular” ring lib off of jetty 9

We use the Sun Ng Jetty adapter in production and we're very happy with it. http-kit is another option but we found support for observability was very poor with New Relic which made us switch back to Jetty.

Food for thought: hazy memories led to this https://www.slideshare.net/metosin/naked-performance-with-clojure. It doesn't look like experimental repos they were playing with got any further, not that I'm complaining. Metosin has given me so much already.

What are your performance requirements? Are there any other constraints?

I'm moving to a spicedb-based permission system, which has all sorts of design ripple effects. I wanted to reexamine my priors just in case there had been advancements in the basics while my attention was elsewhere

Well, do you need to maximize throughput or response time? Do you have any soft upper bounds for response time percentiles at different request rates?

Throughput, but (and this is a larger topic than the choice of http component), by removing permissions from the db schema, I am starting to treat the db like an object store, so the datomic-style client-side cache starts to make sense. I'm going to play with that idea next, but on a Jetty server 🙂

I've seen all big Clojure server options handle large loads, so I doubt any choice will have adverse effects.

maps are unordered so you couldn’t (safely) reuse bindings

collect=> (let [x 1
                y (inc x)]
            y)
2
collect=> 
(let {x 1
      y (inc x)}
  y)

duh, thanks!

so to handle that you’d have to either do some heavy analysis on dependency, or just use an ordered structure

(also, lisps traditionally use lists and Clojure (tastefully) uses a similar but distinct [ for binding pairs)

Yup, and did away with the distinction between let and let* entirely. Thanks again

if it were possible to use a map, what would destructuring look like?

I guess the key of the may would be a map or vector. map in that case seems to lose a bit of value

oh. the fatal flaw.

(let [x 1
      x (+ x x)]
  x)
2

there is a map-let like thing though. prismatic plumbing graph

https://github.com/plumatic/plumbing

maybe OP could find some use with that

feels pretty close