that's how we do it in clojure tests

You cannot catch a compiler error in the same compilation unit as the error

upgrade your tools.analyzer dependency. could maybe just be a pedestal dependency bump or specifically [org.clojure/tools.analyzer "1.1.1"]

Seems like you’re putting arguments where they don’t belong.

You are calling a function with connection where the argument should have been a prepared statement

it does seem that way

any idea why that should happen only in one environment but not in another?

hard to say without seeing the whole setup and code

The migratus config

(ns ai-clojure.database.migrations
  (:require [ai-clojure.config :refer [env]] 
            [clojure.pprint :as pp]
            [migratus.core :as migratus]))


(defn config []
  {:store                :database
   :migration-dir        "resources/migrations"
   :init-script          "init.sql" ;script should be located in the :migration-dir path 
   :init-in-transaction? false
   :migration-table-name "migrations"
   :db {:dbtype "mysql"
        :dbname (:db-database env)
        :host (:db-host env)
        :port (:db-port env)
        :user (:db-user env)
        :password (:db-password env)}})

(defn migrate []
  (migratus/migrate (config)))

the calling code

(defn start-app [args]
  (doseq [component (-> args
                        (parse-opts cli-options)
                        mount/start-with-args
                        :started)]
    (log/info component "started"))

  (when (= (:app-env env) "PRODUCTION")
    (try (do (log/info "Production environment; running migrations")
             (migrate))
         (catch Exception e (logging-utils/log-exception e)))) 
  
  (.addShutdownHook (Runtime/getRuntime) (Thread. stop-app)))

in dev the env variables come from dev-config.edn, in live they are environment properties for a aws elastic beanstalk instance

do you have just SQL files for migrations or do you also have clojure code migrations?

just SQL

and how does dev env run migrate? via repl?

same way

i just switched out the env = prod for true

to test

Well one thing you can do is print out return of (config) in migrate then compare what you get there in both environments

good idea

The other thing you could do is open up the jar you use in production and look if there are all files from resources in there that you expect

jar is just a zip file

interesting

ok i'll try those things thank you

i did print out full contents of env in prod and found it to be as expected

i dont see the resources folder inside the .jar at all

but something is working, because the html template does get served

there shouldn’t be a literal resource folder inside jar

but the contents of the resource folder should be in the jar

where in the jar should the contents be?

i was looking under the folder named like my app

config looks good

change migration dir to “migrations/”

i don't need to prefix it with resources?

no, why would you need to do that?

contents of resources dir get included in jar at base folder

so you get a dir migrations in jar

i see great thank you

so when you try to read a file resources/migrations/a.sql it works when you run it locally, but in production there is no such file. This is why most things also try to load via classloader (e.g. io/resource ) in which case the actual path is “migrations/a.sql”

classloader loads from classpath, in your local dev environment your resources folder is on your classpath, in production the jar is on your classpath

wow that is so clarifying

i still have the same error as before, but we certainly just prevented another error i'd have after getting past this one

you can of course have any number of “resources” folders on classpath, any number of jars. JVM makes no special distinction for resources, it’s just a folder on classpath, same as src folder, the fact that you put noncode there and code in src is just your project structure and makes no difference to JVM, you could put your migration in `/src/migrations/a.sql” and set migration folder to “migrations/” and it would work in dev

so if i have two 'resources' folders on the classpath, and in each one i have a folder called 'x', what happens in the .jar file? merged?

no, it will load one, which one is undefined behaviour, but likely the one from the folder that appears on classpath the first, there is no guarantee from JVM spec. It’s the same situation if you have 2 jars on the classpath that supply the same class (e.g. two versions of Jackson).

i see

Normally in clojure people use uberjar

which already does this, because it includes just one copy of everything

you can tell uberjar to combine files and how to combine them

sounds hairy

very important for duct-hierarchy.edn

in lein project:

:uberjar-merge-with {"duct_hierarchy.edn" [(fn [in] (clojure.edn/read-string (slurp in)))
                                             clojure.core/merge
                                             (fn [out datum] (spit out (prn-str datum)))]}

since you need to have 1 duct-hierarchy.edn file that is a combination of all of them

cool

any other ideas about what may be causing the error I'm having?

btw i'm about to leave the computer for around an hour+ if i don't response

thanks for all your help, this has been enlightening

I stumbled across this, too. My thought was: Why doesn’t it make a simple identity (pointer comparison) as a first check?

Some reasons I can think of right now: • Computing a vector's hash necessarily traverses the whole vector (and the hash is not always calculated - only when it's needed, and cached afterwards), whereas comparing two vectors might short-circuit on the first element or even on the length comparison • Using hashes would rely on all the items being good citizens and also using the hash semantics correctly, which might not always be the case (although it would probably become apparent by itself)

Although, which makes it all even more interesting, comparing a vector to a list does use hashes of the collections.

Worth a post on http://ask.clojure.org, I'd say. ;)

Hm… but then instead of using hashCode which requires that you always calculate hashcode on call, we could have a method called, hashCodeIfCached, which would use it if it was calculated already, otherwise return 0.

or _hasheq if cached

which is more likely to have been calculated, if the object was used as a map key at any time

An existing relevant thread, might make sense to add a comment there instead of creating a new post: https://ask.clojure.org/index.php/11124/persistent-collections-implement-equiv-more-efficiently?show=11124

equiv already does the identity check. I think clojure kind of assumes that you go through =

You can’t use hashes to check for positive equivalence because of collisions

You can use them for negative equivalence (fail fast) in some cases but keep in mind that the set of collection implementations is open

ham-fisted allows you to create hashtables with various hash and equals providers if you want to mess with these things. Also, I am open to changing the chunked vector implementation to whatever works best - I don't have the compatibility or stability requirements of core Clojure. For instance I think there is an argument for not using murmur at all for integers - the java hashmap impl has a clever work around and overally their impl is much faster in all tested cases. So if you want to explore this further - that is why I created ham-fisted. We can safely explore these questions and you can use the results in your program without changing the entire Clojure ecosystem.

Another thought is that I think there isn't a need for separate hasheq and hashCode impls - I haven't seen concretely a program that breaks if we just use hasheq for hashcode in our datastructures. Similar for equiv and equals although I imagine there are serialization instances. So another thing I question is whether it is necessary to have a parallel and slightly different arch for equals and equiv, hashcode and hasheq.

Again, ham-fisted is where you can test this out and try it in your systems in a limited sense without needing nearly the rigour we need if we want to force these changes on the entire ecosystem.

it is necessary - algorithms for Java colls are defined in the collections lib so we need to match those for hashcodes

and similarly, clojure's notion of equivalence is different than Java's

I think the first is in question for me. It isn't clear to me why a persistent hash map must have the same hash, equals as a java hashmap or persistent vectors and arraylists but I can definitely respect the decision. I think clojure's equiv pathway overall makes a lot more logical sense than java's rather fragile equals pathway. I think mumur3 for integers in general isn't necessary - java in general just returns the int itself or the long xoring the high bits into the low bits and as I said the java hashmap also does a mask of high bits into low bits to make sure the high bits participate in hashmaps that aren't gargantuan. In any case, there are easy wins in just the Util.equiv without changing any definitions and this tends to be the slow path in lots of stuff. One thing I noticed is that special casing long-long and double-double comparisons helps quote a lot and doesn't risk breaking other things - https://github.com/cnuernber/ham-fisted/blob/master/java/ham_fisted/CljHash.java#L36.

java's collection api defines things that must be true about implementations of the collection interfaces, such as: https://docs.oracle.com/javase/8/docs/api/java/util/Map.html#hashCode--

we wish Clojure maps to be good citizens in the Java world in this respect so that Clojure maps can (invisibly) acts as java.util.Maps if passed to such interfaces

we found the integer thing was problematic because it was trivial to get hashcode collisions for combinations of positive and negative integers (but changing to murmur on hashed colls may have made that a non-issue). we did look at re-hashing, can't remember what the decision was based on (that was almost a decade ago now so it's a little fuzzy, and I think Rich did some of that work by himself)

Yes, if two objects are equal, they must have same hash, that’s the contract. Of course this really impossible to guarantee in the java world. You immutable collection might have an element that is a java object that can be mutated and returns a different hashcode every time. In general, map implementations ignore this possibility, which leads to “fun” things e.g.: Let’s say you have class X that has a single int member that can be mutated via a setter and hashCode for X just returns the member (actual implementation doesn’t matter, any hashcode implementation relying on the mutable member exhibits same problem.) You can actually insert the same object (instance of X) into HashSet 3 times, if you just change the int member to 1, 2 and 3 before each .add call. Hashcode changes and it lands in different, empty bucket, so it gets added. If it happens to land in the same bucket where it’s already there, it will not be added, because .equals will figure out that it’s the same object as the one in the bucket already. Fun stuff.

I've been wondering about this too. It seems to work correctly but there is some performance penalty for ThreadLocals with "big objects" but what exactly they (JVM docs) mean by this, isn't clear and I couldn't repro any case so far where it matters

Perhaps @U050ECB92 ?

thread binding means each thread have such a var

it's not virtual thread, it's native thread

@U01125FLA02 yes, but there is an interaction between real threads and virtual threads

https://blogs.oracle.com/javamagazine/post/java-loom-virtual-threads-platform-threads

That is a great article. The advice at the bottom of the article - use semaphores instead of thread pools and use reentrant locks instead of syncrhonized (`locking` in clojure) - those are somewhat architectural changes. I wonder if we can leverage the Clojure compiler to make those changes a bit more automatic.

It appears the advice against thread locals is simply that if you can have millions of threads the cost of each thread local is much greater so be careful and attempt to limit them. Specifically relating to binding I think that uses 1 thread local (https://github.com/clojure/clojure/blob/e6fce5a42ba78fadcde00186c0b0c3cd00f45435/src/jvm/clojure/lang/Var.java#L71) so I think you are fine aside from the normal perf hits you take from dynamic variables.

Related - scoped local variables - https://openjdk.org/jeps/429

well make up your mind - do you want dynamic typing or not? :)

if you want to avoid reflection, then you need to pick a lane

if you want to support multiple types, then you need to explicitly check and type hint each path

the runtime check is generally much faster than the reflective call

(maybe not true in very recent jvms that are replacing reflection impl with indy)

I only care because I've had some errors within this code path and the reflection calls obfuscate the source of the error in the stack

and because it's a fun rabbit hole to fall down lol

the jvm can generally optimize this kind of conditional well, esp if it's usually one type (monomorphic)

cool, thanks

out of curiosity, how would that compare to using a protocol? something like:

(defprotocol IDecode
  (decode [this]))

(extend-protocol IDecode
  String
  (decode [s]
    (.decode (java.util.Base64/getDecoder) s)))
(extend (Class/forName "[B")
  IDecode
  {:decode (fn [^bytes b]
             (.decode (java.util.Base64/getDecoder) b))})

https://github.com/puredanger/poly-timing

that doesn't have an explicit if test but you can get an idea

Wow, those timings are wild. Thanks for the link!

> (maybe not true in very recent jvms that are replacing reflection impl with indy) what's this?

ah, invoke dynamic

see https://openjdk.org/jeps/416

I think that's in Java 18?

@U7RJTCH6J that’s what i would reach for

Does https://openjdk.org/jeps/416 mean that Clojure code can get potentially significant speedups? (in the JEP they seem to be talking about reducing the maintenance and development cost )

Don’t know, maybe

If you have a handful of known types then dispatching by using instanceof is still way faster than reflection.