never heard of this. However it is more common to do it the other way around, execute JS/CLJS from the JVM via Nashorn.

@U01EFUL1A8M thanks for the recommendation, it's to call some java from an electron app, do you know if it is possible via Nashorn? Also Nashorn seems to have been deprecated in favor of GraalJS

@UGH9KH1DF Sorry I cannot help you there!

Not sure how that works but I see no reason it wouldn't work with Clojure too, being that Clojure is just Java

Though might either need to AOT compile the Clojure, or use the Clojure Java APIs maybe

@U0K064KQV I managed to make it work it was pretty straightforward in the end, and indeed it needs AOT compile

If you care about performance, don't use merge* • terms and conditions may apply

generally, something like (persistent! (reduce-kv assoc! (transient to) from) is faster

you can also try without the transient, i.e. (reduce-kv assoc to from)

thanks @UK0810AQ2 , even those solutions suffer from the same problem even going slightly faster. I could do to try and make use of the hash-map first if it’s there sometimes:

(defn merge*
    [m1 m2]
    (if (> (count m1) (count m2))
      (reduce-kv assoc m1 m2)
      (reduce-kv assoc m2 m1)))

it might not be out of the question to avoid map literals and use the related constructors directly? then you can sprinkle specs for ensuring your perf expectations will hold

You mean using (hash-map) ? Have tried, doesn’t help

(c/quick-bench (merge (hash-map :a 1 :b 6) {:a 1 :b 2 :c 3 :d 4 :e 5 :f 6 :g 8 :o 9 :l 10 :v 11})

Evaluation count : 513588 in 6 samples of 85598 calls.
             Execution time mean : 1.168965 µs
    Execution time std-deviation : 7.250814 ns
   Execution time lower quantile : 1.161225 µs ( 2.5%)
   Execution time upper quantile : 1.178713 µs (97.5%)
                   Overhead used : 2.144573 ns

I think I will just be careful with merge and try not to abuse it, maybe have a bit more verbose code sometimes and use map literals

Your solution based on count is unfortunately incorrect when keys overlap 😞

If you know all the keys in advance and the merge is of literal maps unroll it to a series of assoc-s

also, is your repl started with lein?

True, it will fail on overlapping keys. No, I use Calva with deps.edn.

Just to make sure

The assoc-s have the same problem, 1 assoc takes 60ns but at the moment where the map is 8 keys length, the next assoc takes 500ns

(import '(java.lang.management ManagementFactory RuntimeMXBean))

(into [] (.getInputArguments (ManagementFactory/getRuntimeMXBean)))

["-Dclojure.basis=.cpcache/881021756.basis"]

alright, you're running with full JIT

> the next assoc takes ~500ns That's because you're growing from an ArrayMap to a HashMap

You can unroll everything into assoc! into a transient empty hash map

and make sure it's a hash map to begin with

yeah that seems to be the problem in all cases, that moment when it needs to grow

If you start off with a hash map you won't pay that cost

but you might get worse performance for small maps as a result of the merge

> alright, you’re running with full JIT is this good or bad for performance testing? Not very familiar with those bits

Not only good, mandatory

unfortunately (hash-map) returns {} which is a persistenArrayMap. Any way to create an empty hash-map? directly from the class?

Yes, writing an example now

Try this

(defn merge2 [m1 m2] (persistent! (reduce-kv assoc! (reduce-kv assoc! (transient clojure.lang.PersistentHashMap/EMPTY) m1) m2)))

The "problem" is that when growing from ArrayMap to HashMap you're basically running the inner form of this function. This implementation only iterates over each map once

I spent lots of time trying to optimize merge, I don't like any implementation I managed to come up with, yet

yeah, it improves the performance for that specific moment of growing, but seems to be worse overall

last bit of black magic

(defn -merge [m1 m2] (if (< 8 (+ (count m1) (count m2))) (merge2 m1 m2) (reduce-kv assoc m1 m2)))

Maybe with transient for the second case, too, not sure

(c/quick-bench (-merge {:a 1 :b 2 :c 3 :d 4 :e 5 :f 6 :g 8 :o 9 :l 10 :v 11} {:a 1 :b 6}))

Evaluation count : 1381080 in 6 samples of 230180 calls.
             Execution time mean : 436.719037 ns
    Execution time std-deviation : 17.330955 ns
   Execution time lower quantile : 428.597133 ns ( 2.5%)
   Execution time upper quantile : 466.822495 ns (97.5%)
                   Overhead used : 2.144573 ns

This one runs much slower though

Yeah, it's the math, hang on I'll open a real editor

Need to decompile

It’s interesting because I would think that if the problem was the moment of transforming from array-map to hash-map, something like

(merge clojure.lang.PersistentHashMap/EMPTY m1 m2)

yuck

(defn merge-meh
  [m1 m2]
  (persistent!
   (reduce-kv
    assoc!
    (reduce-kv
     assoc!
     (transient clojure.lang.PersistentHashMap/EMPTY)
     m1)
    m2)))

(defn merge-ok
  [m1 m2]
  (persistent!
   (reduce-kv
    assoc!
    (transient m1)
    m2)))

(defn merge2
  [m1 m2]
  (let [c1 (count m1)]
    (if (> c1 8)
      (merge-ok m1 m2)
      (if (< 8 (unchecked-add c1 (count m2)))
        (merge-meh m1 m2)
        (merge-ok m1 m2)))))

but it's fast =\

yeah

that’s quite good, like 0.6 of the bad scenario and keeps the performance of the good scenario

And that's before doing any interop or any ugly tricks

thanks @UK0810AQ2 it has been interesting 🙂 I think I will still be cautious when to use merge and probably keep using the core one, but I will keep this around just in case!

yeah…I wanted to avoid as much as possible ugly tricks or hacks, I’d rather be cautious on my uses of merge than abusing a hacky implementation

yeah it's easier to not use merge on a hot path

Taking some inspiration from the faster fast-merge issue on clj-fast, this one covers the case where one of the other might already be persistent hash maps to use them as the “source” to merge into, still “slow” when merging two array-maps that become a hash-map, and it only accepts 2 maps

(defn rassoc! [^clojure.lang.ITransientAssociative acc k v]
    (if-not (acc k)
      (.assoc acc k v)
      acc))

  (defn lassoc! [^clojure.lang.ITransientAssociative acc k v]
    (.assoc acc k v))

  (defn rmerge!
    [l  r]
    (if (instance? clojure.lang.IKVReduce l)
      (.kvreduce ^clojure.lang.IKVReduce l  rassoc! r)
      (clojure.core.protocols/kv-reduce l  rassoc! r)))

  (defn lmerge!
    [l  r]
    (if (instance? clojure.lang.IKVReduce r)
      (.kvreduce ^clojure.lang.IKVReduce r  lassoc! l)
      (clojure.core.protocols/kv-reduce r  lassoc! l)))

  (defn merge! [m1 m2]
    (if (instance? clojure.lang.PersistentHashMap m1)
      (persistent! (lmerge! (transient m1) m2))
      (persistent! (rmerge! m1 (transient m2)))))

https://github.com/bsless/clj-fast/issues/1

As you see, I've been at it for a while 😄

yes haha even if I could live with the performance penalty or be cautious about it, difficult to not enter the rabbit hole

will leave the above code in the issue in case someone finds it useful

re: the order of maps You shouldn't be surprised by big into small being slower than small into big. Why? You need to iterate over the map!

Iterating over 10 keys is slow than than iterating over 2 keys

that makes sense, yeah

We could do something clever, though

I’ll change my impl

instead of checking persistenthashmap

always use as source the bigger map

no matter the type

no, that brings you back to the problem of key collision

(defn merge! [m1 m2]
    (if (< (count m1) (count m2))
      (persistent! (rmerge! m1 (transient m2)))
      (persistent! (lmerge! (transient m1) m2))))

in combination with the rmerge! or lmerge! works fine

What we really need is an "update" method in the map which can handle collisions explicitly

how are lmerge and rmerge implemented?

(defn rassoc! [^clojure.lang.ITransientAssociative acc k v]
    (if-not (acc k)
      (.assoc acc k v)
      acc))

  (defn lassoc! [^clojure.lang.ITransientAssociative acc k v]
    (.assoc acc k v))

  (defn rmerge!
    [l  r]
    (if (instance? clojure.lang.IKVReduce l)
      (.kvreduce ^clojure.lang.IKVReduce l  rassoc! r)
      (clojure.core.protocols/kv-reduce l  rassoc! r)))

  (defn lmerge!
    [l  r]
    (if (instance? clojure.lang.IKVReduce r)
      (.kvreduce ^clojure.lang.IKVReduce r  lassoc! l)
      (clojure.core.protocols/kv-reduce r  lassoc! l)))

right, rassoc is what I wanted to avoid

I don't want to pay for lookup

no choice though

unless we get something like update

true, but update would always create the given key even if it didn’t existed

realised that is fine on rmerge, so either take right key or left key

(into {} cat [m1 m2 m3 ...]) should be faster once it uses transient

@U2J4FRT2T try it with two maps with sizes 4 and 5 with no intersection 🙂

I wouldn't say it's recommended. It's an option, recommendations don't exist yet I would say

Personally, I'd recommend you publish your libraries to Maven Central

That's the friendliest to legitimate professional and enterprise consumption

Because people can now pull it from a Java or Scala project, and they can easily mirror maven central internally as well.

I'm not sure anything is "easily solved" with git submodules or subtrees 😛

How so?

I'm comfortable with git submodules, but I worked with plenty of teams that stumble over git submodules. Support for submodules has improved, but it does add complexity. tldr: I'm just poking fun at git

The plugins/tools could get up to speed, sure. Especially given that tools.build is made with the reuse in mind.

Or it could become a part of Java app's build process - again, by reusing tools.build.

That seems like a huge difference between an artifact on maven vs. trying to consume a git only library. Presumably, the artifact on maven would already work, out of the box, with all of these tools.

Difference - of course. Although making a Git dep work in a Java app would likely by itself be a reusable process, so it's not that huge. Definitely not enough to make Git deps an island.

git deps is best, imo, for tools and utilities

for libs, I think there is a lot of benefit to being available to a wider set of existing tools via Maven-located artifacts

git deps are also very useful to try things during dev - you can put a fix on a branch and have someone try a sha before you apply a fix in the main release (I do this all the time)

I've seen usages of Git deps for libraries in cases where some ad-hoc patching is needed or a version in Maven hasn't been updated. Seems like it's a similar argument to using e.g. Flatpak/AppImage/Snap vs using your OS's software distribution system. Both have pros and cons.

Right, I'm excited about the new features clojure's cli is getting. I've used git deps and local deps in situations that would have been awkward otherwise. Just trying to point out that there are trade-offs vs using something like clojars.

yep, agreed

All of the things we currently use as git deps at work are dev/test pipeline stuff. For anything that goes into our deployable artifacts, we use Maven/Clojars deps. That's more happenstance right now than by design but I figured I'd offer that as anecdata 🙂

Yeah I agree with that sentiment. Git deps for tools and active development, releases for libraries intended for use in production applications

Glad everyone agrees that libs are better published in a Maven-repo. My preference for Maven Central is simply because it tends to be already setup for all JVM build tooling and mirroring systems. Assuming a company where Clojure is not the dominant language, but Java is, and the company has an internal Artifactory mirror, you'd need to convince leadership to add the Clojar repo as well to be mirrored. But this is much better already then git deps, where you'd ask leadership to build a whole custom mirroring facility just because you've decided you prefer Clojure over Java? Or because you'd like to use a Clojure lib from Java.

I didn't say maven was strictly better. There are tradeoffs. It's worth considering the intended audience for your project.

> Glad everyone agrees that libs are better published in a Maven-repo. I don't agree -- and what I said was not agreement with your position: I was just noting that the production artifacts we build today do not rely on git deps. As I said, that's "more happenstance ... than by design". Our CI setup already has git available and does git-based operations (tagging etc) as well as the test/build tooling relying on git deps -- so it would not be problematic at all for our production artifacts to depend on such git deps.

To throw in my 2c, lein’s git plugin is a huge pain. It doesn’t handle transitive dependencies well, ssh key management is a frequent gotcha. At work we’re actively trying to stop using it. Just a little evidence that deploying libraries to maven repos is the most compatible option.

@U01EB0V3H39 or switch to the CLI and deps.edn instead of lein? 🙂

Easier said than done! I was intending to directly respond to the title comment: > Lein definitely has at least one plugin for working with Git dependencies

We switched from lein to boot nearly six years ago and then to deps.edn about three years ago. Very happy with that switch.

What's the argument for Git, beyond just that the library author doesn't have to build and publish to Clojar or Maven Central? I mean specifically for library publishing, I agree with everything Alex said with regards to its usefulness at dev time and for tooling its fine as well.

Can use the most recent version of the code. Can use any branch. Can use your own changes added on top.

Agreed, if you're a Clojure only shop (as in no other JVM language is used), for your own internal use, its nice not to have to setup a separate shared internal Maven repo. I was talking more for the Clojure OSS.

(also at @U02FU7RMG8M’s shop) I definitely spearheaded our headlong jump into git deps after becoming increasingly infuriated with the ongoing maintenance of our internal Nexus server and setting new devs up for signing. That said, automation makes it trivial to also deploy our versioned releases to something like Clojars or Maven Central and I agree it is probably best for the community at large to do so, the social choice.

With the recent addition of the checked tag + sha form of git deps though I will say that I feel solid sticking with git deps on our own projects.

I tried .set on the static compiler attribute.

(.set Compiler/COMPILE_FILES true)
Execution error (IllegalStateException) at erp12.clark-example.play/eval2822 (form-init13317944995014988813.clj:1).
Can't change/establish root binding of: *compile-files* with set

what are you trying to achieve?

Probably nothing good. Just experimenting at the moment. I want to grab the class files for the in-memory classes so that I can package them in a jar and ship them off to a cluster. I explained a bit more background in the comments here. https://stackoverflow.com/questions/69356231/get-or-write-class-files-for-the-classes-in-the-dynamicclassloader

Looks like this works, but I’m not sure if there will be any negative consequences

(.bindRoot Compiler/COMPILE_FILES true)

just use reflection to pull them out of the dynamic classloaders classCache

That will give the Class objects right? I’m not sure how to get the bytecode from those.

ah, of course, I was misrembering

In my case, I am working with an Apache Spark cluster. It has a mechanism to accept jars (or individual class files) and distribute them amongst the workers without restarting the JVMs. For Scala folks, their REPLs write class files for each command and ships that to the cluster. I would like to emulate that.

it may be better to ship clojure source

I thought about that too, but I wasn’t sure how to ensure that the Clojure source doesn’t reference something that isn’t accessible on the workers. For example, a fn defined outside the source sent to the cluster.

if you are entirely talking about a repl, then everything is either something on the classpath (and gettable as a resource) or created by evaluating code in the repl, so just capture the code being read in at the repl

the type clojure uses for *in* by default at the repl got some features that can help capture that https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LineNumberingPushbackReader.java#L47-L59

Hmm… That’s an interesting idea. I need to solve this problem when running the application with clj or IDE or whatever as well but that still gives me something to think about.

There are tools people have written for aws lambdas that also might be good to look at