Fork me on GitHub

@aria2 - a few people who hang out in #data-science have some new things involving just that use case (Jupyter)


(I haven't tried it, ymmv)


I want to update values in a map of any depth. Specifically any value that contains the key :db/ident I want to replace with the value of :db/ident


{:a {:b {:db/ident :enumerated/value}}} -> {:a {:b :enumerated/value}}


(clojure.walk/postwalk #(if (and (map? %) (:db/ident %)) (:db/ident %) %) {:c {:b {:db/ident :relation/spouse}}})


Still need to clean it up a bit but that’s basically it


@caleb.macdonaldblack with specter it's:

(def ALL-MAPS (recursive-path [] p (stay-then-continue map? MAP-VALS p)))
(transform [ALL-MAPS #(contains? % :db/ident)] :db/ident data)


a lot faster than doing a postwalk


you can also re-use ALL-MAPS for doing queries or other transformations on that kind of data structure


@nathanmarz Sorry I didn’t mention this in my problem. My data structure contains a map in a vector.


The postwalk works with vector or map and takes about a millisecond.


I couldn’t get spector to work with a vector as well but it took about 5 milliseconds even when it didn’t work


@caleb.macdonaldblack running just once in time is not a good way to benchmark


nothing has a chance to optimize the code (both specter and the jvm)


this is better:

user=> (time (dotimes [_ 1000000] (clojure.walk/postwalk #(if (and (map? %) (:db/ident %)) (:db/ident %) %) data)))
"Elapsed time: 5055.333616 msecs"
user=> (time (dotimes [_ 1000000] (transform [ALL-MAPS #(contains? % :db/ident)] :db/ident data)))
"Elapsed time: 522.451801 msecs"


that shows 10x performance difference


it's easy to make modify ALL-MAPS to handle whatever structure your data has


I also find it's better to be exact about what you're targeting to manipulate – postwalk can lead to surprising bugs like descending into records, keys, etc.

Victor Ferreira01:01:58

Hey guys, what you think about DDD in clojure?


Does anyone know whether Clojure ended up using invokedynamic? All I can find from googling and searching the mailing list is discussion about it from years ago, during which Rich seemed lukewarm.


Not it hasn’t and that’s due to a few things


Invokedynamic had some nasty bugs in older jvms


And today this would require adding runtime switches to enable or disable invokedynamic depending on the jvm. Support for compiler switches is fairly new.


But in the end, Clojure isn’t as dependent on invokedynamic due to it being mostly statically typed. That is to say protocols already use interfaces so they’re already quite fast


Not that invokedynamic would be useless, it not super critical


Thanks @U07TDTQNL. I suspected as much. Tbh, my understanding of how invokedynamic works is very shaky. I get the sense from talking to people across JVM languages that it maybe didn't quite pan out as expected.


The main benefit I was imagining for Clojure would be closing the performance gap between protocols and multimethods, even if that is of relatively little importance to me personally.

Alex Miller (Clojure team)02:01:03

There are several places where invokedynamic could be very useful and I think in late java 8 it’s finally reliable enough to use (without catastrophic edge cases). I have no doubt we will eventually explore it for things like multimethods, lazy var initialization, etc.


Thanks @U064X3EF3. I never thought about it being useful for lazy-seqs (I'm imagining those generated through recursive calls using lazy-seq directly, which I do use often enough), but that makes complete sense.


Can you think of anything I can read on invokedynamic more recent than the speculative blog posts from around when it was introduced? As mentioned, I hardly understand how it actually works and would like to.


I don't think he meant lazy-seqs, but lazily loading vars, maybe for improving startup time


That doesn't make sense to me. I assumed Alex meant actual lazy vars, i.e. the thunks that make up lazy-seqs.


@U4R5K5M0A the more I think about it you may be right, although I doubt it would improve startup time.


@U2TCUSM2R here’s documentation that lazy var loading is a startup time optimization

Alex Miller (Clojure team)18:01:57

Lazy vars do improve startup time noticeably if you’re loading AOT code (esp in combination with direct linking). Some data is at the end of (patch is an attachment on that page too) - improvements are 25-30%. The downside is that you also make all subsequent var accesses slower (as you have to go through a guard). invokedynamic has the ability to actually remove the guard once you load, which mitigates that issue completely.


Woah, thanks guys. I see you what you meant now @U064X3EF3.


@nathanmarz Thanks, I wasn’t aware of benchmarking that way. I can see the specter transform is clearly faster than postwalk now. I’ll read a little more into specter and try to understand how it works.


I struggle to pass a pattern for a hash-map to clojure.core.match/match. This examples shows would I would like to do (pass pattern as a variable):

(let [pattern {:a 3 :b 4}
      x {:a 3 :b 4 :c 5}]
  (clojure.core.match/match [[x]]
    [pattern] :foo))

;; => IllegalArgumentException No matching clause: [{:a 3, :b 4, :c 5}]  server.repl/eval69751 (form-init673047409690541468.clj:3)


How can this be done? I tried a couple of variants, especially around using more/less square brackets, but nothing worked so far.


core.match is macro-based, so you need to pass a literal hash-map I think


ie try inlining both variables


Yeah that is the exact problem. However, I still want to pass it in. Is there a way to achieve that?


you’ll have to quote it and expand, something like (untested):

(let [pattern {:a 3 :b 4}
      x {:a 3 :b 4 :c 5}]
  `(clojure.core.match/match [~x]
    [~pattern] :foo))


and you might have to turn the whole into a macro or eval it


Alright, thanks


I have a simple program:

(ns myprogram.core

(defn -main []
  (pmap identity [1 2 3]))
that hangs with lein run. If I exchange pmap with map, it exists correctly. Should I somehow clean up after pmap?


I think pmap uses the agents threadpool, so you need to call shutdown-agents on exit:


thanks! 🙂


How come reify is so much faster than wrapping in a fn here:

(let [f0 (reify IFn
           (invoke [_ x] (Integer/parseInt x)))
      f1 (fn [x] (Integer/parseInt x))]
  (doseq [f [f0 f1]]
    (time (dotimes [_ 1000000000] (f "1")))))
Shouldn't they be pretty much identical?


they compile to different bytecode


f0 compiles into something like public final class ricochet_robots.core$reify__12494 implements clojure.lang.IFn, clojure.lang.IObj while f1 is public final class ricochet_robots.core$f1 extends clojure.lang.AFunction


@schmee Yeah I did that. But there isn't a whole lot of difference. They both just implement a class that defines the .invoke(Object x) in the same manner. Reify is straight class implementing IFn and the function will exten AFunction


Yeah 🙂


I’m looking at it now and I can’t see much difference either ¯\(ツ)


But still, a huge 5x difference just because of extending from a function? I'm surprised...


I guess that the JVM optimizes them differently because one is interface dispatch and one is not


Yup, seems like it. Just surprised that it's such a big difference...


I really want to learn more about this stuff…


My hunch is the JVM inlines the one call but not the other... But then, checking that would require some JVM-fu.


if you figure it please let me know!


btw what JVM flags are you running with?


the problem is in the bench. try changing the order, the fastest is always the first to run


ehh, when I do this, the results are reversed…

(def f0
  (reify clojure.lang.IFn
     (invoke [_ x] (Integer/parseInt x))))

(defn f1 [x] (Integer/parseInt x))

(defn asdf []
  (time (dotimes [_ 1000000000] (f0 "1")))
  (time (dotimes [_ 1000000000] (f1 "1"))))


haha @leonoel, right on time! 😄


Damn 😄


okay, so now the question becomes, why is that? 🙂


@schmee Looks like the JVM first optimizes the call site for exactly one class and can dispatch right to the implementation (or inlines it). Once seen a second class it deoptimizes.


Doing (doseq [f [f0 f0 f0]]) (good), vs (doseq [f [f0 f1 f0 f1]]) (bad), (doseq [f [f0 f0 f0 f1]]) (bad after 3rd)


I thought the JVM can easily handle 2-3 different classes and only becomes megamorphic after that... But maybe that was JS and not the JVM.


yeah, I thought the same


Howdy folks, I'm struggling with a very strange exception when writing midje tests: java.lang.ClassCastException: clojure.lang.AFunction$1 cannot be cast to clojure.lang.IFn$OLOO . I'm deeply confused, does anyone know what it means?


@rauh haha, running a REPL with -XX:+PrintCompilation is… interesting 🙂


did you redefine the function


Yes, I have a few provided functions, that's probably where the issue reside since the code works in normal conditions (not under midje)


@rborer IIRC you should redefine them with the same primitive type hints. In your instance the second arg (the L in OLOO) is a primitive long (the other 1st/3rd are objects, return is object)


@rauh good point! Although I'm not sure it's doable with midje


yep, so with-redefs only, let me try


works, thanks @rauh!


How would I do this in Clojure?

new AttributeProviderFactory() {
    public AttributeProvider create(AttributeProviderContext context) {
        return new ImageAttributeProvider();


proxy if AttributeProviderFactory is concrete, reify otherwise


if it's a java interface, it's not, otherwise it is concrete


Nice, I got it to work:

(def testfac
  (proxy [AttributeProviderFactory] []
    (create [context]
      (reify AttributeProvider
        (setAttributes [_ node tag-name attrs]
          (prn node))))))
thanks @ghadi! Also Clojure is so concise, this is the Java version of what I’m doing: ❤️


@martinklepsch Just in case you've missed it, have you seen ?


Clojure is better at doing Java than Java is at doing Java 😄


until you need to do things with numbers maybe


what’s the best channel to discuss cljfmt related issues?


a veteran like you should know better by now than asking where to ask :)


haha - trying to be more careful about using the right channels, anyway turns out there’s a known bug in cljfmt for indenting letfn and a workaround is documented in the github issue


noisesmith: I’ve been working on a cljfmt fork that does a lot more than the original version. Still a few outstanding bugs to clean up, but may solve your problem:


currently [mvxcvi/lein-cljfmt "0.7.0-SNAPSHOT"] on clojars


@U8XJ15DTK nice, will we get parallel fix and check there? cljfmt's maintainer ignored my PR for that 😞


@U5LPUJ7AP parallel fix sounds good to me, but parallel check seems like it would interleave diff output


I could imagine a nice compromise where the actual checking is parallel, and the printing of the check output is owned by a single thread


well, maybe my PR does that, but it's not difficult to avoid that, just put the diffs on a channel and print them in a go-loop


so basically what noisesmith said 😛


I am open to said PR. 🙂 At this point, my fork is ~3x the size of the original cljfmt, so I’m not sure if merging back upstream will be viable. We’re using the current snapshot release at work though, so it’s seen some battle testing. You should be able to ignore any forms that it misbehaves on, anyway. (another new feature!)


will take a look when I have the time


I suggest #cljfmt I’d be very interested in following and probably participating in an active cljfmt fork. It looks like James doesn’t have a whole lot of time for the project at present.


noisesmith: I’ve been working on a cljfmt fork that does a lot more than the original version. Still a few outstanding bugs to clean up, but may solve your problem:


Has anyone written a ring test-runner for clojure.test tests?


I’ve made a ring handler that reloads all my test files and shows the results in a div in the frontend


It seems like this would be a cool way of deploying little executable system checks to all your environments (whitelisting on production only "production safe" tests). All those ad-hoc scripts you write and paste into your ssh session to inspect the health of the system when something has gone wrong can be encoded as clojure tests and run at the click of a button. Does anyone do this?


I would differentiate health checks from unit tests, both are useful


I haven’t considered using clojure.test to implement health checks though


or, put differently, I don’t run ad-hoc scripts to inspect the health of the system, I have health check endpoints on the http server that allow that (it is accessed by an automated system that sends us alerts if the checks don’t respond, or if they fail, etc.)


Yeah, they just have to be automated somewhere.


a transducer question. I have a long chain of comped transducers doing various map and filter operations where the data once it gets applied will be a sequence of maps. Now I would like for one of the comped transducer steps to assoc a “ranking” value to the maps where the ranking is that map’s index if sorting the full sequence of maps based on the value of one of the keys. I get the feeling this makes the ranking operation unsuitable for my transducer chain since it’s no longer “step based” but rather operates on the whole sequence of maps, but I’m still noob with transducers. Am I correct in assuming it would be a bad fit to try to do this kind of ranking using a transducer in the existing chain?


I guess I could do a custom stateful transducer with a completion that does the ranking…


That seems not great - what you’re describing requires fully realizing the collection first, so it’s not really a (per-element) transformation


yeah, agreed, the more I think about it the worse it seems


sounds more like a transduce that happens to use the transducer you’ve defined


that has all the pieces you need - building up a result step by step, final transformation before returning the value, not lazy


yeah that more or less chimes with what I suspected. Thanks for the sanity check!