Fork me on GitHub
#clojure
<
2018-01-25
>
jsa-aerial00:01:30

@aria2 - a few people who hang out in #data-science have some new things involving just that use case (Jupyter)

ghadi00:01:51

(I haven't tried it, ymmv)

caleb.macdonaldblack01:01:22

I want to update values in a map of any depth. Specifically any value that contains the key :db/ident I want to replace with the value of :db/ident

caleb.macdonaldblack01:01:58

{:a {:b {:db/ident :enumerated/value}}} -> {:a {:b :enumerated/value}}

caleb.macdonaldblack01:01:28

(clojure.walk/postwalk #(if (and (map? %) (:db/ident %)) (:db/ident %) %) {:c {:b {:db/ident :relation/spouse}}})

caleb.macdonaldblack01:01:41

Still need to clean it up a bit but that’s basically it

nathanmarz01:01:59

@caleb.macdonaldblack with specter it's:

(def ALL-MAPS (recursive-path [] p (stay-then-continue map? MAP-VALS p)))
(transform [ALL-MAPS #(contains? % :db/ident)] :db/ident data)

nathanmarz01:01:56

a lot faster than doing a postwalk

nathanmarz01:01:39

you can also re-use ALL-MAPS for doing queries or other transformations on that kind of data structure

caleb.macdonaldblack01:01:18

@nathanmarz Sorry I didn’t mention this in my problem. My data structure contains a map in a vector.

caleb.macdonaldblack01:01:26

The postwalk works with vector or map and takes about a millisecond.

caleb.macdonaldblack01:01:17

I couldn’t get spector to work with a vector as well but it took about 5 milliseconds even when it didn’t work

nathanmarz01:01:24

@caleb.macdonaldblack running just once in time is not a good way to benchmark

nathanmarz01:01:43

nothing has a chance to optimize the code (both specter and the jvm)

nathanmarz01:01:49

this is better:

user=> (time (dotimes [_ 1000000] (clojure.walk/postwalk #(if (and (map? %) (:db/ident %)) (:db/ident %) %) data)))
"Elapsed time: 5055.333616 msecs"
user=> (time (dotimes [_ 1000000] (transform [ALL-MAPS #(contains? % :db/ident)] :db/ident data)))
"Elapsed time: 522.451801 msecs"

nathanmarz01:01:55

that shows 10x performance difference

nathanmarz01:01:24

it's easy to make modify ALL-MAPS to handle whatever structure your data has

nathanmarz01:01:12

I also find it's better to be exact about what you're targeting to manipulate – postwalk can lead to surprising bugs like descending into records, keys, etc.

Victor Ferreira01:01:58

Hey guys, what you think about DDD in clojure?

sophiago01:01:21

Does anyone know whether Clojure ended up using invokedynamic? All I can find from googling and searching the mailing list is discussion about it from years ago, during which Rich seemed lukewarm.

tbaldridge02:01:09

Not it hasn’t and that’s due to a few things

tbaldridge02:01:28

Invokedynamic had some nasty bugs in older jvms

tbaldridge02:01:50

And today this would require adding runtime switches to enable or disable invokedynamic depending on the jvm. Support for compiler switches is fairly new.

tbaldridge02:01:02

But in the end, Clojure isn’t as dependent on invokedynamic due to it being mostly statically typed. That is to say protocols already use interfaces so they’re already quite fast

tbaldridge02:01:29

Not that invokedynamic would be useless, it not super critical

sophiago02:01:36

Thanks @U07TDTQNL. I suspected as much. Tbh, my understanding of how invokedynamic works is very shaky. I get the sense from talking to people across JVM languages that it maybe didn't quite pan out as expected.

sophiago02:01:29

The main benefit I was imagining for Clojure would be closing the performance gap between protocols and multimethods, even if that is of relatively little importance to me personally.

Alex Miller (Clojure team)02:01:03

There are several places where invokedynamic could be very useful and I think in late java 8 it’s finally reliable enough to use (without catastrophic edge cases). I have no doubt we will eventually explore it for things like multimethods, lazy var initialization, etc.

sophiago03:01:15

Thanks @U064X3EF3. I never thought about it being useful for lazy-seqs (I'm imagining those generated through recursive calls using lazy-seq directly, which I do use often enough), but that makes complete sense.

sophiago03:01:24

Can you think of anything I can read on invokedynamic more recent than the speculative blog posts from around when it was introduced? As mentioned, I hardly understand how it actually works and would like to.

johnj05:01:14

I don't think he meant lazy-seqs, but lazily loading vars, maybe for improving startup time

sophiago05:01:39

That doesn't make sense to me. I assumed Alex meant actual lazy vars, i.e. the thunks that make up lazy-seqs.

sophiago05:01:37

@U4R5K5M0A the more I think about it you may be right, although I doubt it would improve startup time.

noisesmith18:01:26

@U2TCUSM2R here’s documentation that lazy var loading is a startup time optimization https://dev.clojure.org/display/design/Lazy+var+loading

Alex Miller (Clojure team)18:01:57

Lazy vars do improve startup time noticeably if you’re loading AOT code (esp in combination with direct linking). Some data is at the end of https://dev.clojure.org/display/design/Improving+Clojure+Start+Time (patch is an attachment on that page too) - improvements are 25-30%. The downside is that you also make all subsequent var accesses slower (as you have to go through a guard). invokedynamic has the ability to actually remove the guard once you load, which mitigates that issue completely.

sophiago18:01:23

Woah, thanks guys. I see you what you meant now @U064X3EF3.

caleb.macdonaldblack02:01:10

@nathanmarz Thanks, I wasn’t aware of benchmarking that way. I can see the specter transform is clearly faster than postwalk now. I’ll read a little more into specter and try to understand how it works.

nblumoe08:01:17

I struggle to pass a pattern for a hash-map to clojure.core.match/match. This examples shows would I would like to do (pass pattern as a variable):

(let [pattern {:a 3 :b 4}
      x {:a 3 :b 4 :c 5}]
  (clojure.core.match/match [[x]]
    [pattern] :foo))

;; => IllegalArgumentException No matching clause: [{:a 3, :b 4, :c 5}]  server.repl/eval69751 (form-init673047409690541468.clj:3)

nblumoe08:01:04

How can this be done? I tried a couple of variants, especially around using more/less square brackets, but nothing worked so far.

schmee08:01:58

core.match is macro-based, so you need to pass a literal hash-map I think

schmee08:01:30

ie try inlining both variables

nblumoe10:01:35

Yeah that is the exact problem. However, I still want to pass it in. Is there a way to achieve that?

schmee10:01:10

you’ll have to quote it and expand, something like (untested):

(let [pattern {:a 3 :b 4}
      x {:a 3 :b 4 :c 5}]
  `(clojure.core.match/match [~x]
    [~pattern] :foo))

schmee10:01:40

and you might have to turn the whole into a macro or eval it

nblumoe13:01:29

Alright, thanks

grav08:01:05

I have a simple program:

(ns myprogram.core
  (:gen-class))

(defn -main []
  (pmap identity [1 2 3]))
that hangs with lein run. If I exchange pmap with map, it exists correctly. Should I somehow clean up after pmap?

schmee08:01:11

I think pmap uses the agents threadpool, so you need to call shutdown-agents on exit: http://clojuredocs.org/clojure.core/shutdown-agents

grav13:01:40

thanks! 🙂

rauh08:01:40

How come reify is so much faster than wrapping in a fn here:

(let [f0 (reify IFn
           (invoke [_ x] (Integer/parseInt x)))
      f1 (fn [x] (Integer/parseInt x))]
  (doseq [f [f0 f1]]
    (time (dotimes [_ 1000000000] (f "1")))))
Shouldn't they be pretty much identical?

schmee08:01:18

they compile to different bytecode

schmee09:01:30

f0 compiles into something like public final class ricochet_robots.core$reify__12494 implements clojure.lang.IFn, clojure.lang.IObj while f1 is public final class ricochet_robots.core$f1 extends clojure.lang.AFunction

rauh09:01:59

@schmee Yeah I did that. But there isn't a whole lot of difference. They both just implement a class that defines the .invoke(Object x) in the same manner. Reify is straight class implementing IFn and the function will exten AFunction

rauh09:01:14

Yeah 🙂

schmee09:01:30

I’m looking at it now and I can’t see much difference either ¯\(ツ)

rauh09:01:53

But still, a huge 5x difference just because of extending from a function? I'm surprised...

schmee09:01:56

I guess that the JVM optimizes them differently because one is interface dispatch and one is not

rauh09:01:21

Yup, seems like it. Just surprised that it's such a big difference...

schmee09:01:26

I really want to learn more about this stuff…

rauh09:01:26

My hunch is the JVM inlines the one call but not the other... But then, checking that would require some JVM-fu.

schmee09:01:13

if you figure it please let me know!

schmee09:01:21

btw what JVM flags are you running with?

leonoel09:01:39

the problem is in the bench. try changing the order, the fastest is always the first to run

schmee09:01:41

ehh, when I do this, the results are reversed…

(def f0
  (reify clojure.lang.IFn
     (invoke [_ x] (Integer/parseInt x))))

(defn f1 [x] (Integer/parseInt x))

(defn asdf []
  (time (dotimes [_ 1000000000] (f0 "1")))
  (time (dotimes [_ 1000000000] (f1 "1"))))

schmee09:01:59

haha @leonoel, right on time! 😄

rauh09:01:22

Damn 😄

schmee09:01:24

okay, so now the question becomes, why is that? 🙂

rauh09:01:12

@schmee Looks like the JVM first optimizes the call site for exactly one class and can dispatch right to the implementation (or inlines it). Once seen a second class it deoptimizes.

rauh09:01:59

Doing (doseq [f [f0 f0 f0]]) (good), vs (doseq [f [f0 f1 f0 f1]]) (bad), (doseq [f [f0 f0 f0 f1]]) (bad after 3rd)

rauh09:01:06

I thought the JVM can easily handle 2-3 different classes and only becomes megamorphic after that... But maybe that was JS and not the JVM.

schmee09:01:10

yeah, I thought the same

rborer09:01:00

Howdy folks, I'm struggling with a very strange exception when writing midje tests: java.lang.ClassCastException: clojure.lang.AFunction$1 cannot be cast to clojure.lang.IFn$OLOO . I'm deeply confused, does anyone know what it means?

schmee09:01:52

@rauh haha, running a REPL with -XX:+PrintCompilation is… interesting 🙂

bronsa09:01:52

did you redefine the function

rborer09:01:13

Yes, I have a few provided functions, that's probably where the issue reside since the code works in normal conditions (not under midje)

rauh09:01:35

@rborer IIRC you should redefine them with the same primitive type hints. In your instance the second arg (the L in OLOO) is a primitive long (the other 1st/3rd are objects, return is object)

rborer09:01:34

@rauh good point! Although I'm not sure it's doable with midje

rborer09:01:47

yep, so with-redefs only, let me try

rborer10:01:58

works, thanks @rauh!

martinklepsch14:01:27

How would I do this in Clojure?

new AttributeProviderFactory() {
    public AttributeProvider create(AttributeProviderContext context) {
        return new ImageAttributeProvider();
    }
})

ghadi14:01:17

proxy if AttributeProviderFactory is concrete, reify otherwise

ghadi14:01:21

if it's a java interface, it's not, otherwise it is concrete

martinklepsch14:01:38

Nice, I got it to work:

(def testfac
  (proxy [AttributeProviderFactory] []
    (create [context]
      (reify AttributeProvider
        (setAttributes [_ node tag-name attrs]
          (prn node))))))
thanks @ghadi! Also Clojure is so concise, this is the Java version of what I’m doing: https://github.com/atlassian/commonmark-java#add-or-change-attributes-of-html-elements ❤️

maxt14:01:16

@martinklepsch Just in case you've missed it, have you seen https://github.com/yogthos/markdown-clj ?

schmee14:01:04

Clojure is better at doing Java than Java is at doing Java 😄

noisesmith18:01:36

until you need to do things with numbers maybe

noisesmith18:01:54

what’s the best channel to discuss cljfmt related issues?

bronsa18:01:37

a veteran like you should know better by now than asking where to ask :)

noisesmith18:01:10

haha - trying to be more careful about using the right channels, anyway turns out there’s a known bug in cljfmt for indenting letfn and a workaround is documented in the github issue https://github.com/weavejester/cljfmt/issues/73

greglook21:01:05

noisesmith: I’ve been working on a cljfmt fork that does a lot more than the original version. Still a few outstanding bugs to clean up, but may solve your problem: https://github.com/greglook/cljfmt

greglook21:01:21

currently [mvxcvi/lein-cljfmt "0.7.0-SNAPSHOT"] on clojars

joelsanchez21:01:55

@U8XJ15DTK nice, will we get parallel fix and check there? cljfmt's maintainer ignored my PR for that 😞 https://github.com/weavejester/cljfmt/pull/104

greglook21:01:41

@U5LPUJ7AP parallel fix sounds good to me, but parallel check seems like it would interleave diff output

noisesmith21:01:40

I could imagine a nice compromise where the actual checking is parallel, and the printing of the check output is owned by a single thread

joelsanchez21:01:56

well, maybe my PR does that, but it's not difficult to avoid that, just put the diffs on a channel and print them in a go-loop

joelsanchez21:01:02

so basically what noisesmith said 😛

greglook21:01:58

I am open to said PR. 🙂 At this point, my fork is ~3x the size of the original cljfmt, so I’m not sure if merging back upstream will be viable. We’re using the current snapshot release at work though, so it’s seen some battle testing. You should be able to ignore any forms that it misbehaves on, anyway. (another new feature!)

joelsanchez21:01:57

will take a look when I have the time

arrdem22:01:49

I suggest #cljfmt I’d be very interested in following and probably participating in an active cljfmt fork. It looks like James doesn’t have a whole lot of time for the project at present.

greglook21:01:05

noisesmith: I’ve been working on a cljfmt fork that does a lot more than the original version. Still a few outstanding bugs to clean up, but may solve your problem: https://github.com/greglook/cljfmt

cddr21:01:46

Has anyone written a ring test-runner for clojure.test tests?

noisesmith21:01:36

I’ve made a ring handler that reloads all my test files and shows the results in a div in the frontend

cddr21:01:52

It seems like this would be a cool way of deploying little executable system checks to all your environments (whitelisting on production only "production safe" tests). All those ad-hoc scripts you write and paste into your ssh session to inspect the health of the system when something has gone wrong can be encoded as clojure tests and run at the click of a button. Does anyone do this?

noisesmith21:01:23

I would differentiate health checks from unit tests, both are useful

noisesmith21:01:40

I haven’t considered using clojure.test to implement health checks though

noisesmith21:01:45

or, put differently, I don’t run ad-hoc scripts to inspect the health of the system, I have health check endpoints on the http server that allow that (it is accessed by an automated system that sends us alerts if the checks don’t respond, or if they fail, etc.)

cddr21:01:08

Yeah, they just have to be automated somewhere.

mbjarland22:01:14

a transducer question. I have a long chain of comped transducers doing various map and filter operations where the data once it gets applied will be a sequence of maps. Now I would like for one of the comped transducer steps to assoc a “ranking” value to the maps where the ranking is that map’s index if sorting the full sequence of maps based on the value of one of the keys. I get the feeling this makes the ranking operation unsuitable for my transducer chain since it’s no longer “step based” but rather operates on the whole sequence of maps, but I’m still noob with transducers. Am I correct in assuming it would be a bad fit to try to do this kind of ranking using a transducer in the existing chain?

mbjarland22:01:14

I guess I could do a custom stateful transducer with a completion that does the ranking…

greglook22:01:21

That seems not great - what you’re describing requires fully realizing the collection first, so it’s not really a (per-element) transformation

mbjarland22:01:01

yeah, agreed, the more I think about it the worse it seems

noisesmith22:01:53

sounds more like a transduce that happens to use the transducer you’ve defined

noisesmith22:01:26

that has all the pieces you need - building up a result step by step, final transformation before returning the value, not lazy

mbjarland22:01:03

yeah that more or less chimes with what I suspected. Thanks for the sanity check!