Fork me on GitHub
#clojure
<
2020-01-02
>
ribelo08:01:09

What is the current situation with core.async? I have the impression that this is not the preferred library by the community. Everyone seems to be recommending manilofd or promesa. Why?

hiredman08:01:50

I haven't seen any of that. core.async is pretty nice and gets stuff right that manifold misses. My major complaint is patches languishing in jira

hiredman08:01:58

(and I have never seen any chatter one way or another about promesa)

lmergen08:01:54

I recommend manifold because of the monitoring; core.async provides little control over the threadpools behind it, no monitoring, and it’s generally discouraged to use it for blocking operations. manifold allows you to control all the thread pools, control which call is executed on which pool, and generally feels more “real world compatible”. if these things do not interest you that much, chances are core.async is fine for your use case.

yonatanel09:01:11

I have used core.async and manifold in very real scenarios successfully.

hiredman09:01:13

And I have monitoring and metrics collecting on core.async's threadpool

dpsutton09:01:06

How do you do the metrics?

ribelo09:01:20

@hiredman metosin uses both promesa and manifold. You can see it at https://github.com/metosin/talvi. http://juxt.pro, grammarly, or http://deps.co also use manifold

ribelo09:01:10

I personally like and prefer core.async, but I was curious why so few people recommend it, and recommend everything else

hiredman09:01:32

I have code that looks at the threadpool and sends information to newrelic

hiredman09:01:26

I think for the core.async threadpool we just measure latency (by periodically throwing a task on it and timing it to completion)

👍 4
hiredman09:01:38

Talvi looks like it doesn't exist

ribelo09:01:45

STATUS: Pre-alpha, in design and prototyping phase.

hiredman09:01:36

What about that?

ribelo09:01:16

that's why I even know that talvi exists

hiredman09:01:16

Yeah, ok, there is a github repo with a list of libraries and a wishlist of what they want to be in the github repo

hiredman09:01:50

But like, as software, it doesn't exist

hiredman09:01:01

The repo is just that readme

hiredman09:01:25

Anyway, core.async is at least as nice as anything else. I haven't seen any evidence of a strong swell of opinion for anything. I will say aleph (which manifold is often used in conjunction with) is older so it is kind of built in to older stable stuff

hiredman09:01:28

Promesa appears to mostly be of interest to the cljs crowd, which is likely because of js promises

mpenet09:01:29

Aleph was mostly re-written when manifold was released

hiredman09:01:00

Which, good riddance, promises are gross so they belong with js

hiredman09:01:54

With js's generators you can write something like core.async pretty easily https://gist.github.com/hiredman/2271c48e1f036253ce37913abd3a680a

ribelo10:01:33

@hiredman I'm absolutely not claiming that manifold or promesa is better than core.async

ribelo10:01:07

I'm even glad you're saying the opposite. Like I said, I prefer core.async to everything else.

dazld10:01:22

one small problem with generators is that they are (iirc) synchronous

dazld10:01:54

can be good or bad depending on use case

borkdude13:01:23

on CI when AOT-ing a lib with lein uberjar I sometimes get an error:

Execution error (IllegalArgumentException) at sci.impl.vars/fn$G (vars.cljc:121).
No implementation of method: :unbind of protocol: #'sci.impl.vars/IVar found for class: sci.impl.vars.SciVar
This usually resolves itself when I re-trigger the build, but I'd like to fix it. The relevant file is this: https://github.com/borkdude/sci/blob/master/src/sci/impl/vars.cljc

jeroenvandijk13:01:18

Maybe the protocol is redefined by the previous uberjar or class files (cached under target?). Leiningen includes target in the classpath i believe

borkdude14:01:59

@U0FT7SRLP It runs: lein with-profiles +clojure-1.10.1 do clean, uberjar https://github.com/borkdude/sci/blob/master/script/compile

jeroenvandijk14:01:09

@U04V15CAJ yeah clean should fix that indeed

borkdude14:01:22

Looks like some race condition maybe

jeroenvandijk14:01:21

It could also happen if the protocol ns is loaded twice e.g. via reload, but i didn't see that in the code

vinai13:01:30

We are using [com.stuartsierra/component "0.4.0"]. When running (reset) or (reset-all) in the REPL an error is thrown:

#error{:cause "namespace 'foo.video' not found", :message "Syntax error compiling at (foo/video/test_provider.clj:1:1).",
The foo.video ns is only
(ns foo.video)

(defprotocol VideoProvider
  (embed-html [this video]
    "Return the HTML for embedding the given video on a page.
     Throws an ExceptionInfo if something goes wrong."))

(defmulti new-video-provider :provider)
The foo.video.test-provider ns is
(ns foo.video.test-provider
  (:require  [foo.video :as video]))

(defrecord TestVideoProvider [stub-html]
  video/VideoProvider
  (embed-html [this video] stub-html))

(defmethod video/new-video-provider ::provider
  [properties] (map->TestVideoProvider (dissoc properties :provider)))
The project compiles fine, it is only the reset that is failing. I can’t figure out why. Any idea what is wrong?

vemv13:01:41

Is something like

(set-refresh-dirs "src" "test")
correctly set beforehand?

vinai15:01:12

Thanks for the reply - sorry for getting back late, got pulled into meeting. Nothing is being set besides the component system map. However, this setup used to work fine. I’m not sure when it stopped working. I’ll try setting the dirs

vinai15:01:57

Hmm, no set-refresh-dirs is exposed by the com.stuartsierra.component ns…

vinai15:01:43

Oh no, wait, it’s from clojure.tools.namespace.repl

vinai15:01:01

They are set, yes:

(set-refresh-dirs "src" "dev")

vinai15:01:19

The foo.video.test-provider is part of the normal system, not the tests despite it’s name.

vinai15:01:56

We are using 0.2.3, the latest version is 0.2.4. I’ll try updating.

vinai15:01:18

well, that didn’t fix it

vemv15:01:52

Seems pretty weird. Maybe a different error happens before you get namespace 'foo.video' not found? Or maybe the error contains useful info within its stacktrace?

vemv15:01:15

...if it's occasional, (clojure.tools.namespace.repl/clear) can help

vinai15:01:57

I happens every time I reset

vinai15:01:14

I’ll have another look at the stacktrace, maybe I missed something.

vinai15:01:35

More strangeness! I’ve removed the namespace foo.video.test-provider and all references to it and the error just pops around to another one and so on. It doesn’t seem to be tied to any specific ns. Just happens to be that one. So strange!

vemv15:01:12

Indeed! so, lein check, or lein test all pass?

seancorfield15:01:26

I'll be interested to hear the solution -- since this is one of those cases where "reloaded breaks" which I was discussing with some folks the other day (and why people like Eric Normand and Stu Halloway recommend against it).

vemv15:01:15

Likewise there's https://github.com/aredington/clojure-repl-sufficient-p pointing in the opposite direction. Personally I'd like things to be better implemented and documented, instead of discarding any approach altogether

vinai16:01:46

@U45T93RA6 lein check works fine. All tests pass, too. I can also uberjar the app.

vinai16:01:20

I must admit, with the benefit of hindsight I wouldn’t choose to build around component again. However, here we are 🙂 Someone have any idea what else I might try to figure out what is the cause?

vinai16:01:59

by the way, the reset was working fine in the project for the last 18 months at least.

vemv16:01:37

One thing you could try (and that I recommend in all cases) is moving the protocols out of src - constantly reloading them tends to be problematic. Protocols rarely change anyway e.g. you could have src + dev + protocols as your :source-paths. and "protocols" is excluded from the (set-refresh-dirs "src" "dev") call ...to be honest the namespace 'foo.video' not found error boggles me. The wording itself doesn't come from the Clojure compiler, I think? Maybe you have a more custom setup? (it's best to reproduce these in the terminal - not an IDE)

vemv16:01:14

> by the way, the reset was working fine in the project for the last 18 months at least. (needless to say, this can be bisected... could be some work though)

vinai17:01:54

Good idea, I’ll bisect. Thanks also for the suggestion to move the protocols out of src. If I run into an obstacle bisecting then I’ll try that. I think the project layout originally was based on some version of https://github.com/plexus/chestnut

vinai17:01:12

I don’t see anything out of the norm…

vemv17:01:50

Good luck! As a last idea you could run Eastwood. Specifically its :implicit-dependencies linter (https://github.com/jonase/eastwood/blob/9ab3e600906915194938f060eee544d56092ef0a/src/eastwood/lint.clj#L112 ) could be relevant

seancorfield17:01:49

Moving protocols out of your src tree makes for a very unusual project structure -- I wouldn't recommend that. Especially if it's just to try to get around a bug/quirk in the "reloaded" workflow (which I will again argue is unnecessary if you have a healthy REPL workflow!).

vemv17:01:25

Lein :source-paths can be extended arbitrarily. Sometimes I have dedicated migrations folder, or tasks folder, etc. There was a rationale for that and it works fine for us at work. Dedicated protocols folder works fine. It also can be made unnecessary if you ditch defrecord (alternatives: reify, metadata-based protocol extension). In general JVM classes are not particularly reload-friendly, so defrecord wil be always a bit inferior to reify. At work we chose metadata-based extension 1 year ago, has been a smooth trip.

seancorfield17:01:11

Yeah, I think the ability to extend protocols on a per-instance basis with metadata is a great enhancement!

vinai14:01:36

Follow-up to my trouble with component reset yesterday: found that a lein clean resolves the issue. Figured that out during git bisect. Resetting the bisect session, checking out the HEAD and doing lein clean and I can (reset) and (reset-all) as my heart desires. Now I only have to remember this solution until the next time I run into this issue 🙂 Thank you all very much for your help.

vemv14:01:04

Glad you got it! btw, whenever I uberjar, test etc I make sure to clean anyway. A Lein setup or bash aliases can help. Specifically this Lein bit kills some headaches: :target-path "target/%s/" - it increases per-env artifact isolation

vinai11:01:41

Thank you, that looks like a good idea. Added it to the project.clj

pinkfrog14:01:19

Is it a bad signal for code such as:

if (@some-atom) then (update some-atom) ?

pinkfrog14:01:33

I feel the if-stmt exposes a TOCTTOU issue.

Alex Miller (Clojure team)14:01:03

generally, that's a bad pattern

Alex Miller (Clojure team)14:01:23

put the check inside the update function

Alex Miller (Clojure team)14:01:06

there are cases where you can have other constraints that can make it ok

pinkfrog14:01:19

in that case, can I do (locking some-atom (expr)) ?

sgerguri14:01:13

You can do (swap! some-atom #(when % ...)).

otwieracz14:01:13

(swap! some-atom #(if (check %) (do-update) %)

otwieracz14:01:40

Swap fn returning nil won't cause atom value to be swapped to nil?

pinkfrog15:01:01

it will be updated to nil. We shall return the oldvalue.

sgerguri14:01:03

Well if it's nil it'll return nil, otherwise it will transform.

sgerguri14:01:13

The STM should take care of the ordering.

otwieracz14:01:35

Ah, but I assumed @i want's some other check there.

sgerguri15:01:34

The only other false-y value that I can see there would be false itself, and that's equivalent to nil in boolean contexts - though technically you're right, the when w_ould_ turn a false into a nil.

Alex Miller (Clojure team)14:01:58

Atoms do not use the STM

sgerguri15:01:29

But the update operations cannot interleave, can they?

sgerguri15:01:36

Otherwise, what would be the point.

sgerguri15:01:06

That is, if two concurrent threads are both updating an atom the updates will happen in some order, but won't interleave.

Alex Miller (Clojure team)15:01:34

Right (just saying that’s not the STM)

sgerguri15:01:56

Fair enough - though good to know I'm not off the mark. Thanks for the correction.

pinkfrog15:01:21

I feel if the checks are there, it is time for STM to kick in.

Alex Miller (Clojure team)15:01:53

STM is specifically the system that manages updates to refs

Alex Miller (Clojure team)15:01:06

atoms do not involve the STM

Alex Miller (Clojure team)15:01:34

there is no "transaction" of multiple operations with an atom - it's just a single operation on a single atom, performed atomically

pinkfrog15:01:40

I mean, given the TOCTOU case, the some-atoms should be redefined as some-ref.

didibus17:01:30

It kind of depends what you're trying to do

pinkfrog15:01:57

as a critical section is needed

tdantas15:01:22

I’m refactoring the code which is used by the web handler business rule says: “we do not accept any bet less than 15 min to the match start”

(defn- can-bet-match? [{:keys [starts-at]}]
  (time/before? (time/now) (time/minus (c/from-sql-time starts-at) (time/minutes 15))))

(defn bet
  [{:keys [id] :as participant}
   {:keys [match-id team-a-score team-b-score]}
   {:keys [starts-at] :as match}]

  (when-not (can-bet-match? match)
    (exceptions/precondition-failed! "bets gate closes 15 min before match start" {:match-id match-id
                                                                                   :starts-at starts-at}))

  (repository/execute-one! datasource
                           (insert-into-bet-query
                             id
                             match-id
                             team-a-score
                             team-b-score)))

tdantas15:01:00

I don’t like the exception to control the business rule, which other way you guys suggest ?,

lmergen15:01:21

there are numerous ways to handle this, all depending on personal taste

lmergen15:01:39

exceptions can be fine if it shouldn't happen in practice

lmergen15:01:07

in some other situations it would be more appropriate if it returned nil if nothing happened

lmergen15:01:22

and on the other side of the spectrum we have a monadic-like approach where you can wrap the thing in a maybe http://funcool.github.io/cats/latest/#maybe

lmergen15:01:47

you can also just do a Go/Erlang-like approach, where you return a pair [:ok val] or [:error "precondition failed: blah blah"] and you destructure that in the calling function

jumpnbrownweasel15:01:50

@intronic I don't understand why you say that STM is needed or better in some way. A conditional update works very well using swap! as @slawek098 posted above: (swap! some-atom #(if (check %) (do-update) %)

jumpnbrownweasel15:01:22

For example:

(swap! counter #(if (< 0 %) (dec %) %))

pinkfrog07:01:46

semantically, f under (swap .. f) should always successfully update a value.

pinkfrog07:01:08

adding the check inside f mixes the failed case with the successfully case.

pinkfrog07:01:20

one way is for the function f to throw exception to indicate check error. But I wonder wrapping try catch around swap! is not idiomatic.

seancorfield16:01:59

(swap! counter #(cond-> % (pos? %) (dec)))
I prefer cond-> for things that would otherwise be (if (pred) (f x) x) but I don't know how common that is.

👏 12
😮 4
4
seancorfield16:01:49

(and we have a variant at work called condp-> that threads through the predicate as well, making that #(condp-> % pos? dec))

sveri16:01:41

Hi. I just saw that partition-by returns ((x y)) if the predicate is true or false for all elements of the collection that it is applied to. Is there a way to always return two lists, even if the predicate returns true / false for all items like this:

(partition-by odd? [3 5 8]) => ((3 5 8)())
?

sveri16:01:47

I'd like to add that I am aware of filter and remove, but ideally would like to pass over the list only once.

hiredman16:01:01

You may be confused about what partition-by does. It creates a new partition everytime the result of the predicate changes

Alex Miller (Clojure team)16:01:17

maybe you want group-by?

hiredman16:01:12

So the number of partitions returned for a two value function is 0 to n

sveri16:01:53

@alexmiller Thanks, that would work, although I find it a bit unintuitive to have a map of false and true returned. But it makes sense, thank you 🙂

sveri16:01:07

@hiredman I read that, yes, but, my mind hoped for something else.

sveri16:01:50

The disadvantage of group-by with false and true is that I cannot destructure the resulting map.

hiredman16:01:38

Review the section on associative destructuring

sveri16:01:51

I just solved it by adapting my predicate to return :dead or :alive instead of false and true. @hiredman Thank you, that's even better.

Alex Miller (Clojure team)16:01:12

(let [{evens false, odds true} (group-by odd? (range 10))] (println odds evens))

vemv17:01:38

Out of curiosity, is there some sort of lib/framework for executing and plotting some work? For visualising scalability. e.g. I have a system with a thread pool, I want to find out the ideal pool size, so I plot pool size vs throughput. Or I have a queue, and want to determine its ideal size, plotting queue size vs latency.

hiredman17:01:24

A queue always adds latency, so the ideal queue size to reduce latency is 0

hiredman17:01:16

I usually plot stuff using gnuplot

hiredman17:01:21

I have sometimes used this to plot things right in the repl:

hiredman17:01:25

(defn plot [n]
  (when (> (count n) 1)
    (let [xmn (apply min (map :x n))
          xmx (apply max (map :x n))
          x-scale (/ (- xmx xmn) 20.0)
          ymn (apply min (map :y n))
          ymx (apply max (map :y n))
          y-scale (/ (- ymx ymn) 20.0)
          scaled-values (->> (sort-by :x n)
                             (map (fn [x]
                                    {:y (long (/ (- (:y  x) ymn) y-scale))
                                     :x (long (/ (- (:x x) xmn) x-scale))}))
                             (partition-by :x)
                             (map (fn [x] (apply max (map :y x)))))]
      (printf "%s\n" (java.util.Date.))
      (dotimes [_ 65] (print "-"))
      (printf "\n")
      (dotimes [r 21]
        (let [row (- 21 r)]
          (dotimes [col 21]
            (when (= col 0) (print "|"))
            (if (and (> (count scaled-values) col)
                     (= row (nth scaled-values col)))
              (print " * ")
              (print "   "))
            (when (= col 20) (print "|")))
          (printf "\n")))
      (dotimes [_ 65] (print "-"))
      (printf "\n")
      (flush))))

vemv17:01:01

> A queue always adds latency, so the ideal queue size to reduce latency is 0 Was a fictional example although by intuition, I'd say that the example can make sense (if you take the need for queues as a given) --- The parts are there (plotting libraries, Component for reifying a system that you can stop and make repeated experiments with, etc), I was mostly wondering if someone tied the parts generically

Eduardo Mata17:01:49

What is a good library to read csv files? Or should I implement it myself?

Alex Miller (Clojure team)18:01:53

I just created #clojure-survey as I'm starting to prep the State of Clojure 2020 survey. If you want to help me bike shed various answer sets, would be happy for feedback there

didibus20:01:26

I'm a bit lost with the hinting for primitives. It seems that calls to long and int slow down my code, by coercing back and forth between num and back

didibus20:01:42

But it also seems I can't type hint ^long a constant.

Alex Miller (Clojure team)20:01:55

you can, just not with that syntax

Alex Miller (Clojure team)20:01:14

var metadata are evaluated so that becomes the long function object

p-himik21:01:04

But the page at https://clojure.org/reference/java_interop#primitives mentions this code:

(defn foo ^long [^long n])
And I've seen it multiple times in the wild as well.

vemv21:01:20

Quite sure (defn foo ^long [^long n]) is correct. I have specifically analyzed that kind of code with https://github.com/clojure-goes-fast/clj-java-decompiler and the corresponding Java code effectively handles primitive longs.

Alex Miller (Clojure team)21:01:18

note that that hint is on the return type position of the args and ^long is ok there

andy.fingerhut21:01:44

Metadata and its effects are very position-in-the-code-dependent

Alex Miller (Clojure team)21:01:45

you can't however do (defn ^long foo 1) where you are type hinting a var

Alex Miller (Clojure team)21:01:07

metadata before the symbol in a var definition are evaluated (this is part of compilation), type hints before or in the arg signature are NOT evaluated and so things like ^long work (this is part of defn macros etc). the reasons for the difference are complicated and intentional but based in older times of decision.

p-himik21:01:13

Gotcha, thanks! In the box with case and its (:a :b)-like clauses it goes. :) So using (def {:tag 'long} foo 1) should work then.

bronsa20:01:59

it kinda depends what you're type hinting and in which position

bronsa21:01:09

can you share the code?

didibus21:01:17

I'm still lost. I have:

(let [m [10 20 30 40]]
 (loop [sum 0 i 0]
  (if (< i 4)
    (recur (+ sum (get m i)) (inc i))
    sum)))

didibus21:01:41

The + is reflective, and believes the types are: (argument types: java.lang.Number, java.lang.Object)

bronsa21:01:05

(long (get ..))

didibus21:01:55

Hum.. ok, that works, and so does ^long (get..

bronsa21:01:03

that's incorrect tho

didibus21:01:05

But why is sum a Number ?

bronsa21:01:08

you want the cast not the type hint

bronsa21:01:31

because (get m i) is an Object

bronsa21:01:41

so (+ long Object) => Object

bronsa21:01:51

and U Object long -> Number

bronsa21:01:36

so you want to unbox the Object of (get m i) to a long, you do that via (long ..)

bronsa21:01:55

a type hint is not supposed to be a cast/unbox, it sometimes works but it's not correct to do it that way

p-himik21:01:34

If the only difference between type hinting and casting is checkcast, why is it not correct to use type hinting when we know for sure that the hinted object is of the correct type?

bronsa21:01:09

it works, but it's unsafe(r)

p-himik21:01:52

Hmm. Does it mean that during runtime, if we end up with a type that's not the one that's used in the hint, we can get ourselves some UB?

p-himik21:01:35

Undefined Behavior. So far I've seen only ClastCastExceptions, so maybe there is checkcast but it's at some other place.

bronsa21:01:21

clojure does no check on the validity of type hints, it takes them at face value, so not in this particular case but in some cases you can get some weird errors by using type hints to do unsafe unboxing/castings

p-himik21:01:35

Is there any link you could suggest for more in-depth reading? I'm especially interested in the weird errors.

bronsa21:01:02

i'm not aware of anything written up

p-himik21:01:49

Ah, the Java code provided by didibus made me look at the difference between uncheckedLongCast and longCast. Now it's more or less clear, thanks!

didibus21:01:13

But I don't want to cast

bronsa21:01:21

you want to unbox to a long

didibus21:01:27

I want my vector to have primitive longs, and the get to return primitive long ?

bronsa21:01:36

you can't store primitive values in a vector

bronsa21:01:51

clojure collections only support Objects

didibus21:01:07

What about vetor-of ?

bfabry21:01:08

@didibus you can't store java primitives inside anything except primitive arrays. in java as well

bronsa21:01:16

you can use a vector-of, sure

andy.fingerhut21:01:26

Well, there is (vector-of :long 10 20 30) which store primitive longs, but when you get values out of them, I am pretty sure they are boxed.

didibus21:01:37

(let [m (vector-of :long 10 20 30 40)]
 (loop [sum 0 i 0]
  (if (< i 4)
    (recur (+ sum (get m i)) (inc i))
    sum)))

didibus21:01:42

This is still reflictive though

didibus21:01:47

So now do I cast or type hint ?

andy.fingerhut21:01:52

Because get returns Object

bronsa21:01:55

get always returns an Object

vemv21:01:56

Worth studying this section btw https://clojure.org/reference/java_interop#optimization As it is setting *unchecked-math* :warn-on-boxed somewhere. Either as a one-off experiment, or as part of the build (I run lein check with the flag set, merely for informative purposes)

noisesmith21:01:00

right - a primitive can't sit on the stack (as a function arg) or be attached to a field (eg. val in a map or normal vector)

didibus21:01:11

Well, what's the point of vector-of if when you get the elements back out they are boxed ?

bfabry21:01:16

The
resulting vector complies with the interface of vectors in general,
but stores the values unboxed internally.
the interface of vectors is put(i, Object) get(i) :: Object. vector-of is just a more efficient internal representation

andy.fingerhut21:01:19

I think it is fair to say that primitive values are fairly fragile things in Clojure source code, that require a bit of practice and semi-intimate knowledge to know when they become boxed?

8
andy.fingerhut21:01:53

vector-of vectors of longs require about 1/3 the memory of a general vector that contains Long objects.

bronsa21:01:53

if you reduce over a vector-of the values won't be boxed actually they will cause the impl doesn't take priminvoke into account

bronsa21:01:04

but there's not exposed primitive get api

didibus21:01:50

The type hint version is way faster then the cast to long

hiredman21:01:52

I have a gist for this

dpsutton21:01:22

i would spend good money on a book of your gists annotated with your thoughts 🙂

andy.fingerhut21:01:22

Ahhh! The goggles, they do nothing! (Seriously, though, thanks for that link)

bfabry21:01:07

if the loop is really this hot I think areduce would be your winner 🙈

bronsa21:01:58

they compile to the same bytecode

bronsa21:01:07

the cast version only has an extra checkcast

bronsa21:01:35

which in hotloops will not matter

bronsa21:01:09

but really you don't want to use a vector for this if you don't want the boxing/unboxing

didibus21:01:43

Well, array would require me to know the size in advance

didibus21:01:57

Here I use the vector to memoize prior results

didibus21:01:25

I'm not sure how much space it'll take before finding the result

didibus21:01:51

So for my original question. The compiler does treat sum as a primitive long, but because the get returns Object, it was casting the primitive long sum back into a Number ?Is that it?

bronsa21:01:44

well, not exactly

didibus21:01:54

The funny thing is, that cast to num made it slower then if it was a Long

bronsa21:01:21

it initially treats sum as a primitive long, but when doing analysis on the (+ sum (get m i)) expression it infers that to be a Number

bronsa21:01:56

and so it makes sum a Number because it can't unify long and Number

didibus21:01:18

Hum ok I see.

didibus21:01:02

I guess I was trying to figure out why there was a lot of calls to num

didibus21:01:40

10% of my profiled samples went to it. But maybe that was from Object to num from the return of get

bronsa21:01:25

oh that call to num must be for i in (get m i) actually

bronsa21:01:39

try using nth instead of get

bronsa21:01:41

and i (int 0) instead of i 0

bronsa21:01:17

get is much more polymorphic than nth since it works for any associative, so it boxes the key to an Object

bronsa21:01:29

nth knows that the key is an int so you can avoid the boxing there

bronsa21:01:47

(but again, if you're this worried about avoiding boxing I wouldn't use a clojure collection in the first place)

8
didibus21:01:05

Cool, I'll try that. I'm mostly just learning here more than anything.

didibus21:01:16

This is with a type hint of long on the get:

public static Object invokeStatic() {
        final Object m = const__4;
        long sum = 0L;
        long add;
        for (long i = 0L; (i, 4L); i = PrimitiveMath.inc(i), sum = add) {
            add = PrimitiveMath.add(sum, RT.uncheckedLongCast(RT.get(m, Numbers.num(i))));
        }
        return Numbers.num(sum);
    }
And with a cast instead:
public static Object invokeStatic() {
        final Object m = const__4;
        long sum = 0L;
        long add;
        for (long i = 0L; (i, 4L); i = PrimitiveMath.inc(i), sum = add) {
            add = PrimitiveMath.add(sum, RT.longCast(RT.get(m, Numbers.num(i))));
        }
        return Numbers.num(sum);
    }
Only difference is uncheckedLongCast instead of longCast

didibus21:01:58

And as you said, I can see the call to num for i

seancorfield21:01:16

What are you using to get Java source code back from compiled Clojure @didibus ?

didibus21:01:25

com.clojure-goes-fast/clj-java-decompiler {:mvn/version "0.2.1"}

👀 8
didibus21:01:12

You can call it from the repl, super easy using decompile macro, and it prints the body decompiled to out

didibus21:01:44

Also worth checking out all the other libs from clojure-goes-fast, like the profiler and memory meter

seancorfield21:01:59

I should add that to my dot-clojure setup! Nice!

bfabry21:01:16

aye that's cool

didibus21:01:14

Switch to nth shaved off that extra 10% btw, nice!

seancorfield21:01:56

Ah, I already have the memory meter in my dot-clojure setup... will look at the profiler too...

didibus22:01:17

So it seems unchecked-math true does not make casts into unchecked casts. So you need to explicitly say (unchecked-long ...)

didibus22:01:35

And for some reason, type hinting to ^long sometimes work as well to insert an uncheckedLongCast, but not always

noisesmith22:01:56

I thought unchecked math just turned on warnings for checked math? I thought wrong, thanks for the info

andy.fingerhut22:01:35

Clojure 1.10.1
user=> (+ Long/MAX_VALUE 1)
Execution error (ArithmeticException) at user/eval1 (REPL:1).
integer overflow
user=> (set! *unchecked-math* true)
true
user=> (+ Long/MAX_VALUE 1)
-9223372036854775808

4
andy.fingerhut22:01:19

And I was about to add that example to http://ClojureDocs.org, but exactly that example is already there: https://clojuredocs.org/clojure.core/*unchecked-math*

ghadi22:01:41

*unchecked-math* will affect interop calls, like if you're calling nth which expands out to (clojure.lang.RT/nth coll index) <-- the cast of index from long -> int will be unchecked

didibus22:01:58

Oh I see, but the result of nth which is then boxed, and casted with long is unafected?

ghadi22:01:25

the result of nth is always Object

didibus22:01:38

I mean: (long (nth coll index))

bronsa22:01:44

you want to call unchecked-long instead of long

didibus22:01:59

index will be casted to uncheckdIntCast, but long will compile to longCast

didibus22:01:23

Right, I just wanted to confirm that casts are not affected by unchecked-math true. I feel like.. they should?

bronsa22:01:04

for some reason int takes *unchecked-math* into account while long doesn't

bronsa22:01:05

¯\(ツ)

didibus22:01:16

I would expect that all fns that have an equivalent unchecked- variant would use the unchecked variant when that flag is true

didibus22:01:31

Hum... bug report time I guess

seancorfield22:01:43

@didibus Which JVM are you using with the decompiler?

seancorfield22:01:41

I just tried it on Adopt OpenJDK 11 and got an error that looks like a JVM version compatibility issue...

seancorfield22:01:55

Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:581).
sun.misc.URLClassPath

didibus22:01:04

Hum, right now I happen to have Zulu 8.40.0.25-CA-linux64, but I used to use it as well with the OpenJDK 8 build of OpenSuse

seancorfield22:01:18

'k... will try again with JDK 8.

didibus22:01:45

Hum, I think in JDK 11 all the sun. packages have become hidden modules

andy.fingerhut22:01:29

The sun has set within the Oracle empire ...

littleli22:01:11

I remember Brian Goetz mentioned somewhere they are about to remove them entirely. Especially unsafe package

jumpnbrownweasel23:01:59

Oracle decided not to remove Unsafe due to its widespread use for off-heap memory allocation. However, there is a new public API for memory allocation in JDK 14, so perhaps then they'll remove Unsafe.

didibus22:01:03

Seems its fixed in latest snapshot

4
didibus23:01:59

Hum, and can a let/loop local be of type int ?

didibus23:01:08

Or its always long and double ?

didibus23:01:25

Hum, answering my own question, it seems [i (int 100] does create an int. So I'm guessing I have something else inside that changes it back to long

bfabry23:01:58

it can be of type, but you can't have a literal int https://clojure.org/reference/java_interop#primitives

didibus23:01:13

Man, its really hard to figure out how to get the primitives you want :p

andy.fingerhut23:01:26

long and double are supported within loop, but I thought that it might not be possible to get int primitive for a loop 'variable'

andy.fingerhut23:01:09

You can of course create an int from any number using (int ...) , but that involves at least one extra method call

andy.fingerhut23:01:30

well, at least one more JVM byte code operation... maybe not always a method call

andy.fingerhut23:01:57

If you really, really care about pedal to the metal performance on the JVM with primitives, you should at least consider whether you should be writing Java code.

didibus23:01:12

Haha, ya definitly in this case, if I wasn't just experimenting, I would probably move to Java. But then again, even Java gets annoying, trying to use an unsigned int :S

didibus23:01:13

So only array allows a get/put that is not boxed ?

didibus23:01:28

that uncheckedLongCast is taking 18% of the time 😛

bfabry23:01:02

that's also the case for java isn't it?

hiredman23:01:03

for generic collections, yes

didibus23:01:29

Hum, is it. I didn't know that

bfabry23:01:03

my point is if you're ever at the point where you're at that tweaky level of performance you should be using primitive arrays. arrays are stored contiguously in memory and so things like cache lines are going to dominate things like stack pushes