This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-01-02
Channels
- # announcements (1)
- # aws (5)
- # babashka (13)
- # beginners (202)
- # bristol-clojurians (3)
- # cider (16)
- # clojure (283)
- # clojure-dev (8)
- # clojure-finland (30)
- # clojure-italy (4)
- # clojure-nl (6)
- # clojure-spec (17)
- # clojure-survey (161)
- # clojure-sweden (7)
- # clojure-uk (62)
- # clojurescript (4)
- # core-async (31)
- # cursive (3)
- # datomic (7)
- # defnpodcast (1)
- # fulcro (8)
- # jobs (2)
- # lumo (2)
- # malli (2)
- # off-topic (24)
- # other-languages (1)
- # overtone (1)
- # re-frame (6)
- # remote-jobs (3)
- # shadow-cljs (6)
- # spacemacs (17)
- # tools-deps (20)
What is the current situation with core.async
?
I have the impression that this is not the preferred library by the community. Everyone seems to be recommending manilofd
or promesa
. Why?
I haven't seen any of that. core.async is pretty nice and gets stuff right that manifold misses. My major complaint is patches languishing in jira
I recommend manifold because of the monitoring; core.async provides little control over the threadpools behind it, no monitoring, and it’s generally discouraged to use it for blocking operations. manifold allows you to control all the thread pools, control which call is executed on which pool, and generally feels more “real world compatible”. if these things do not interest you that much, chances are core.async is fine for your use case.
@hiredman
metosin uses both promesa
and manifold
. You can see it at https://github.com/metosin/talvi.
http://juxt.pro, grammarly, or http://deps.co also use manifold
I personally like and prefer core.async
, but I was curious why so few people recommend it, and recommend everything else
I think for the core.async threadpool we just measure latency (by periodically throwing a task on it and timing it to completion)
Yeah, ok, there is a github repo with a list of libraries and a wishlist of what they want to be in the github repo
Anyway, core.async is at least as nice as anything else. I haven't seen any evidence of a strong swell of opinion for anything. I will say aleph (which manifold is often used in conjunction with) is older so it is kind of built in to older stable stuff
Promesa appears to mostly be of interest to the cljs crowd, which is likely because of js promises
With js's generators you can write something like core.async pretty easily https://gist.github.com/hiredman/2271c48e1f036253ce37913abd3a680a
@hiredman
I'm absolutely not claiming that manifold
or promesa
is better than core.async
I'm even glad you're saying the opposite. Like I said, I prefer core.async
to everything else.
on CI when AOT-ing a lib with lein uberjar I sometimes get an error:
Execution error (IllegalArgumentException) at sci.impl.vars/fn$G (vars.cljc:121).
No implementation of method: :unbind of protocol: #'sci.impl.vars/IVar found for class: sci.impl.vars.SciVar
This usually resolves itself when I re-trigger the build, but I'd like to fix it. The relevant file is this:
https://github.com/borkdude/sci/blob/master/src/sci/impl/vars.cljcThis is the failing step: https://circleci.com/gh/borkdude/sci/2800
Maybe the protocol is redefined by the previous uberjar or class files (cached under target?). Leiningen includes target in the classpath i believe
@U0FT7SRLP It runs: lein with-profiles +clojure-1.10.1 do clean, uberjar
https://github.com/borkdude/sci/blob/master/script/compile
@U04V15CAJ yeah clean
should fix that indeed
It could also happen if the protocol ns is loaded twice e.g. via reload, but i didn't see that in the code
We are using [com.stuartsierra/component "0.4.0"]
. When running (reset)
or (reset-all)
in the REPL an error is thrown:
#error{:cause "namespace 'foo.video' not found", :message "Syntax error compiling at (foo/video/test_provider.clj:1:1).",
The foo.video
ns is only
(ns foo.video)
(defprotocol VideoProvider
(embed-html [this video]
"Return the HTML for embedding the given video on a page.
Throws an ExceptionInfo if something goes wrong."))
(defmulti new-video-provider :provider)
The foo.video.test-provider
ns is
(ns foo.video.test-provider
(:require [foo.video :as video]))
(defrecord TestVideoProvider [stub-html]
video/VideoProvider
(embed-html [this video] stub-html))
(defmethod video/new-video-provider ::provider
[properties] (map->TestVideoProvider (dissoc properties :provider)))
The project compiles fine, it is only the reset that is failing. I can’t figure out why. Any idea what is wrong?Thanks for the reply - sorry for getting back late, got pulled into meeting. Nothing is being set besides the component system map. However, this setup used to work fine. I’m not sure when it stopped working. I’ll try setting the dirs
The foo.video.test-provider
is part of the normal system, not the tests despite it’s name.
Seems pretty weird. Maybe a different error happens before you get namespace 'foo.video' not found
?
Or maybe the error contains useful info within its stacktrace?
More strangeness! I’ve removed the namespace foo.video.test-provider
and all references to it and the error just pops around to another one and so on. It doesn’t seem to be tied to any specific ns. Just happens to be that one. So strange!
I'll be interested to hear the solution -- since this is one of those cases where "reloaded breaks" which I was discussing with some folks the other day (and why people like Eric Normand and Stu Halloway recommend against it).
Likewise there's https://github.com/aredington/clojure-repl-sufficient-p pointing in the opposite direction. Personally I'd like things to be better implemented and documented, instead of discarding any approach altogether
@U45T93RA6 lein check
works fine. All tests pass, too. I can also uberjar the app.
I must admit, with the benefit of hindsight I wouldn’t choose to build around component again. However, here we are 🙂 Someone have any idea what else I might try to figure out what is the cause?
One thing you could try (and that I recommend in all cases) is moving the protocols out of src
- constantly reloading them tends to be problematic. Protocols rarely change anyway
e.g. you could have src
+ dev
+ protocols
as your :source-paths
. and "protocols"
is excluded from the (set-refresh-dirs "src" "dev")
call
...to be honest the namespace 'foo.video' not found
error boggles me. The wording itself doesn't come from the Clojure compiler, I think? Maybe you have a more custom setup? (it's best to reproduce these in the terminal - not an IDE)
> by the way, the reset was working fine in the project for the last 18 months at least. (needless to say, this can be bisected... could be some work though)
Good idea, I’ll bisect.
Thanks also for the suggestion to move the protocols out of src
. If I run into an obstacle bisecting then I’ll try that.
I think the project layout originally was based on some version of https://github.com/plexus/chestnut
Good luck!
As a last idea you could run Eastwood. Specifically its :implicit-dependencies
linter (https://github.com/jonase/eastwood/blob/9ab3e600906915194938f060eee544d56092ef0a/src/eastwood/lint.clj#L112 ) could be relevant
Moving protocols out of your src
tree makes for a very unusual project structure -- I wouldn't recommend that. Especially if it's just to try to get around a bug/quirk in the "reloaded" workflow (which I will again argue is unnecessary if you have a healthy REPL workflow!).
Lein :source-paths
can be extended arbitrarily. Sometimes I have dedicated migrations
folder, or tasks
folder, etc. There was a rationale for that and it works fine for us at work.
Dedicated protocols
folder works fine. It also can be made unnecessary if you ditch defrecord
(alternatives: reify
, metadata-based protocol extension). In general JVM classes are not particularly reload-friendly, so defrecord wil be always a bit inferior to reify
.
At work we chose metadata-based extension 1 year ago, has been a smooth trip.
Yeah, I think the ability to extend protocols on a per-instance basis with metadata is a great enhancement!
Follow-up to my trouble with component reset yesterday: found that a lein clean
resolves the issue.
Figured that out during git bisect. Resetting the bisect session, checking out the HEAD and doing lein clean
and I can (reset)
and (reset-all)
as my heart desires.
Now I only have to remember this solution until the next time I run into this issue 🙂
Thank you all very much for your help.
Glad you got it!
btw, whenever I uberjar
, test
etc I make sure to clean anyway. A Lein setup or bash aliases can help.
Specifically this Lein bit kills some headaches: :target-path "target/%s/"
- it increases per-env artifact isolation
generally, that's a bad pattern
put the check inside the update function
there are cases where you can have other constraints that can make it ok
but use caution
There's https://clojuredocs.org/clojure.core/compare-and-set! as well
The only other false-y value that I can see there would be false
itself, and that's equivalent to nil
in boolean contexts - though technically you're right, the when
w_ould_ turn a false into a nil.
Atoms do not use the STM
That’s for refs
That is, if two concurrent threads are both updating an atom the updates will happen in some order, but won't interleave.
Right (just saying that’s not the STM)
STM is specifically the system that manages updates to refs
atoms do not involve the STM
there is no "transaction" of multiple operations with an atom - it's just a single operation on a single atom, performed atomically
I’m refactoring the code which is used by the web handler business rule says: “we do not accept any bet less than 15 min to the match start”
(defn- can-bet-match? [{:keys [starts-at]}]
(time/before? (time/now) (time/minus (c/from-sql-time starts-at) (time/minutes 15))))
(defn bet
[{:keys [id] :as participant}
{:keys [match-id team-a-score team-b-score]}
{:keys [starts-at] :as match}]
(when-not (can-bet-match? match)
(exceptions/precondition-failed! "bets gate closes 15 min before match start" {:match-id match-id
:starts-at starts-at}))
(repository/execute-one! datasource
(insert-into-bet-query
id
match-id
team-a-score
team-b-score)))
I don’t like the exception to control the business rule, which other way you guys suggest ?,
in some other situations it would be more appropriate if it returned nil
if nothing happened
We talked a bit about that in https://clojurians.slack.com/archives/C03RZGPG3/p1577579852221100
and on the other side of the spectrum we have a monadic-like approach where you can wrap the thing in a maybe
http://funcool.github.io/cats/latest/#maybe
you can also just do a Go/Erlang-like approach, where you return a pair [:ok val]
or [:error "precondition failed: blah blah"]
and you destructure that in the calling function
@intronic I don't understand why you say that STM is needed or better in some way. A conditional update works very well using swap!
as @slawek098 posted above:
(swap! some-atom #(if (check %) (do-update) %)
For example:
(swap! counter #(if (< 0 %) (dec %) %))
one way is for the function f to throw exception to indicate check error. But I wonder wrapping try catch around swap! is not idiomatic.
(swap! counter #(cond-> % (pos? %) (dec)))
I prefer cond->
for things that would otherwise be (if (pred) (f x) x)
but I don't know how common that is.(and we have a variant at work called condp->
that threads through the predicate as well, making that #(condp-> % pos? dec)
)
Hi. I just saw that partition-by
returns ((x y))
if the predicate is true or false for all elements of the collection that it is applied to.
Is there a way to always return two lists, even if the predicate returns true / false for all items like this:
(partition-by odd? [3 5 8]) => ((3 5 8)())
?I'd like to add that I am aware of filter and remove, but ideally would like to pass over the list only once.
You may be confused about what partition-by does. It creates a new partition everytime the result of the predicate changes
maybe you want group-by
?
@alexmiller Thanks, that would work, although I find it a bit unintuitive to have a map of false and true returned. But it makes sense, thank you 🙂
The disadvantage of group-by with false and true is that I cannot destructure the resulting map.
I just solved it by adapting my predicate to return :dead
or :alive
instead of false and true.
@hiredman Thank you, that's even better.
(let [{evens false, odds true} (group-by odd? (range 10))] (println odds evens))
@alexmiller Thank you 🙂
Out of curiosity, is there some sort of lib/framework for executing and plotting some work? For visualising scalability. e.g. I have a system with a thread pool, I want to find out the ideal pool size, so I plot pool size vs throughput. Or I have a queue, and want to determine its ideal size, plotting queue size vs latency.
(defn plot [n]
(when (> (count n) 1)
(let [xmn (apply min (map :x n))
xmx (apply max (map :x n))
x-scale (/ (- xmx xmn) 20.0)
ymn (apply min (map :y n))
ymx (apply max (map :y n))
y-scale (/ (- ymx ymn) 20.0)
scaled-values (->> (sort-by :x n)
(map (fn [x]
{:y (long (/ (- (:y x) ymn) y-scale))
:x (long (/ (- (:x x) xmn) x-scale))}))
(partition-by :x)
(map (fn [x] (apply max (map :y x)))))]
(printf "%s\n" (java.util.Date.))
(dotimes [_ 65] (print "-"))
(printf "\n")
(dotimes [r 21]
(let [row (- 21 r)]
(dotimes [col 21]
(when (= col 0) (print "|"))
(if (and (> (count scaled-values) col)
(= row (nth scaled-values col)))
(print " * ")
(print " "))
(when (= col 20) (print "|")))
(printf "\n")))
(dotimes [_ 65] (print "-"))
(printf "\n")
(flush))))
> A queue always adds latency, so the ideal queue size to reduce latency is 0
Was a fictional example although by intuition, I'd say that the example can make sense (if you take the need for queues as a given)
---
The parts are there (plotting libraries, Component
for reifying a system that you can stop and make repeated experiments with, etc), I was mostly wondering if someone tied the parts generically
What is a good library to read csv files? Or should I implement it myself?
I just created #clojure-survey as I'm starting to prep the State of Clojure 2020 survey. If you want to help me bike shed various answer sets, would be happy for feedback there
I'm a bit lost with the hinting for primitives. It seems that calls to long and int slow down my code, by coercing back and forth between num and back
you can, just not with that syntax
var metadata are evaluated so that becomes the long
function object
But the page at https://clojure.org/reference/java_interop#primitives mentions this code:
(defn foo ^long [^long n])
And I've seen it multiple times in the wild as well.Quite sure (defn foo ^long [^long n])
is correct. I have specifically analyzed that kind of code with https://github.com/clojure-goes-fast/clj-java-decompiler and the corresponding Java code effectively handles primitive longs.
note that that hint is on the return type position of the args and ^long is ok there
Metadata and its effects are very position-in-the-code-dependent
you can't however do (defn ^long foo 1) where you are type hinting a var
metadata before the symbol in a var definition are evaluated (this is part of compilation), type hints before or in the arg signature are NOT evaluated and so things like ^long work (this is part of defn macros etc). the reasons for the difference are complicated and intentional but based in older times of decision.
Gotcha, thanks! In the box with case
and its (:a :b)
-like clauses it goes. :)
So using (def {:tag 'long} foo 1)
should work then.
yes, that's one option
I'm still lost. I have:
(let [m [10 20 30 40]]
(loop [sum 0 i 0]
(if (< i 4)
(recur (+ sum (get m i)) (inc i))
sum)))
The + is reflective, and believes the types are: (argument types: java.lang.Number, java.lang.Object)
a type hint is not supposed to be a cast/unbox, it sometimes works but it's not correct to do it that way
If the only difference between type hinting and casting is checkcast
, why is it not correct to use type hinting when we know for sure that the hinted object is of the correct type?
Hmm. Does it mean that during runtime, if we end up with a type that's not the one that's used in the hint, we can get ourselves some UB?
Undefined Behavior.
So far I've seen only ClastCastExceptions, so maybe there is checkcast
but it's at some other place.
clojure does no check on the validity of type hints, it takes them at face value, so not in this particular case but in some cases you can get some weird errors by using type hints to do unsafe unboxing/castings
Is there any link you could suggest for more in-depth reading? I'm especially interested in the weird errors.
Ah, the Java code provided by didibus made me look at the difference between uncheckedLongCast
and longCast
. Now it's more or less clear, thanks!
@didibus you can't store java primitives inside anything except primitive arrays. in java as well
Well, there is (vector-of :long 10 20 30)
which store primitive longs, but when you get values out of them, I am pretty sure they are boxed.
(let [m (vector-of :long 10 20 30 40)]
(loop [sum 0 i 0]
(if (< i 4)
(recur (+ sum (get m i)) (inc i))
sum)))
Because get
returns Object
Worth studying this section btw https://clojure.org/reference/java_interop#optimization
As it is setting *unchecked-math* :warn-on-boxed
somewhere. Either as a one-off experiment, or as part of the build (I run lein check
with the flag set, merely for informative purposes)
right - a primitive can't sit on the stack (as a function arg) or be attached to a field (eg. val in a map or normal vector)
Well, what's the point of vector-of if when you get the elements back out they are boxed ?
The
resulting vector complies with the interface of vectors in general,
but stores the values unboxed internally.
the interface of vectors is put(i, Object) get(i) :: Object. vector-of is just a more efficient internal representationI think it is fair to say that primitive values are fairly fragile things in Clojure source code, that require a bit of practice and semi-intimate knowledge to know when they become boxed?
vector-of
vectors of longs require about 1/3 the memory of a general vector
that contains Long
objects.
if you reduce over a vector-of the values won't be boxed actually they will cause the impl doesn't take priminvoke into account
Ahhh! The goggles, they do nothing! (Seriously, though, thanks for that link)
but really you don't want to use a vector for this if you don't want the boxing/unboxing
So for my original question. The compiler does treat sum as a primitive long, but because the get returns Object, it was casting the primitive long sum back into a Number ?Is that it?
it initially treats sum as a primitive long, but when doing analysis on the (+ sum (get m i))
expression it infers that to be a Number
10% of my profiled samples went to it. But maybe that was from Object to num from the return of get
get
is much more polymorphic than nth
since it works for any associative, so it boxes the key to an Object
(but again, if you're this worried about avoiding boxing I wouldn't use a clojure collection in the first place)
This is with a type hint of long on the get:
public static Object invokeStatic() {
final Object m = const__4;
long sum = 0L;
long add;
for (long i = 0L; (i, 4L); i = PrimitiveMath.inc(i), sum = add) {
add = PrimitiveMath.add(sum, RT.uncheckedLongCast(RT.get(m, Numbers.num(i))));
}
return Numbers.num(sum);
}
And with a cast instead:
public static Object invokeStatic() {
final Object m = const__4;
long sum = 0L;
long add;
for (long i = 0L; (i, 4L); i = PrimitiveMath.inc(i), sum = add) {
add = PrimitiveMath.add(sum, RT.longCast(RT.get(m, Numbers.num(i))));
}
return Numbers.num(sum);
}
Only difference is uncheckedLongCast instead of longCastWhat are you using to get Java source code back from compiled Clojure @didibus ?
You can call it from the repl, super easy using decompile macro, and it prints the body decompiled to out
Also worth checking out all the other libs from clojure-goes-fast, like the profiler and memory meter
I should add that to my dot-clojure setup! Nice!
https://www.youtube.com/watch?v=iQwQXVM6oiY shares a few tips in that direction
Ah, I already have the memory meter in my dot-clojure setup... will look at the profiler too...
So it seems unchecked-math true does not make casts into unchecked casts. So you need to explicitly say (unchecked-long ...)
And for some reason, type hinting to ^long sometimes work as well to insert an uncheckedLongCast, but not always
I thought unchecked math just turned on warnings for checked math? I thought wrong, thanks for the info
Clojure 1.10.1
user=> (+ Long/MAX_VALUE 1)
Execution error (ArithmeticException) at user/eval1 (REPL:1).
integer overflow
user=> (set! *unchecked-math* true)
true
user=> (+ Long/MAX_VALUE 1)
-9223372036854775808
And I was about to add that example to http://ClojureDocs.org, but exactly that example is already there: https://clojuredocs.org/clojure.core/*unchecked-math*
*unchecked-math*
will affect interop calls, like if you're calling nth
which expands out to (clojure.lang.RT/nth coll index)
<-- the cast of index from long -> int will be unchecked
Oh I see, but the result of nth which is then boxed, and casted with long is unafected?
Right, I just wanted to confirm that casts are not affected by unchecked-math true. I feel like.. they should?
I would expect that all fns that have an equivalent unchecked- variant would use the unchecked variant when that flag is true
@didibus Which JVM are you using with the decompiler?
I just tried it on Adopt OpenJDK 11 and got an error that looks like a JVM version compatibility issue...
Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:581).
sun.misc.URLClassPath
Hum, right now I happen to have Zulu 8.40.0.25-CA-linux64, but I used to use it as well with the OpenJDK 8 build of OpenSuse
'k... will try again with JDK 8.
The sun has set within the Oracle empire ...
I remember Brian Goetz mentioned somewhere they are about to remove them entirely. Especially unsafe package
Oracle decided not to remove Unsafe due to its widespread use for off-heap memory allocation. However, there is a new public API for memory allocation in JDK 14, so perhaps then they'll remove Unsafe.
Thanks.
Hum, answering my own question, it seems [i (int 100] does create an int. So I'm guessing I have something else inside that changes it back to long
it can be of type, but you can't have a literal int https://clojure.org/reference/java_interop#primitives
long
and double
are supported within loop, but I thought that it might not be possible to get int
primitive for a loop
'variable'
You can of course create an int
from any number using (int ...)
, but that involves at least one extra method call
well, at least one more JVM byte code operation... maybe not always a method call
If you really, really care about pedal to the metal performance on the JVM with primitives, you should at least consider whether you should be writing Java code.