This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2022-07-05
Channels
- # announcements (1)
- # asami (21)
- # aws (19)
- # babashka (37)
- # beginners (38)
- # clj-kondo (7)
- # clj-otel (8)
- # clojure (29)
- # clojure-europe (54)
- # clojure-nl (3)
- # clojure-spec (2)
- # clojure-uk (2)
- # clojurescript (15)
- # conjure (1)
- # data-science (1)
- # datomic (21)
- # emacs (6)
- # events (3)
- # figwheel-main (1)
- # gratitude (13)
- # holy-lambda (11)
- # joyride (6)
- # klipse (3)
- # malli (14)
- # missionary (26)
- # nbb (31)
- # omni-trace (2)
- # pathom (3)
- # reagent (1)
- # reitit (1)
- # releases (1)
- # shadow-cljs (24)
- # sql (27)
- # tools-deps (4)
- # vim (21)
Is there any construct like this in clojure where all sub expression would be run parallel and results will be returned after the longest running expression returns?
(run-parallelly
(slurp "")
(Thread/sleep 2000)
(println "All run parallel"))
=> ("<!doctype....", nil, nil)
There are actually built in things for this, maybe pcalls? I forget the name, but it does something slightly more complicated similar to pmap, which I have a low opinion of and never use
It doesn't launch parallel tasks until you consume the lazy seq it produces, but also produces a little more than what you ask for in parallel, and also produces results in order
Basically bakes in a bunch of decisions that don't often match what I want from parallel execution
I think the claypool library tries to provide more flexible versions of pmap and pcalls, but I've never used those either
https://github.com/clojure/clojure/blob/clojure-1.10.1/src/clj/clojure/core.clj#L7037 yeah pcalls
The docstring for pvalues would make you believe it is run-parallelly, but it is implemented using pmap, the parallel execution is entwined with the lazy realization of the seq, which complicated things
I tried writing a macro to run all expressions in a body in parallel. This seems to work:
(defmacro run-parallel [& body]
(list 'doall
(list 'map 'deref
(into []
(for [x body] (list 'future x))))))
(macroexpand-1 '(run-parallel (+ 1 2) (Thread/sleep 1000))) => (doall (map deref [(future (+ 1 2)) (future (Thread/sleep 1000))]))
(run-parallel (+ 1 2)
(Thread/sleep 1000)) => (3 nil)
I can't seem to write this macro with backquote though. Everything I try fails. Any help to write with backquote?(defmacro run-parallel [& body]
`(doall
(map deref
~(into []
(for [x body]
`(future ~x))))))
although sticking with vectors:
(defmacro run-parallel [& body]
`(mapv deref
~(mapv (partial list `future)
body)))
may be preferablehttps://clojure.org/guides/comparators#_clojures_default_comparator strings are sorted in lexicographic order by default
thanks for response @U04V4KLKC, Is there any way to handle it? except changing to Numbers ?
becasue my column may also have string , so I have converted all the numbers into strings and testing
sort
accepts optional comparator. you can configure it to do anything you want including comparison of numbers with strings
(compare "0" 1)
will throw by default because doesn’t know how to compare string and number. you can create a function that will return 0, 1 or -1 in that case and use it as a comparator
@U01J3DB39R6 if your data is always a number, but sometimes it as string, you can create a function that first "left-pad" the shorter string with zeros, and then calls the comparison. This will ensure that even string's lexicographical comparison will work for your number comparison
I am trying to get my head around the idea that Clojure code is executed at compile time: I have just switched to using Omniconf (after having just hardcoded strings for config during local prototyping) - where previously compilation was successful due to all the config values being available (at this point I was even unaware that code is executed at compile time), now they are loaded at runtime with Omiconf I am getting a raft of null pointer exceptions and similar at compile time. What is the recommended method to mitigate this? Should I somehow defer execution of code that relies on runtime config?
in short, yes
compiling will load every namespace being compiled, so top-level def's are usually problematic in this regard (I treat them all with suspicion) - consider pushing more into defn's, invoked via starts, or using delay
Thanks 🙂 Presumably then top-level defs should be used sparingly, for things with no/few side effects/dependencies? Would it be sufficient to move the defs to an ‘init’ (start?) defn and call that at runtime (via -main)?
Ah OK, yeah it did seem like cheating
What is meant by ‘starts’?
sorry, meant a starting function, whatever your entry point is
Ah I see, thanks 🙂
which presumably will not be called during compilation (just loading) but will be called when you run your app
this may require you to pass more state through things rather than using global defs. that is a good direction to go.
Great, yeah I think that was starting to dawn on me. Here I guess really the top level defs are just a lazy way to share a config dependency across defns
So yeah I think I will pull them into a defn and call that in the entry point and pass them into the defns that need it
you'll thank me a year from now :)
😄 I certainly also thank you today
I was definitely over-thinking it, the concept of code executing at compile time blew my tiny mind and threw me off common sense