This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2020-02-27
Channels
- # announcements (8)
- # architecture (3)
- # aws (18)
- # beginners (96)
- # bristol-clojurians (3)
- # calva (15)
- # cider (7)
- # clj-kondo (8)
- # clojure (135)
- # clojure-denmark (1)
- # clojure-dev (14)
- # clojure-europe (37)
- # clojure-italy (9)
- # clojure-nl (14)
- # clojure-sanfrancisco (1)
- # clojure-spec (1)
- # clojure-uk (54)
- # clojurescript (27)
- # core-async (243)
- # cursive (28)
- # data-science (6)
- # datomic (33)
- # fulcro (25)
- # graalvm (24)
- # hoplon (2)
- # instaparse (12)
- # jackdaw (1)
- # java (21)
- # juxt (12)
- # meander (10)
- # nyc (4)
- # off-topic (6)
- # om (3)
- # pathom (17)
- # perun (1)
- # re-frame (29)
- # reitit (4)
- # rum (3)
- # shadow-cljs (119)
- # spacemacs (31)
- # xtdb (14)
I have the following deps.edn file in .clojure:
{:aliases
{:meander {:deps {meander/epsilon {:mvn/version "0.0.389"}}}}}
But when I run clj -A:meander
, I get:
WARNING: Specified aliases are undeclared: [:meander]
What am I doing wrong?Is that deps.edn file in a directory named .clojure
in your home directory? What OS are you using?
Ubuntu (on WSL)
(base) d4hines@ghel:/mnt/c/Users/d4hin$ ls .clojure/
deps.edn
(base) d4hines@ghel:/mnt/c/Users/d4hin$
Looks right to meWhat is the output of the command clojure -Sdescribe
? You need not share it here if you prefer not to, but the output should include the full path names to files that it reads.
Ahhh I see the issue!
@U8QTB156K what does clojure -Sdescribe
... yeah, what he asked 🙂
That directory is not the WSL user directory.
I moved it to "~/.clojure" now (I got mixed it up)
Here's some output:
(base) d4hines@ghel:~/.clojure$ clj -Sdescribe
{:version "1.10.1.510"
:config-files ["/usr/local/lib/clojure/deps.edn" "/home/d4hines/.clojure/deps.edn" "deps.edn" ]
:config-user "/home/d4hines/.clojure/deps.edn"
:config-project "deps.edn"
:install-dir "/usr/local/lib/clojure"
:config-dir "/home/d4hines/.clojure"
:cache-dir ".cpcache"
:force false
:repro false
:resolve-aliases ""
:classpath-aliases ""
:jvm-aliases ""
:main-aliases ""
:all-aliases ""}
Yeah, WSL is a bit confusing in that respect since your home directory is not the same as your Windows home directory.
Also, is it clj -A:meander
or clj -Ameander
?
You should use the :
Even tho' it will (mostly) work without.
Ok, so the warning went away, but it still isn't on the classpath:
(base) d4hines@ghel:~$ clj -A:meander
Clojure 1.10.1
user=> (require '[meander.epsilon :as m])
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate meander/epsilon__init.class, meander/epsilon.clj or meander/epsilon.cljc on classpath.
clj -Spath
will tell you your path
Hmmm.
(base) d4hines@ghel:~$ clj -Spath -A:meander
src:/home/d4hines/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/d4hines/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/d4hines/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
Why isn't meander on my path? I added it as :deps {meander/epsilon {:mvn/version "..."}}
clj -Sdeps '{:deps {meander/epsilon {:mvn/version "0.0.389"}}}'
will just load it dynamically
must be something not connecting though
Hey that worked! Now why didn't the alias work...
clj -Sverbose
will tell you each of the places it's looking for deps
oh, maybe you've got a cached classpath
I found the issue. I hadn't actually added the alias! I added it to the wrong file
It all started because I forgot my home folder is different on WSL than the host machine.
Sweet! Now clj -A:meander
works. Thanks all!
@scott.archer did anyone figure out the reitit thing? I think your middleware vector does not need subvecs unless you are passing args*
I'll check, but I think I'm just going to give up and use compojure. I've had a really hard time getting something simple working. I found an article on medium that walks through building a simple API and it looks promising. https://medium.com/swlh/building-a-rest-api-in-clojure-3a1e1ae096e If you have any suggestions or pointers, I'd be glad to take them.
e.g. [middleware1 middleware2 middleware3] instead of the current [[mw1] [mw2] [mw3]]
It’s true that you don’t need to wrap each piece of middleware in its own vector, but I don’t believe it does any harm if you do. If you have middleware that requires arguments, then you have to wrap the middleware+args in a vector to keep them together, but they’re optional if the middleware doesn’t take args.
I don't understand this
why when scientist is passed through openData apparently isn't the same object?
:face_palm: my bad, I have used defnc
instead defn
ha I was about to ask what defnc
was
I'm really noob
[helix.core :as hx :refer [$ <> defnc]]
Hey friends, I'm a bit confused about reducers.
(r/map #(+ % 1) [1 2 3 4 5 6 7 8 9])
gives me an object, which makes sense as it's intended to be composed with other operations. But I can't figure our how to execute it and get the result. Are ruducers only used for operations that, in the end, produce a single result rather than a collection?
For context, I was doing some profiling and found that using pmap instead of map wasn't actually buying me any speed, and concluded that the calculations were too small and numerous for pmap to be the right tool. I saw some stuff about fold, which got me to reducers
Thanks!
(r/fold + (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9]))
executes and gives me an answer, but in my case I don't actually want to reduce
the result set at all
If you want reducer to produce result, you may use into, like that
(into [] (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9]))
That’s because reducers, like transducers, compose in certain way. One reducer may be an argument to another one. That’s why r/map
gives you an object, so it can fed to next reducer.
There’s some explanation here: https://clojure.org/reference/reducers
Aha, thank you! That worked, and thanks as well for the link, maybe it will help me wrap my head around this
Sure. If you want more fun, check out transducers later 🙂 https://clojure.org/reference/transducers
Creating custom ones may be though, but there’s a lot of them ready for you to use. https://github.com/cgrand/xforms seems to be way to go rn
Interesting. So I can see that they support composing really well, is that the main appeal? I can't quite tell how misguided it is to use them just for multithreaded map operations
You can attach a transducer to a channel. So that, data that enters from one end comes out transformed from the other end. There is a perf/memory benefit as well. Multiple operations composed into a transducer happen at one go (but can still be lazy). Intermediate sequences are skipped.
If you want concurrent/parallel execution, you either want r/fold
if your computation fits that model or use executors directly. Transducers have nothing to do with concurrent/parallel execution.
As you've discovered, pmap
is nearly always the wrong solution 🙂
Oh nice, now I need to go read up on whatever executors are! 😆
But if you're not getting speed boosts, its because your mapping operation is too fast
I benchmarked it to determine that pmap wasn't doing me any good, jury is still out on weather threading this problem is just a total waste of time
Yes, I think it's too fast. There are enough elements in the collection to slow my computer down, but they aren't hard to compute
If you have a fast mapping function, but a super large collection, you could still see benefits with a chunked pmap instead
Like just split the list in half, make two threads and process them each in a thread? Is there a specific correct way to implement that pattern or is it manual?
Thank you!
Wouldn't go blocks and channels be a more elegant solution if you really need concurrency?
go
blocks and channels don't give you concurrency unless you use threads as well. Also "Also note that async channels are not intended for fine-grained computational parallelism, though you might see examples in that vein."
I thought go
blocks are concurrent but to actually execute them in parallel, you need multiple threads.
On your second point, looking at various examples I felt go
blocks are lightweight enough to be used liberally. Good thing you pointed that out.
You could try claypoole's upmap https://github.com/TheClimateCorporation/claypoole or pmap
The difference with Clojure's default is that they don't wait for the first to be done to continue mapping
(defn ppmap
"Partitioned pmap, for grouping map ops together to make parallel
overhead worthwhile"
[grain-size f & colls]
(apply concat
(apply pmap
(fn [& pgroups] (doall (apply map f pgroups)))
(map (partial partition-all grain-size) colls))))
(time (dorun (ppmap 1000 clojure.string/lower-case orc-name-abbrevs)))
; => "Elapsed time: 44.902 msecs"
Thanks everyone, I read that chapter and messed around in the repl and I think I have my head around it now next step is to do some serious benchmarking on my application
If reducers is not going to be able to return a collection of the same length as the input collection, are there any other strategies for parallelizing map for large collections of small problems?
As you saw by yourself, parallelizing only makes sense for certain, big amounts of data and costly calculation. What’s actually your case?
Maybe! I need to try this other method to find out!
I am getting weird data back using Postgres with the jTDS driver. I get net.sourceforge.jtds.jdbc.ClobImpl@
. Quick googling shows that this is probably happening because these columns are set to VARCHAR(MAX)
and one solution to this problem is to use useLOBS=false
in my connection string. I dont quite understand this solution and I am wondering if there are any drawbacks to doing this?
I very much doubt you are getting a object of that type back from the postgres jdbc driver
the javadoc for ClobImpl has some methods for justing getting the content of the clob
@mario.cordova.862 Are you using next.jdbc
or clojure.java.jdbc
?
OK. Well, the next.jdbc
docs explain how to work with CLOBs https://cljdoc.org/d/seancorfield/next.jdbc/1.0.13/doc/getting-started/friendly-sql-functions#clob--blob-sql-types and you could do something similar with c.j.j.
It's a different protocol in c.j.j. but the same principle applies to turn CLOBs into regular strings.