Fork me on GitHub
#beginners
<
2020-02-27
>
d4hines00:02:53

I have the following deps.edn file in .clojure:

{:aliases
  {:meander {:deps {meander/epsilon {:mvn/version "0.0.389"}}}}}
But when I run clj -A:meander, I get:
WARNING: Specified aliases are undeclared: [:meander]
What am I doing wrong?

andy.fingerhut01:02:18

Is that deps.edn file in a directory named .clojure in your home directory? What OS are you using?

d4hines01:02:49

Ubuntu (on WSL)

d4hines01:02:26

(base) [email protected]:/mnt/c/Users/d4hin$ ls .clojure/
deps.edn
(base) [email protected]:/mnt/c/Users/d4hin$
Looks right to me

andy.fingerhut01:02:44

What is the output of the command clojure -Sdescribe ? You need not share it here if you prefer not to, but the output should include the full path names to files that it reads.

d4hines01:02:50

Ahhh I see the issue!

seancorfield01:02:59

@ what does clojure -Sdescribe ... yeah, what he asked 🙂

seancorfield01:02:00

That directory is not the WSL user directory.

d4hines01:02:45

I moved it to "~/.clojure" now (I got mixed it up)

d4hines01:02:55

Here's some output:

(base) [email protected]:~/.clojure$ clj -Sdescribe
{:version "1.10.1.510"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/d4hines/.clojure/deps.edn" "deps.edn" ]
 :config-user "/home/d4hines/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/d4hines/.clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :resolve-aliases ""
 :classpath-aliases ""
 :jvm-aliases ""
 :main-aliases ""
 :all-aliases ""}

seancorfield01:02:26

Yeah, WSL is a bit confusing in that respect since your home directory is not the same as your Windows home directory.

d4hines01:02:56

Also, is it clj -A:meander or clj -Ameander?

seancorfield01:02:04

You should use the :

seancorfield01:02:15

Even tho' it will (mostly) work without.

d4hines01:02:39

Ok, so the warning went away, but it still isn't on the classpath:

(base) [email protected]:~$ clj -A:meander
Clojure 1.10.1
user=> (require '[meander.epsilon :as m])
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate meander/epsilon__init.class, meander/epsilon.clj or meander/epsilon.cljc on classpath.

alexmiller01:02:01

clj -Spath will tell you your path

d4hines01:02:44

Hmmm.

(base) [email protected]:~$ clj -Spath -A:meander
src:/home/d4hines/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/d4hines/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/d4hines/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
Why isn't meander on my path? I added it as :deps {meander/epsilon {:mvn/version "..."}}

alexmiller01:02:23

clj -Sdeps '{:deps {meander/epsilon {:mvn/version "0.0.389"}}}' will just load it dynamically

alexmiller01:02:40

must be something not connecting though

d4hines01:02:51

Hey that worked! Now why didn't the alias work...

alexmiller01:02:33

clj -Sverbose will tell you each of the places it's looking for deps

alexmiller01:02:58

oh, maybe you've got a cached classpath

d4hines01:02:00

I found the issue. I hadn't actually added the alias! I added it to the wrong file

d4hines01:02:28

It all started because I forgot my home folder is different on WSL than the host machine.

d4hines01:02:34

Sweet! Now clj -A:meander works. Thanks all!

alexmiller02:02:54

happy hacking!

sova16:02:23

@scott.archer did anyone figure out the reitit thing? I think your middleware vector does not need subvecs unless you are passing args*

scott.archer16:02:21

I'll check, but I think I'm just going to give up and use compojure. I've had a really hard time getting something simple working. I found an article on medium that walks through building a simple API and it looks promising. https://medium.com/swlh/building-a-rest-api-in-clojure-3a1e1ae096e If you have any suggestions or pointers, I'd be glad to take them.

sova16:02:13

e.g. [middleware1 middleware2 middleware3] instead of the current [[mw1] [mw2] [mw3]]

manutter5116:02:33

It’s true that you don’t need to wrap each piece of middleware in its own vector, but I don’t believe it does any harm if you do. If you have middleware that requires arguments, then you have to wrap the middleware+args in a vector to keep them together, but they’re optional if the middleware doesn’t take args.

luis.cairampoma11916:02:15

I don't understand this

luis.cairampoma11916:02:34

why when scientist is passed through openData apparently isn't the same object?

luis.cairampoma11916:02:38

:face_palm: my bad, I have used defnc instead defn

nbtheduke16:02:48

ha I was about to ask what defnc was

luis.cairampoma11916:02:23

[helix.core :as hx :refer [$ <> defnc]]

doby16217:02:27

Hey friends, I'm a bit confused about reducers. (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9]) gives me an object, which makes sense as it's intended to be composed with other operations. But I can't figure our how to execute it and get the result. Are ruducers only used for operations that, in the end, produce a single result rather than a collection? For context, I was doing some profiling and found that using pmap instead of map wasn't actually buying me any speed, and concluded that the calculations were too small and numerous for pmap to be the right tool. I saw some stuff about fold, which got me to reducers Thanks!

doby16217:02:52

(r/fold + (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9])) executes and gives me an answer, but in my case I don't actually want to reduce the result set at all

andrzej.fricze18:02:42

If you want reducer to produce result, you may use into, like that (into [] (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9]))

andrzej.fricze18:02:12

That’s because reducers, like transducers, compose in certain way. One reducer may be an argument to another one. That’s why r/map gives you an object, so it can fed to next reducer.

andrzej.fricze18:02:28

into actually says: enough composing, let’s produce the data

doby16218:02:32

Aha, thank you! That worked, and thanks as well for the link, maybe it will help me wrap my head around this

andrzej.fricze18:02:12

Sure. If you want more fun, check out transducers later 🙂 https://clojure.org/reference/transducers

andrzej.fricze18:02:40

they’re, currently, the sexiest stuff for composing data transformation

andrzej.fricze18:02:23

Creating custom ones may be though, but there’s a lot of them ready for you to use. https://github.com/cgrand/xforms seems to be way to go rn

doby16218:02:32

Interesting. So I can see that they support composing really well, is that the main appeal? I can't quite tell how misguided it is to use them just for multithreaded map operations simple_smile

hindol.adhya18:02:18

You can attach a transducer to a channel. So that, data that enters from one end comes out transformed from the other end. There is a perf/memory benefit as well. Multiple operations composed into a transducer happen at one go (but can still be lazy). Intermediate sequences are skipped.

seancorfield18:02:48

If you want concurrent/parallel execution, you either want r/fold if your computation fits that model or use executors directly. Transducers have nothing to do with concurrent/parallel execution.

seancorfield18:02:11

As you've discovered, pmap is nearly always the wrong solution 🙂

didibus18:02:54

Actually, I do think you want pmap in this case

doby16218:02:15

Oh nice, now I need to go read up on whatever executors are! 😆

didibus18:02:43

But if you're not getting speed boosts, its because your mapping operation is too fast

didibus18:02:55

Faster then it is to switch threads

doby16218:02:59

I benchmarked it to determine that pmap wasn't doing me any good, jury is still out on weather threading this problem is just a total waste of time simple_smile

doby16218:02:34

Yes, I think it's too fast. There are enough elements in the collection to slow my computer down, but they aren't hard to compute

didibus18:02:44

If you have a fast mapping function, but a super large collection, you could still see benefits with a chunked pmap instead

didibus18:02:10

Hum... I think brave abd true shows an example of it

doby16218:02:19

Like just split the list in half, make two threads and process them each in a thread? Is there a specific correct way to implement that pattern or is it manual?

didibus18:02:20

Let me try and find it

doby16218:02:26

Thank you!

didibus18:02:39

Well, basically you want to call partition on tbe input first

didibus18:02:52

So that you paralelize but in batches

hindol.adhya18:02:42

Wouldn't go blocks and channels be a more elegant solution if you really need concurrency?

seancorfield19:02:36

go blocks and channels don't give you concurrency unless you use threads as well. Also "Also note that async channels are not intended for fine-grained computational parallelism, though you might see examples in that vein."

hindol.adhya19:02:19

I thought go blocks are concurrent but to actually execute them in parallel, you need multiple threads.

hindol.adhya19:02:15

On your second point, looking at various examples I felt go blocks are lightweight enough to be used liberally. Good thing you pointed that out.

didibus07:02:34

They're concurrent but not parallel

didibus07:02:46

Well, not necessarily parallel

didibus07:02:05

They are actually n:m so there's some parallelism involved

didibus07:02:01

I can't seem to find the batch pmap I was looking for hum...

didibus07:02:21

The difference with Clojure's default is that they don't wait for the first to be done to continue mapping

didibus07:02:42

It's not what I was talking about though.

didibus07:02:48

(defn ppmap
  "Partitioned pmap, for grouping map ops together to make parallel
  overhead worthwhile"
  [grain-size f & colls]
  (apply concat
   (apply pmap
          (fn [& pgroups] (doall (apply map f pgroups)))
          (map (partial partition-all grain-size) colls))))
(time (dorun (ppmap 1000 clojure.string/lower-case orc-name-abbrevs)))
; => "Elapsed time: 44.902 msecs"

didibus07:02:25

It's in chapter 10 of brave and true

doby16214:03:44

Thanks everyone, I read that chapter and messed around in the repl and I think I have my head around it now simple_smile next step is to do some serious benchmarking on my application

doby16217:02:30

If reducers is not going to be able to return a collection of the same length as the input collection, are there any other strategies for parallelizing map for large collections of small problems?

andrzej.fricze18:02:07

As you saw by yourself, parallelizing only makes sense for certain, big amounts of data and costly calculation. What’s actually your case?

doby16218:02:26

simple_smile Maybe! I need to try this other method to find out!

mario.cordova.86219:02:13

I am getting weird data back using Postgres with the jTDS driver. I get [email protected]. Quick googling shows that this is probably happening because these columns are set to VARCHAR(MAX) and one solution to this problem is to use useLOBS=false in my connection string. I dont quite understand this solution and I am wondering if there are any drawbacks to doing this?

hiredman19:02:13

I very much doubt you are getting a object of that type back from the postgres jdbc driver

hiredman19:02:00

some googling says jtds is a java package from microsoft sql server

mario.cordova.86219:02:13

Yes sorry meant jTDS driver

hiredman19:02:06

I feel like I have to ask, have you considered using the postgres jdbc driver?

hiredman19:02:48

the javadoc for ClobImpl has some methods for justing getting the content of the clob

mario.cordova.86219:02:34

hmm well I am using both

mario.cordova.86219:02:47

Using the jTDS to connect to sql server though

seancorfield19:02:02

@mario.cordova.862 Are you using next.jdbc or clojure.java.jdbc?

mario.cordova.86219:02:05

clojure.java.jdbc

seancorfield19:02:36

OK. Well, the next.jdbc docs explain how to work with CLOBs https://cljdoc.org/d/seancorfield/next.jdbc/1.0.13/doc/getting-started/friendly-sql-functions#clob--blob-sql-types and you could do something similar with c.j.j.

seancorfield19:02:31

It's a different protocol in c.j.j. but the same principle applies to turn CLOBs into regular strings.

mario.cordova.86220:02:47

Thanks! I had a feeling I would have to do some protocol extensions. Let me look into this