Fork me on GitHub
#beginners
<
2020-02-27
>
Daniel Hines00:02:53

I have the following deps.edn file in .clojure:

{:aliases
  {:meander {:deps {meander/epsilon {:mvn/version "0.0.389"}}}}}
But when I run clj -A:meander, I get:
WARNING: Specified aliases are undeclared: [:meander]
What am I doing wrong?

andy.fingerhut01:02:18

Is that deps.edn file in a directory named .clojure in your home directory? What OS are you using?

Daniel Hines01:02:49

Ubuntu (on WSL)

Daniel Hines01:02:26

(base) d4hines@ghel:/mnt/c/Users/d4hin$ ls .clojure/
deps.edn
(base) d4hines@ghel:/mnt/c/Users/d4hin$
Looks right to me

andy.fingerhut01:02:44

What is the output of the command clojure -Sdescribe ? You need not share it here if you prefer not to, but the output should include the full path names to files that it reads.

Daniel Hines01:02:50

Ahhh I see the issue!

seancorfield01:02:59

@U8QTB156K what does clojure -Sdescribe ... yeah, what he asked 🙂

seancorfield01:02:00

That directory is not the WSL user directory.

Daniel Hines01:02:45

I moved it to "~/.clojure" now (I got mixed it up)

Daniel Hines01:02:55

Here's some output:

(base) d4hines@ghel:~/.clojure$ clj -Sdescribe
{:version "1.10.1.510"
 :config-files ["/usr/local/lib/clojure/deps.edn" "/home/d4hines/.clojure/deps.edn" "deps.edn" ]
 :config-user "/home/d4hines/.clojure/deps.edn"
 :config-project "deps.edn"
 :install-dir "/usr/local/lib/clojure"
 :config-dir "/home/d4hines/.clojure"
 :cache-dir ".cpcache"
 :force false
 :repro false
 :resolve-aliases ""
 :classpath-aliases ""
 :jvm-aliases ""
 :main-aliases ""
 :all-aliases ""}

seancorfield01:02:26

Yeah, WSL is a bit confusing in that respect since your home directory is not the same as your Windows home directory.

Daniel Hines01:02:56

Also, is it clj -A:meander or clj -Ameander?

seancorfield01:02:04

You should use the :

seancorfield01:02:15

Even tho' it will (mostly) work without.

Daniel Hines01:02:39

Ok, so the warning went away, but it still isn't on the classpath:

(base) d4hines@ghel:~$ clj -A:meander
Clojure 1.10.1
user=> (require '[meander.epsilon :as m])
Execution error (FileNotFoundException) at user/eval1 (REPL:1).
Could not locate meander/epsilon__init.class, meander/epsilon.clj or meander/epsilon.cljc on classpath.

Alex Miller (Clojure team)01:02:01

clj -Spath will tell you your path

Daniel Hines01:02:44

Hmmm.

(base) d4hines@ghel:~$ clj -Spath -A:meander
src:/home/d4hines/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar:/home/d4hines/.m2/repository/org/clojure/spec.alpha/0.2.176/spec.alpha-0.2.176.jar:/home/d4hines/.m2/repository/org/clojure/core.specs.alpha/0.2.44/core.specs.alpha-0.2.44.jar
Why isn't meander on my path? I added it as :deps {meander/epsilon {:mvn/version "..."}}

Alex Miller (Clojure team)01:02:23

clj -Sdeps '{:deps {meander/epsilon {:mvn/version "0.0.389"}}}' will just load it dynamically

Alex Miller (Clojure team)01:02:40

must be something not connecting though

Daniel Hines01:02:51

Hey that worked! Now why didn't the alias work...

Alex Miller (Clojure team)01:02:33

clj -Sverbose will tell you each of the places it's looking for deps

Alex Miller (Clojure team)01:02:58

oh, maybe you've got a cached classpath

Daniel Hines01:02:00

I found the issue. I hadn't actually added the alias! I added it to the wrong file

Daniel Hines01:02:28

It all started because I forgot my home folder is different on WSL than the host machine.

Daniel Hines01:02:34

Sweet! Now clj -A:meander works. Thanks all!

sova-soars-the-sora16:02:23

@scott.archer did anyone figure out the reitit thing? I think your middleware vector does not need subvecs unless you are passing args*

👍 4
ScArcher16:02:21

I'll check, but I think I'm just going to give up and use compojure. I've had a really hard time getting something simple working. I found an article on medium that walks through building a simple API and it looks promising. https://medium.com/swlh/building-a-rest-api-in-clojure-3a1e1ae096e If you have any suggestions or pointers, I'd be glad to take them.

sova-soars-the-sora16:02:13

e.g. [middleware1 middleware2 middleware3] instead of the current [[mw1] [mw2] [mw3]]

manutter5116:02:33

It’s true that you don’t need to wrap each piece of middleware in its own vector, but I don’t believe it does any harm if you do. If you have middleware that requires arguments, then you have to wrap the middleware+args in a vector to keep them together, but they’re optional if the middleware doesn’t take args.

🎯 4
Luis C. Arbildo16:02:15

I don't understand this

Luis C. Arbildo16:02:34

why when scientist is passed through openData apparently isn't the same object?

Luis C. Arbildo16:02:38

:face_palm: my bad, I have used defnc instead defn

Noah Bogart16:02:48

ha I was about to ask what defnc was

Luis C. Arbildo16:02:23

[helix.core :as hx :refer [$ <> defnc]]

Michael J Dorian17:02:27

Hey friends, I'm a bit confused about reducers. (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9]) gives me an object, which makes sense as it's intended to be composed with other operations. But I can't figure our how to execute it and get the result. Are ruducers only used for operations that, in the end, produce a single result rather than a collection? For context, I was doing some profiling and found that using pmap instead of map wasn't actually buying me any speed, and concluded that the calculations were too small and numerous for pmap to be the right tool. I saw some stuff about fold, which got me to reducers Thanks!

Michael J Dorian17:02:52

(r/fold + (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9])) executes and gives me an answer, but in my case I don't actually want to reduce the result set at all

fricze18:02:42

If you want reducer to produce result, you may use into, like that (into [] (r/map #(+ % 1) [1 2 3 4 5 6 7 8 9]))

fricze18:02:12

That’s because reducers, like transducers, compose in certain way. One reducer may be an argument to another one. That’s why r/map gives you an object, so it can fed to next reducer.

fricze18:02:28

into actually says: enough composing, let’s produce the data

Michael J Dorian18:02:32

Aha, thank you! That worked, and thanks as well for the link, maybe it will help me wrap my head around this

fricze18:02:12

Sure. If you want more fun, check out transducers later 🙂 https://clojure.org/reference/transducers

👍 4
fricze18:02:40

they’re, currently, the sexiest stuff for composing data transformation

fricze18:02:23

Creating custom ones may be though, but there’s a lot of them ready for you to use. https://github.com/cgrand/xforms seems to be way to go rn

Michael J Dorian18:02:32

Interesting. So I can see that they support composing really well, is that the main appeal? I can't quite tell how misguided it is to use them just for multithreaded map operations simple_smile

hindol18:02:18

You can attach a transducer to a channel. So that, data that enters from one end comes out transformed from the other end. There is a perf/memory benefit as well. Multiple operations composed into a transducer happen at one go (but can still be lazy). Intermediate sequences are skipped.

seancorfield18:02:48

If you want concurrent/parallel execution, you either want r/fold if your computation fits that model or use executors directly. Transducers have nothing to do with concurrent/parallel execution.

seancorfield18:02:11

As you've discovered, pmap is nearly always the wrong solution 🙂

didibus18:02:54

Actually, I do think you want pmap in this case

Michael J Dorian18:02:15

Oh nice, now I need to go read up on whatever executors are! 😆

didibus18:02:43

But if you're not getting speed boosts, its because your mapping operation is too fast

didibus18:02:55

Faster then it is to switch threads

Michael J Dorian18:02:59

I benchmarked it to determine that pmap wasn't doing me any good, jury is still out on weather threading this problem is just a total waste of time simple_smile

Michael J Dorian18:02:34

Yes, I think it's too fast. There are enough elements in the collection to slow my computer down, but they aren't hard to compute

didibus18:02:44

If you have a fast mapping function, but a super large collection, you could still see benefits with a chunked pmap instead

didibus18:02:10

Hum... I think brave abd true shows an example of it

Michael J Dorian18:02:19

Like just split the list in half, make two threads and process them each in a thread? Is there a specific correct way to implement that pattern or is it manual?

didibus18:02:20

Let me try and find it

didibus18:02:39

Well, basically you want to call partition on tbe input first

didibus18:02:52

So that you paralelize but in batches

hindol18:02:42

Wouldn't go blocks and channels be a more elegant solution if you really need concurrency?

seancorfield19:02:36

go blocks and channels don't give you concurrency unless you use threads as well. Also "Also note that async channels are not intended for fine-grained computational parallelism, though you might see examples in that vein."

hindol19:02:19

I thought go blocks are concurrent but to actually execute them in parallel, you need multiple threads.

hindol19:02:15

On your second point, looking at various examples I felt go blocks are lightweight enough to be used liberally. Good thing you pointed that out.

didibus07:02:34

They're concurrent but not parallel

didibus07:02:46

Well, not necessarily parallel

didibus07:02:05

They are actually n:m so there's some parallelism involved

didibus07:02:01

I can't seem to find the batch pmap I was looking for hum...

didibus07:02:21

The difference with Clojure's default is that they don't wait for the first to be done to continue mapping

didibus07:02:42

It's not what I was talking about though.

didibus07:02:48

(defn ppmap
  "Partitioned pmap, for grouping map ops together to make parallel
  overhead worthwhile"
  [grain-size f & colls]
  (apply concat
   (apply pmap
          (fn [& pgroups] (doall (apply map f pgroups)))
          (map (partial partition-all grain-size) colls))))
(time (dorun (ppmap 1000 clojure.string/lower-case orc-name-abbrevs)))
; => "Elapsed time: 44.902 msecs"

didibus07:02:25

It's in chapter 10 of brave and true

Michael J Dorian14:03:44

Thanks everyone, I read that chapter and messed around in the repl and I think I have my head around it now simple_smile next step is to do some serious benchmarking on my application

Michael J Dorian17:02:30

If reducers is not going to be able to return a collection of the same length as the input collection, are there any other strategies for parallelizing map for large collections of small problems?

fricze18:02:07

As you saw by yourself, parallelizing only makes sense for certain, big amounts of data and costly calculation. What’s actually your case?

Michael J Dorian18:02:26

simple_smile Maybe! I need to try this other method to find out!

Mario C.19:02:13

I am getting weird data back using Postgres with the jTDS driver. I get net.sourceforge.jtds.jdbc.ClobImpl@. Quick googling shows that this is probably happening because these columns are set to VARCHAR(MAX) and one solution to this problem is to use useLOBS=false in my connection string. I dont quite understand this solution and I am wondering if there are any drawbacks to doing this?

hiredman19:02:13

I very much doubt you are getting a object of that type back from the postgres jdbc driver

hiredman19:02:00

some googling says jtds is a java package from microsoft sql server

Mario C.19:02:13

Yes sorry meant jTDS driver

hiredman19:02:06

I feel like I have to ask, have you considered using the postgres jdbc driver?

hiredman19:02:48

the javadoc for ClobImpl has some methods for justing getting the content of the clob

Mario C.19:02:34

hmm well I am using both

Mario C.19:02:47

Using the jTDS to connect to sql server though

seancorfield19:02:02

@mario.cordova.862 Are you using next.jdbc or clojure.java.jdbc?

Mario C.19:02:05

clojure.java.jdbc

seancorfield19:02:36

OK. Well, the next.jdbc docs explain how to work with CLOBs https://cljdoc.org/d/seancorfield/next.jdbc/1.0.13/doc/getting-started/friendly-sql-functions#clob--blob-sql-types and you could do something similar with c.j.j.

seancorfield19:02:31

It's a different protocol in c.j.j. but the same principle applies to turn CLOBs into regular strings.

Mario C.20:02:47

Thanks! I had a feeling I would have to do some protocol extensions. Let me look into this