Fork me on GitHub
#beginners
<
2020-12-29
>
roelof09:12:34

why do I get here a classcastexception on the let part :

(def work (ref (apply list (range 1e5))))


; And the sum will be a ref as well:
(def sum4 (ref 0))

; Write a function which, in a dosync transaction, removes the first number in 
; work and adds it to sum. Then, in two futures, call that function over and 
; over again until there's no work left. Verify that @sum is 4999950000. 
; Experiment with different combinations of alter and commute–if both are 
; correct, is one faster? Does using deref instead of ensure change the result?

(defn add-dosync [work sum4]
  (dosync
  (alter (ensure sum4) + (first @work))
  (ref-set work (next @work))))


(let [worker1 (future (while @work (add-dosync work sum4)))
      worker2 (future (while @work (add-dosync work sum4)))]
@worker1 ;wait for worker1
@worker2 ;wait for worker2
@sum4) ;return the result of their work

(deref sum4)

Ben Sless10:12:32

@marcus.akre you can always pick up where Tim Baldridge left off here https://github.com/halgari/com.tbaldridge.hermod

euccastro10:12:39

@roelof (ensure sum4) returns the current value of sum4 , which is a long. alter takes a ref, not a long

roelof10:12:38

oke, then I have to think how to solve that

roelof10:12:57

as far as I know I have to use ensure somewhere

noisesmith16:12:20

that's incorrect - ensure is only needed for values you are not altering but only reading

roelof17:12:21

then I misunderstood the book

roelof17:12:32

thanks for pointing this to me

euccastro10:12:19

you should use ensure on the ref you're reading the value of in the dosync body, which is not sum4

roelof10:12:22

sorry, I do not

roelof10:12:42

but still if I change the code to this :

(defn add-dosync [work-ref sum-ref]
  (dosync
  (alter sum-ref + (first @work-ref))
  (ref-set work-ref (next @work-ref)))

roelof10:12:54

I get the classCastException

euccastro10:12:02

also, I recommend using alter too to update work: (alter work next)

roelof10:12:56

also changed

euccastro10:12:01

are you sure you have reevaluated add-dosync after the change?

roelof10:12:27

I re-evaluted everything after a change

euccastro10:12:26

one additional problem may be that (first @work-ref) may be null. maybe the easiest way around that would be to check for that condition inside the dosync body

roelof10:12:03

oke, you mean if the work collection is empty ?

euccastro10:12:08

that's even though you're checking in the future bodies, but there's a possible race condition there; you shouldn't rely on that

roelof10:12:01

the exact error message is :

; Execution error (ClassCastException) at ground-up.chapter6/add-dosync$fn (form-init16078161836259666359.clj:80).
; class java.lang.Long cannot be cast to class clojure.lang.Ref (java.lang.Long is in module java.base of loader 'bootstrap'; clojure.lang.Ref is in unnamed module of loader 'app')

euccastro10:12:02

that check is OK to make the future finish when it's done, but you are not guaranteed that work won't be completed between the check and the execution of add-dosync

roelof10:12:10

oke, so you are thinking I have to check for if work is empty also in the do-sync ?

euccastro10:12:40

yes, but that doesn't look like it could cause the particular error you're getting; you should get a nullpointerexception instead

euccastro10:12:13

can you paste the whole program again?

roelof10:12:17

oke, and sentence 80 is alter sum-ref + (first @work-ref))

roelof10:12:42

of course

; Write a function which, in a dosync transaction, removes the first number in 
; work and adds it to sum. Then, in two futures, call that function over and 
; over again until there's no work left. Verify that @sum is 4999950000. 
; Experiment with different combinations of alter and commute–if both are 
; correct, is one faster? Does using deref instead of ensure change the result?

(defn add-dosync [work-ref sum-ref]
  (dosync
  (alter sum-ref + (first @work-ref))
  (alter work-ref (next @work-ref))))

(let [worker1 (future (while @work (add-dosync work @sum4)))
      worker2 (future (while @work (add-dosync work @sum4)))]
@worker1 ;wait for worker1
@worker2 ;wait for worker2
@sum4) ;return the result of their work

(deref sum4)

euccastro10:12:00

remove the @ before sum4 in the future bodies

roelof10:12:52

same error

euccastro10:12:53

that's causing the same problem you were getting before by calling (alter (ensure sum4) ,,,)

roelof10:12:02

Deleting that did not solve the eror message

euccastro10:12:12

have you reevaluated the whole program?

roelof10:12:27

yep, as I said I always do that

euccastro10:12:23

(alter work-ref (next @work-ref)) should be just (alter work-ref next)

roelof10:12:34

now I see this error :

; Execution error (ClassCastException) at ground-up.chapter6/add-dosync$fn (form-init16078161836259666359.clj:81).
; class clojure.lang.PersistentList cannot be cast to class clojure.lang.IFn (clojure.lang.PersistentList and clojure.lang.IFn are in unnamed module of loader 'app')

euccastro11:12:03

yes, see my latest comment

euccastro11:12:25

alter takes a ref and a function, not a value

roelof11:12:41

now a null pointer exception as you said earlier

euccastro11:12:33

so you can check for a null (first @work-ref) and just exit in that case

roelof11:12:06

yep, finally after some many days

🎉 3
roelof11:12:37

need a break

roelof11:12:00

changed it to :

(defn add-dosync [work-ref sum-ref]
  (when (seq @work-ref)
    (dosync
     (alter sum-ref + (first @work-ref))
     (alter work-ref next))))

roelof11:12:10

but still the null pointer exception

euccastro11:12:02

you should definitely check inside the dosync

euccastro11:12:06

probably using ensure instead of deref/`@`(but give both a try to verify that ensure is needed)

roelof11:12:59

I tried this :

(defn add-dosync [work-ref sum-ref]
  (dosync
   (while @work-ref (alter sum-ref + (first @work-ref)))
   (while @work-ref (alter work-ref next))))

roelof11:12:08

but it seems this is running forever

roelof11:12:23

finallly working

roelof11:12:26

(defn add-dosync [work-ref sum-ref]
  (dosync
   (when @work-ref (alter sum-ref + (first @work-ref)))
   (when @work-ref (alter work-ref next))))

roelof11:12:11

and alter or commute makes no difference. the same output

roelof11:12:52

hmm, but both the wrong answer 😞

Timur Latypoff11:12:29

@roelof Please try the following

(defn add-dosync [work-ref sum-ref]
  (dosync
    (when-let [work (seq (ensure work-ref))]
      (alter sum-ref + (first work))
      (ref-set work-ref (next work)))))
— a a single check for non-empty sequence, — single ensure

roelof11:12:48

it should be 4999950000 and both gives 4999984077

Timur Latypoff11:12:48

Not sure if this question was discussed before thought (can't read all those pages above), but this is also not a good case for using refs, maybe only as an exercise. Because this algorithm of summing a list does not benefit from parallelization at all (all entries are processed one by one sequentially anyway).

roelof11:12:09

I know and this is discuused before

👍 3
roelof11:12:21

but when I try your code it seems to run forever

Timur Latypoff11:12:31

Ok, let me fire up the repl 🙂

roelof11:12:33

I still do not see a answer

Timur Latypoff11:12:34

Would you please paste the code?

roelof11:12:16

(def work (ref (apply list (range 1e5))))


; And the sum will be a ref as well:
(def sum4 (ref 0))

; Write a function which, in a dosync transaction, removes the first number in 
; work and adds it to sum. Then, in two futures, call that function over and 
; over again until there's no work left. Verify that @sum is 4999950000. 
; Experiment with different combinations of alter and commute–if both are 
; correct, is one faster? Does using deref instead of ensure change the result?

(defn add-dosync [work-ref sum-ref]
  (dosync
    (when-let [work (seq (ensure work-ref))]
      (alter sum-ref + (first work))
      (ref-set work-ref (next work)))))


(let [worker1 (future (while @work (add-dosync work sum4)))
      worker2 (future (while @work (add-dosync work sum4)))]
@worker1 ;wait for worker1
@worker2 ;wait for worker2
@sum4) ;return the result of their work

Timur Latypoff11:12:51

@roelof from what I see, it's not running forever, it's just being REALLY slow 🙂

roelof11:12:32

oke, I will try again and wait for the answer

roelof11:12:39

and see if it's the right one

roelof11:12:52

does ensure makes is this slow

Timur Latypoff11:12:15

No idea, first time using refs myself. I'm not an expert, I was just curious 🙂

roelof11:12:57

me too first time use it. The "Clojure from the ground up" was telling me to use a ref

Timur Latypoff11:12:33

We'll crack it, no worries 🙂

roelof11:12:56

yep , here still no answer 🙂

Timur Latypoff11:12:16

Overall, ref seems like a very heavy-weight tool, that's why it's rarely used, and people use atoms instead.

roelof11:12:40

now long did I take in min

roelof11:12:54

I think im now at 2 - 3 min

Timur Latypoff11:12:42

I think around that, more or less.

roelof11:12:13

then you have a faster computer then I

roelof11:12:23

here still no answer

Timur Latypoff11:12:11

Yeah, 2:30 on my macbook, but I ran it as uberjar.

roelof11:12:00

aha, and I did it on repl

roelof11:12:25

did it again and it's now at 26.000

roelof11:12:54

so 26 thousand

Timur Latypoff11:12:38

Anyway, I think it’s not a real evaluation of speed of how refs work. Here the workload is so that two CPU threads are constantly fighting and re-doing each-other’s work. I would expect that on a single-core CPU the result will be calculated much faster :)

roelof11:12:28

yep I know

roelof11:12:50

When you do this with reduce it takes almost no time

roelof12:12:28

and I have a 4 core machine so a lot of fighting

roelof12:12:04

84 thousand

roelof12:12:16

and one with the right answer

roelof12:12:26

in repl not on vs code

roelof12:12:18

a lot learned and a lot to learn to really understand closures, ref and so on

roelof12:12:29

I found this a very hard chapter

roelof12:12:44

thanks a lot with your patience

👍 3
euccastro12:12:27

try commute instead of alter, see how fast/correct it is?

roelof12:12:26

I will if I have the layout of vs code right

roelof12:12:52

on some wierd way repl is now a tab instead of a seperate window

roelof12:12:50

after lunch I have to figure out how to make this work

roelof12:12:54

(defn add-dosync [work-ref sum-ref]
  (dosync
    (when-let [work (seq (commute work-ref))]
      (alter sum-ref + (first work))
      (ref-set work-ref (next work)))))

roelof12:12:13

at some way I have to give commute another argument

Timur Latypoff12:12:15

You should use commute instead of alter

roelof12:12:02

it seems to work faster

roelof12:12:23

wait for the answer

Timur Latypoff12:12:06

Cool. Good to know. I am not yet proficient enough to know when commute will do, and when alter is required to get a correct result.

roelof12:12:54

According to the book alter/ensure if the order is important and commute when the order is not important

roelof12:12:38

so as a example 1 + 2 + 3 the order is not important because also 1 + 3 + 2 - 6 also 3 + 1 + 2 = 6

👍 3
Timur Latypoff13:12:00

Ok, makes sense

roelof13:12:11

but on this the order is important (1/2)/3 in not the same as 1/ (2/3)

roelof13:12:19

that 0.16 or 0,666 if im not mistaken

roelof13:12:16

this :

(defn add-dosync [work-ref sum-ref]
  (dosync
    (when-let [work (seq (ensure work-ref))]
      (alter sum-ref + (first work))
      (ref-set work-ref (next work)))))
and this :
(defn add-dosync [work-ref sum-ref]
  (dosync
    (when-let [work (seq (ensure work-ref))]
      (commute sum-ref + (first work))
      (ref-set work-ref (next work)))))

roelof13:12:33

give the same right outcome

👍 3
roelof13:12:01

Did I do it right now ?

euccastro14:12:52

you could use (commute work-ref next) too

roelof14:12:32

also you thanks for the patience with me

Fra11:12:38

Hi, I’ve been trying the fox-goose-corn kata and worked out this solution - https://paste.ofcode.org/mpzmsutwdEK3GczkzjZ6Tx I guess there are still a few things wrong…

Fra11:12:42

Given a configuration, calc-path tries to generate the possible options using for and evaluate them recursively. If a child configuration is invalid or already explored, returns [] . If the child configuration has all the items on the right bank, returns the entire path. Because I’m using for, I had to apply a few quick fixes to make it work, for example I use remove empty? at line 80 and 86. Another one is at line 115. The path is inside a series of lists ((((path)))) so I had to find a way to extract it.

Fra12:12:39

I wonder if there’s a better alternative to for to reach the same goal and avoid these those confusing lines

Timur Latypoff11:12:57
replied to a thread:I still do not see a answer

Result looks correct @roelof

GGfpc12:12:51

Hello! Can someone help me debug this error? Here is my code

(defn get-books-with-extra-data
  [url]
  (let [cached (cache/fetch url)]
    (if (not (nil? cached))
      cached
      (do
        (timbre/info "Fetching book: " url)
        (http/get url {} (fn [{:keys [status headers body error]}]
                           (->>
                             (map :body)
                             (map #(hash-map :book-genres (get-book-genres %) :book-cover (get-book-cover %)))
                             ;(cache/store url)
                             )))))))
Which is called here (I invoked this from the REPL)
(defn books-with-extra-data
  [books]
  (->> books
       (map #(merge %1 (book/get-books-with-extra-data (string/trim (get-in %1 [:book :link])))))))
And I get this strange stacktrace

GGfpc12:12:51

(pst)
clojure.lang.ExceptionInfo: null
                                      main.clj:442 clojure.main/repl[fn]
                                      main.clj:458 clojure.main/repl[fn]
                                      main.clj:458 clojure.main/repl
                                      main.clj:368 clojure.main/repl
                                  RestFn.java:1523 clojure.lang.RestFn.invoke
                         interruptible_eval.clj:79 nrepl.middleware.interruptible-eval/evaluate
                         interruptible_eval.clj:55 nrepl.middleware.interruptible-eval/evaluate
                        interruptible_eval.clj:142 nrepl.middleware.interruptible-eval/interruptible-eval[fn]
                                       AFn.java:22 clojure.lang.AFn.run
                                   session.clj:171 nrepl.middleware.session/session-exec[fn]
                                   session.clj:170 nrepl.middleware.session/session-exec[fn]
                                       AFn.java:22 clojure.lang.AFn.run
                                   Thread.java:748 java.lang.Thread.run
Caused by: java.lang.IllegalArgumentException: Don't know how to create ISeq from: org.httpkit.client$deadlock_guard$reify__17177
                                       RT.java:557 clojure.lang.RT.seqFrom
                                       RT.java:537 clojure.lang.RT.seq
                            APersistentMap.java:40 clojure.lang.APersistentMap.cons
                                       RT.java:677 clojure.lang.RT.conj
                                       core.clj:85 clojure.core/conj
                                     core.clj:3049 clojure.core/merge[fn]
                                      core.clj:944 clojure.core/reduce1
                                      core.clj:934 clojure.core/reduce1
                                     core.clj:3048 clojure.core/merge
                                     core.clj:3041 clojure.core/merge
                                   RestFn.java:421 clojure.lang.RestFn.invoke
                                       stats.clj:2 goodstats.consumer.stats/eval17623[fn]
                                     core.clj:2755 clojure.core/map[fn]
                                   LazySeq.java:42 clojure.lang.LazySeq.sval
                                   LazySeq.java:51 clojure.lang.LazySeq.seq
                                       RT.java:535 clojure.lang.RT.seq
                                      core.clj:137 clojure.core/seq
                                 core_print.clj:53 clojure.core/print-sequential
                                core_print.clj:174 clojure.core/fn
                                core_print.clj:174 clojure.core/fn
                                  MultiFn.java:234 clojure.lang.MultiFn.invoke
                                      print.clj:21 nrepl.middleware.print/pr-on
                                      print.clj:17 nrepl.middleware.print/pr-on
                                     print.clj:148 nrepl.middleware.print/send-nonstreamed[fn]
                                     print.clj:147 nrepl.middleware.print/send-nonstreamed[fn]
                                     core.clj:2742 clojure.core/map[fn]
                                  protocols.clj:49 clojure.core.protocols/iter-reduce
                                  protocols.clj:75 clojure.core.protocols/fn
                                  protocols.clj:75 clojure.core.protocols/fn
                                  protocols.clj:13 clojure.core.protocols/fn[fn]
                                     core.clj:6884 clojure.core/transduce
                                     core.clj:6870 clojure.core/transduce
                                     print.clj:156 nrepl.middleware.print/send-nonstreamed
                                     print.clj:138 nrepl.middleware.print/send-nonstreamed
                                     print.clj:174 nrepl.middleware.print/printing-transport[fn]
                                     caught.clj:58 nrepl.middleware.caught/caught-transport[fn]
                        interruptible_eval.clj:114 nrepl.middleware.interruptible-eval/evaluate[fn]
=> nil

GGfpc12:12:51

I have no idea how to proceed from here because I'm using a callback function to process the request so I think I'm correct to assume that the response has been received when the function is called (which is the only problem I could think of). Previously I was using clj-http and it was working fine, but I switched to http-kit for better performance and most of the requests work correctly but some fail with this error

lassemaatta12:12:30

I believe http/get returns a promise, which you must deref somewhere to get the actual returned value

lassemaatta12:12:46

the Don't know how to create ISeq from: org.httpkit.client$deadlock_guard$reify__17177 error refers to the reified deadlock-guard promise, which you haven't dereffed

GGfpc12:12:18

But that's when you don't give it a callback function right? When I try to deref the body parameter of the callback I get

java.lang.ClassCastException: java.lang.String cannot be cast to java.util.concurrent.Future

GGfpc12:12:39

both with @body and deref body

lassemaatta12:12:46

if I read the httpkit code and docstring correctly, it always returns a promise but depending on whether you gave a callback it will resolve to either the response as-is or pass the response to your callback and resolve the return value

GGfpc12:12:48

I think I know the problem. Since this is asynchronous, I think it's returning immediately to the caller code, which does some processing with the expected response causing it to fail

roelof12:12:55

Can anyne tell me how I can make it work that the repl is next to my code instead of the same part so I do not have to switch tabs

Timur Latypoff13:12:36

I guess if you’re asking about VSCode and Calva, better to ask in #calva

roelof14:12:25

when I try this :

(parse-string (slurp "2008.json"))
I get a file not found error where im sure the 2008.json file is in the same src directory as the chapter7,clj file

andy.fingerhut14:12:04

Is the chapter7.clj source file in a directory like src/some-name-space/chapter7.clj in a Leiningen project, or something like that? If so, the current directory of the running JVM process is likely to be the top level directory of that project when you run the code.

andy.fingerhut14:12:32

If you print out the value of this expression, which should return a string, it should show you the current directory of the running JVM process, or at least what it was when the JVM process began: (System/getProperty "user.dir")

roelof14:12:43

yep, it was the root dir of the projecy

"/home/roelof/ground-up/ground-up"

yubrshen15:12:41

Seeking advice on selection of full stack suits web app of ToDo: I'd like to develop an app of todo. It should support web browser as interface, and eventually maybe native mobile app. I hope the user experience should be better than typical web browser experience, For example, I'd like to support extensive auto-completion of user input, allow context menu by clicking the right mouse button, and have timer etc. For data storage, I guess a SQL database would be enough, as the storage would be mostly in table and sequential, Datomic would be an overkill. It should also support user authentication. Would you kindly advise me which full stack suits I should take? I've studied using ring/hiccup/compojure, rum, jdbc.next, datomic for database web app. I have not tried Coast framework, Fulcro, Luminus, Pedestal, etc. With so many seemingly good choices, I feel anxious. This is a hobby/prototype effort, for MVP (Minimal Viable Product). Thanks a lot for your help!

roelof15:12:50

Wonder why the output is not sorted on the numbers

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :driving_under_influence)
       (take-last 10)
       (map (fn [county]
              [(fips (fips-code county))
               (:driving_under_influence county)]))
       (into {})))

andy.fingerhut15:12:42

In the last step, you put the data into a Clojure map (into {}). Clojure maps are not sorted by keys.

andy.fingerhut15:12:55

There is a sorted map variant of Clojure maps, which are sorted by keys. If that is something you want to try, you could change the last line to (into (sorted-map)) (there is no literal representation of sorted maps that is like {} for the default (unsorted) maps).

roelof15:12:25

still not ordered

roelof15:12:30

{"AZ, Maricopa" 26235,
 "CA, Los Angeles" 45056,
 "CA, Orange" 17572,
 "CA, Riverside" 10814,
 "CA, Sacramento" 8589,
 "CA, San Bernardino" 13983,
 "CA, San Diego" 18562,
 "NV, Clark" 10443,
 "TX, Harris" 10432,
 "WA, King" 11439}

pavlosmelissinos16:12:50

If you don't need constant time lookup, you might want to consider representing your data as a list of maps instead of a map:

[{:county "AZ, Maricopa" :dui 26235}
 {:county "CA, Los Angeles" :dui 45056}
 ...]
That way you don't have to treat maps (associative arrays) as something they're not and as an added bonus you get much prettier output out of the box using print-table! https://clojuredocs.org/clojure.pprint/print-table

roelof16:12:25

then I need to convert my data to it because its from a external source

roelof16:12:11

or do you mean I can change something here to make this output :

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :driving_under_influence)
       (take-last 10)
       (map (fn [county]
              [(fips (fips-code county))
               (:driving_under_influence county)]))
       (into{})
       (sort-by val)))

pavlosmelissinos16:12:15

yup, I'm talking about this:

(fn [county]
              [(fips (fips-code county))
               (:driving_under_influence county)])

roelof16:12:31

oke, then I have to think how to make those changes

👍 3
roelof17:12:06

hmm, I take a break this idea does not work

roelof17:12:10

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :driving_under_influence)
       (take-last 10)
       (map (fn [county]
              [{:county (fips (fips-code county)), :population (:driving_under_influence county)}]))
       (into {})))

roelof17:12:32

Got it working

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :driving_under_influence)
       (take-last 10)
       (map (fn [county]
              [{:county (fips (fips-code county))}, {:population (:driving_under_influence county)} ]))
       (into {})))

roelof17:12:42

hopefully i did it right

pavlosmelissinos17:12:02

you can simplify it further

pavlosmelissinos17:12:19

also this still creates a map

pavlosmelissinos18:12:39

Try this out:

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :driving_under_influence)
       (take-last 10)
       (map (fn [county]
              {:county     (fips (fips-code county)),
               :population (:driving_under_influence county)}))))

roelof18:12:20

that gives this :

({:county "CA, Sacramento", :population 8589}
 {:county "TX, Harris", :population 10432}
 {:county "NV, Clark", :population 10443}
 {:county "CA, Riverside", :population 10814}
 {:county "WA, King", :population 11439}
 {:county "CA, San Bernardino", :population 13983}
 {:county "CA, Orange", :population 17572}
 {:county "CA, San Diego", :population 18562}
 {:county "AZ, Maricopa", :population 26235}
 {:county "CA, Los Angeles", :population 45056})

roelof18:12:54

where my old code gave this :

{{:county "AZ, Maricopa"} {:population 26235},
 {:county "CA, Los Angeles"} {:population 45056},
 {:county "CA, San Diego"} {:population 18562},
 {:county "CA, Sacramento"} {:population 8589},
 {:county "CA, Riverside"} {:population 10814},
 {:county "CA, Orange"} {:population 17572},
 {:county "CA, San Bernardino"} {:population 13983},
 {:county "NV, Clark"} {:population 10443},
 {:county "TX, Harris"} {:population 10432},
 {:county "WA, King"} {:population 11439}}

pavlosmelissinos18:12:11

yup, notice the parens instead of the curly brackets in the beginning/end?

roelof18:12:30

then tomorrow look if I can make it work to calculate the prevalance so this number divided by the population

roelof18:12:45

as I see it now I have to make the calculation two times

roelof18:12:03

hmm, this eems to do it almost

roelof18:12:46

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :driving_under_influence)
       (take-last 10)
       (map (fn [county]
              {:county     (fips (fips-code county)),
               :total_uid (:driving_under_influence county)
               :prevalance ( / (:driving_under_influence county) (:county_population county))
               }))))
only the prevalence is not calculated:
{:county "CA, Los Angeles",
  :total_uid 45056,
  :prevalance 45056/9872263}

pavlosmelissinos18:12:08

it is the correct form but you don't see the calculated value, you can run float on top of the division to get the actual result

roelof18:12:42

next "problem"

roelof18:12:04

it now sorts on drunk driving (sort-by :driving_under_influence)

roelof18:12:27

I think I need to make a new key prevelance to see the 10 most highest ones

pavlosmelissinos18:12:05

how many data points are there in the file?

roelof18:12:43

183 thousand

roelof18:12:14

he, I can do this :

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (sort-by :prevalance)
       (take-last 10)
       (map (fn [county]
              {:county     (fips (fips-code county)),
               :total_uid (:driving_under_influence county)
               :prevalance (float (/ (:driving_under_influence county) (:county_population county)))
               }))))

roelof18:12:41

but I have to make a check against the fact that population can be zero

👌 3
pavlosmelissinos18:12:55

that key doesn't exist yet when you sort

pavlosmelissinos18:12:20

you could either compute it twice or move sort-by and take-last to the end, after the map

roelof19:12:47

last question

roelof19:12:58

I have now this :

(defn calculate_prevalance
  [county]
  ( if (zero? (:county_population county))
         0
    (float (/ (:driving_under_influence county) (:county_population county)))))
    

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (map (fn [county]
              {:county     (fips (fips-code county)),
               :total_uid (:driving_under_influence county)
               :prevalance (calculate_prevalance county)}))
       (sort-by :prevalance)
       (take 10)))

roelof19:12:18

but the highest 10 are now all countys with prevelance of 0

roelof19:12:37

Should I filter these out or is there a better way

pavlosmelissinos19:12:43

sort-by sorts the data in ascending order

pavlosmelissinos19:12:21

so take 10 takes the 10 items with the lowest prevalence

roelof19:12:22

yep, so I get the countys with the lowest prevelance

pavlosmelissinos19:12:23

Also, you might want to turn the fn into an actual function

roelof19:12:28

this seems not to be sorted on the highest prevelance

roelof19:12:34

|       :county | :total_uid | :prevalance | |---------------+------------+-------------| |      NC, Hyde |        125 | 0.024447488 | |      CA, Inyo |        424 | 0.024500173 | |      NC, Ashe |        655 | 0.025449742 | |  CO, Costilla |         85 | 0.026001835 | |   CO, Conejos |        210 | 0.026122652 | |  CO, Cheyenne |         45 | 0.026254376 | |    TX, Kenedy |         11 | 0.028132992 | |    VA, Norton |        118 | 0.031969655 | |    MS, Tunica |        432 |  0.04056338 | | WI, Menominee |        189 | 0.040935673 |

roelof19:12:59

`(defn calculate_prevalance
  [county]
  ( if (zero? (:county_population county))
         0
    (float (/ (:driving_under_influence county) (:county_population county)))))
    

(defn most-duis
  "Given a JSON filename of UCR crime data for a particular year, finds the
  counties with the most DUIs."
  [file]
  (->> file
       load-json
       (map (fn [county]
              {:county     (fips (fips-code county)),
               :total_uid (:driving_under_influence county)
               :prevalance (calculate_prevalance county)}))
       (sort-by :prevalance)
       (take-last 10)))

pavlosmelissinos19:12:20

it's sorted, in ascending order

pavlosmelissinos19:12:33

you're just seeing the last 10 counties in the list

pavlosmelissinos19:12:00

if you want to see it in descending order you need to reverse the result

roelof19:12:06

yep, and I assumed that were the 10 county's with the highest prevelance

roelof19:12:58

yep, and then I see the 10 with prevelance zero because there is no population

roelof19:12:08

so I set the return on zero

roelof19:12:25

so I think I have to filter those out. Right ?

pavlosmelissinos19:12:36

no need to, just add reverse in the end

pavlosmelissinos19:12:50

the result is correct, it's just in ascending order

pavlosmelissinos19:12:40

you have the top 10 counties, starting from the 10th

pavlosmelissinos19:12:53

so if you reverse it you'll get the top county first

roelof19:12:35

yep,, this looks good

|       :county | :total_uid | :prevalance |
|---------------+------------+-------------|
| WI, Menominee |        189 | 0.040935673 |
|    MS, Tunica |        432 |  0.04056338 |
|    VA, Norton |        118 | 0.031969655 |
|    TX, Kenedy |         11 | 0.028132992 |
|  CO, Cheyenne |         45 | 0.026254376 |
|   CO, Conejos |        210 | 0.026122652 |
|  CO, Costilla |         85 | 0.026001835 |
|      NC, Ashe |        655 | 0.025449742 |
|      CA, Inyo |        424 | 0.024500173 |
|      NC, Hyde |        125 | 0.024447488 |

👌 3
roelof19:12:03

next challenge from the "Clojure from the ground up"

|       :county | :prevalance | :report-count | :population |
|---------------+-------------+---------------+-------------|
| WI, Menominee | 0.040935673 |           189 |        4617 |
|    MS, Tunica |  0.04056338 |           432 |       10650 |
|    VA, Norton | 0.031969655 |           118 |        3691 |
|    TX, Kenedy | 0.028132992 |            11 |         391 |
|  CO, Cheyenne | 0.026254376 |            45 |        1714 |
|   CO, Conejos | 0.026122652 |           210 |        8039 |
|  CO, Costilla | 0.026001835 |            85 |        3269 |
|      NC, Ashe | 0.025449742 |           655 |       25737 |
|      CA, Inyo | 0.024500173 |           424 |       17306 |
|      NC, Hyde | 0.024447488 |           125 |        5113 |

roelof19:12:57

hmm, I think the last challenge is nonsens

Write a test to verify that most-prevalent is correct.

roelof19:12:19

how can I make a test if im not knowing the numbers myself

pavlosmelissinos19:12:59

Make them up. The logic should be correct no matter what the actual values of the numbers are.

pavlosmelissinos19:12:14

correctness != truth

roelof19:12:38

oke, I have to think about it

roelof19:12:44

so I should make a test file with only 1 county with some duis and some population

roelof19:12:52

and use that for the test

roelof15:12:35

` (defn most-duis   "Given a JSON filename of UCR crime data for a particular year, finds the   counties with the most DUIs."   [file]   (->> file        load-json        (sort-by :driving_under_influence)        (take-last 10)        (map (fn [county]               [(fips (fips-code county))                (:driving_under_influence county)]))        (into (sorted-map)))) `

delaguardo15:12:47

sorted-map order keys by hash function, not alphabeticaly this is wrong ) my bad

andy.fingerhut15:12:01

@roelof The result you showed last above is sorted alphabetically by the keys. What do you want it sorted by?

andy.fingerhut15:12:06

Oh, I see now you said earlier "on the numbers". One recommendation: write a little print function to print the data in the order you want to see it in, but let it be stored in a default Clojure map, which is not sorted in any particular order (well, it is sorted on a hash function of the keys, but that is unlikely to be an order you are interested in knowing).

andy.fingerhut16:12:28

Example:

(def d1 {"AZ, Maricopa" 26235,
         "CA, Los Angeles" 45056,
         "CA, Orange" 17572,
         "CA, Riverside" 10814,
         "CA, Sacramento" 8589,
         "CA, San Bernardino" 13983,
         "CA, San Diego" 18562,
         "NV, Clark" 10443,
         "TX, Harris" 10432,
         "WA, King" 11439})

user=> (pprint (sort-by val d1))
(["CA, Sacramento" 8589]
 ["TX, Harris" 10432]
 ["NV, Clark" 10443]
 ["CA, Riverside" 10814]
 ["WA, King" 11439]
 ["CA, San Bernardino" 13983]
 ["CA, Orange" 17572]
 ["CA, San Diego" 18562]
 ["AZ, Maricopa" 26235]
 ["CA, Los Angeles" 45056])

andy.fingerhut16:12:54

I know that last print result does not look like a map. That is because it is a sequence of map entries, which is what you get when you take a map and use it as a parameter to a function that expects a sequence, like sort-by does in the expression above. If all you care about is seeing it in the desired order, that shouldn't be a problem.

Andrew Byala16:12:00

I have a pattern type of question for designing APIs. I come from a Java background where there are a lot of pure and impure functions mixed together, especially when reading and writing from a database. I've read Clojure guidance that says to write as much of the logic in pure functions as possible, and then use some sort of wiring layer to mix together these pure functions. It would seem that once you get past simple CRUD APIs, the coordination functions would become both very large and could potentially have a lot of duplication. Can anyone point me to either examples or tutorials that show how this works as the codebase grows?

phronmophobic19:12:42

This is a great question! I'll do my best to answer and hopefully my attempt will nerd snipe someone smarter into writing a better answer. > It would seem that once you get past simple CRUD APIs, the coordination functions would become both very large and could potentially have a lot of duplication. I think the key idea is that if you use simple constructs and generic data, the coordinating code doesn't grow at all as your code base grows. > Computer science offers a standard way to handle complexity:hierarchical structure. > - Leslie Lamport https://lamport.azurewebsites.net/pubs/proof.pdf Typically, the coordinating code relies on a tree-like structure for organizing all of the code/functionality needed for an application. Traversing the tree-like structure doesn't take very much code and it's the same amount of code regardless of the size of the tree. Trees also have the benefit that searching a large, organized tree is very efficient. An example of coordinating code that uses this principle is https://github.com/ring-clojure/ring/wiki. Returning a response to an HTTP request is done by composing pure functions. The coordinating code is the same size for both small and large web sites. Typically, routes are organized hierarchically and hopefully don't have too many interactions between separate branches of the routing tree. I can't think of any tutorials, but the Rich Hickey talks are all very good. On this topic, I think https://www.youtube.com/watch?v=ROor6_NGIWU and https://www.youtube.com/watch?v=drmNlZVkUeE might be good talks to start with.

seancorfield19:12:55

It can be hard to completely separate out database interaction from pure logic because you typically have a lot of situations where the pure logic at some point may well rely on DB activity from earlier in the request (assuming a complex DB-backed web app). The pragmatic question is: how separate does it really need to be? You definitely want to aim for separation and composability but there are practical limits to what you can realistically achieve in general, in my opinion, without resorting to somewhat artificial code structure and monadic-looking code.

clyfe19:12:10

https://www.youtube.com/watch?v=yTkzNHF6rMs https://www.destroyallsoftware.com/screencasts/catalog/functional-core-imperative-shell The 2 vids may help somewhat (functional core, imperative shell). Also, personal anecdote: afaict functional architectures tend to converge into middleware pattern (ring's wrap-*, re-frame interceptors, stuff I seen in enterprises).

seancorfield19:12:13

Years ago, I built a generic "workflow" engine library for Clojure that separated everything into "sources", pure functions, and "sinks". The idea was that sources could (only) be queried -- the sources were passed into the pure code, so you could write mock sources for testing -- and that the pure functions would return data structures that indicated what actions would need to be taken on the sinks, which could be database writes (inserts and updates) and email systems and anything else you needed to apply changes or effects on. As soon as a later part of your pure workflow needs the generated ID from an earlier DB insert, things get complex, and it also gets a bit artificial when the DB inserts you need "at the end" have to include stuff that you got from side-effecting calls to third-party systems. It's doable, but it gets pretty convoluted. And I finally gave up on the engine after using it for a while in production code because it was just too awkward (and monadic) to be readable once the actual workflow got complex. It's still on GH as an archive https://github.com/seancorfield/engine with an explanation of why I stopped using it.

seancorfield19:12:39

Having an imperative shell drive everything is a much more straightforward approach but it can still take quite a bit of work to completely separate out pure and impure logic -- and if you have an imperative shell, you are still going to need some amount of tests to verify that the imperative combining of the pure/impure calls really does what you want. Definitely more maintainable than just arbitrarily mixing pure and impure code together tho' -- and more maintainable than going "full-monad" IMO.

phronmophobic19:12:20

I think it's interesting that many of these functional ideas can even be applied to games, https://fabiensanglard.net/quake3/index.php

Andrew Byala00:12:38

Thanks for the information and the links. I'm working my way through a course on building APIs using Reitit, and once I get past whatever's blocking me right now, I'll try to build a sample API with this pattern in mind.

Bill St. Clair16:12:03

Old lisp expert here. New to Clojure. Writing a Swank client in ClojureScript (replacement for Emacs SLIME, for the browser). Once that’s done, I’ll likely add a cider client as well. This is a retirement project for me. I don’t have the mental energy I once did, so the Clojure startup transient is harder than it would have been 10 years ago, but I’ll get there. Clojure is, after all, just another lisp with some added syntax and a new standard library to learn (he said, not really knowing what he’s saying). https://github.com/billstclair/wilfred Not much here yet. If somebody has ideas for improvement of the basic project, I’d love the feedback. I’m just copying what I found on the web at this point. I think it will be good enough to get started, however.

👍 6
Klavs Klavsen17:12:31

I'm trying to grasp clojure here - playing with stumps of code. How do I loop through a list of maps and rewrite the maps (in other languages filter can be used for this):

(def mylist
  ({:a 1 :b 2 :c 3}
   {:a 1 :b 5 :c 6}
   {:a 7 :b 8 :c 12}
   {:a 10 :b 11 :c 12}))
and I want to modify the key 'b' so it contains the contents of key 'c'

mathias_dw17:12:47

(map (fn [m] (assoc m :b (m c)) mylist)

euccastro17:12:21

with a colon before the c

Klavs Klavsen17:12:21

Thanks. Trying that out however, Calva repl (visual studio code) just gives me:

println mylist #prints out mylist in repl window
(map (fn [m] (assoc m :b (m :c)) mylist)) #just prints out: #function[clojure.core/map/fn--5847]
😞

Klavs Klavsen17:12:53

so println mylist works.. map just fails.. I'm doing ctrl+shift+p and evaluate top form

andy.fingerhut17:12:15

It appears that it evaluated just the sub-expression map, which I would expect to return a result like you showed.

Michael W17:12:41

Parens are wrong, mylist should be in the last form, not inside the fn definition.

Michael W17:12:51

(map #(assoc % :b (% :c)) mylist)

Klavs Klavsen17:12:26

this worked:

println (map (fn [m] (assoc m :b (m :c))) mylist)
Thank you

Michael W17:12:40

It's like you ran map without a collection, which returns a transducer function.

Klavs Klavsen17:12:43

Now I have something to play with.. trying to get the hang of this 🙂

andy.fingerhut17:12:50

FYI the println at the beginning of those lines is probably not even being evaluated at all. You should be able to leave that out.

andy.fingerhut17:12:22

Unless the REPL you are using is quite different than ones I have used before somehow ...

Klavs Klavsen18:12:31

you're right.. now it outputs it without the prinln @andy.fingerhut - odd it didn't at first

andy.fingerhut18:12:46

I would be very surprised if it actually required that -- the P in REPL is for Print, so after reading and evaluating your expression, it should always Print the return value (in addition to printing anything that the expression itself causes to print, e.g. a function call with calls to functions print and/or println inside of it).

👍 3
phronmophobic19:12:42

This is a great question! I'll do my best to answer and hopefully my attempt will nerd snipe someone smarter into writing a better answer. > It would seem that once you get past simple CRUD APIs, the coordination functions would become both very large and could potentially have a lot of duplication. I think the key idea is that if you use simple constructs and generic data, the coordinating code doesn't grow at all as your code base grows. > Computer science offers a standard way to handle complexity:hierarchical structure. > - Leslie Lamport https://lamport.azurewebsites.net/pubs/proof.pdf Typically, the coordinating code relies on a tree-like structure for organizing all of the code/functionality needed for an application. Traversing the tree-like structure doesn't take very much code and it's the same amount of code regardless of the size of the tree. Trees also have the benefit that searching a large, organized tree is very efficient. An example of coordinating code that uses this principle is https://github.com/ring-clojure/ring/wiki. Returning a response to an HTTP request is done by composing pure functions. The coordinating code is the same size for both small and large web sites. Typically, routes are organized hierarchically and hopefully don't have too many interactions between separate branches of the routing tree. I can't think of any tutorials, but the Rich Hickey talks are all very good. On this topic, I think https://www.youtube.com/watch?v=ROor6_NGIWU and https://www.youtube.com/watch?v=drmNlZVkUeE might be good talks to start with.

Christian21:12:11

I want to do some natural language processing (summarizing) and DL (one-shot-learning). I found this: https://dragan.rocks/articles/19/Deep-Learning-in-Clojure-From-Scratch-to-GPU-0-Why-Bother and Dragan recommends to use intel-mkl. He states it's 750 mb, but the only download I can find is a whooping 22gb. Are there other libraries I should use, or is this really the way to go?

dorab21:12:48

It is best if you install via your package manager. Failing that, install via the Intel installer. I don't believe either of those are anywhere close to 22GB. The instructions at https://neanderthal.uncomplicate.org/articles/getting_started.html#the-native-library-used-by-neanderthals-native-engine-optional should be helpful. Which OS are you on?

dorab02:12:30

Yikes! I just checked and on my MacOS machine, /opt/intel (where MKL is installed) is around 3GB. Perhaps 22GB is required during installation but not after the installation is complete?

caumond21:12:10

Hi everybody, maybe this is a dumb question, but, I don't feel confortable with my solution up to now. I work with a defmulti, defined in a ns, and some defmethod, provided in their respective namespace. Now, when I want to call the method, I fall in situations where de defmethod are not found, until I explicitly require the namespace containing the defmethod. I would prefer a behavior where the calling function needs not to know its implementation, as a plugin is expected to act.

Sam Ritchie00:12:50

One thing you can do is expose a namespace that requires all of the defmethod-containing namespaces and make it so users require that

Sam Ritchie00:12:44

Right now it sounds like they are requiring the namespace with the defmulti- maybe stick the defmulti somewhere else, then in this top level NS require all of the namespaces that extend the defmulti

phronmophobic01:12:44

it seems like there is an implicit dependency between the code calling the multi method and the namespace implementing some corresponding defmethod. it seems like the "right" answer is to make the implicit dependency an explicit dependency (ie. make it so there's no way to call the the multi-method with args you expect it to handle without requiring the namespace that provides the implementation for those args). it's hard to offer more specific advice without knowing a little more about the use case.

Sam Ritchie01:12:09

@U3ES97LAC this will have the same problem, I believe

caumond08:12:08

Hi, @U3ES97LAC I don't see the breakthrough with protocols, maybe I missed something. @U7RJTCH6J, I guess this is my best option, I'm not sure how to do it. I will provide a simple example just to share my issue an better see the solution

caumond09:12:31

What I am not happy with are the near useless require of english and french namespace in the core.ns

caumond09:12:44

I finally found this solution:

caumond09:12:26

Starts to make sense to me. Your opinion?

phronmophobic19:12:15

looks good to me. I think it's reasonable to expect someone that uses greeting to load the namespaces it needs to fill in which ever languages they expect

phronmophobic19:12:33

you can also override the default implementation of greeting to throw a more helpful exception

phronmophobic19:12:02

maybe an exception that says something like (str "Did you forget to load the plugin for language " x "?")

phronmophobic20:12:23

reading through Clojure Applied and it mentions something similar:

(defmethod convert :default [u1 u2 q]
  (if (= u1 u2)
    q
    (assert false (str "Unknownunitconversionfrom "u1" to "u2))))

caumond21:12:39

Thx @U7RJTCH6J, yes I put already this :default option in my full version.

👍 3
sova-soars-the-sora03:12:01

Glad you found a solution.

Ihor Horobets23:12:38

Hi everyone! Does anybody use hickory library? https://github.com/davidsantiago/hickory