Fork me on GitHub

Good morning


morning friends and @jasonbell 😉

👍 4

Aloha! :beach_with_umbrella:

Ben Hammond08:04:47

another delightful morning inside Europe


so.... clojure question.... for a change 😉


I have a lazy function:


and it read in a csv file (which can be big hence lazy) and then does an insert in de DB...


but how do I force it to be realised?


when I call that function in a doall I get closed stream exception

Ben Hammond09:04:04

use something eager like a loop or a reduce

Ben Hammond09:04:56

with-open and lazy sequences are contradictory; there's no getting away from it

Ben Hammond09:04:46

you can use IReduceInit to segment your diffferent functionality. Something like

(defn csv-maps
  "generic low overhead way to turn (massive) csv file
   into lazily evaluated stream of maps"
  [ff & opts]
  (reify clojure.lang.IReduceInit
    (reduce [_ f init]
      (with-open [^BufferedReader r (jio/reader ff)]
        (let [hdr (first (apply csv/read-csv (.readLine r) opts))]
          (transduce (map (partial zipmap hdr))
                     (completing f)
                     (apply csv/read-csv r opts)))))))

Ben Hammond09:04:16

you can have 'with-open` inside the reducible

Ben Hammond09:04:58

and then when you call it from reduce or into everything is kept nice and eager

Ben Hammond10:04:49

either that or you have to get into specific lifecycle state function; which is less fun than it sounds

Ben Hammond10:04:21

so you end up writing your loop as something like

(transduce (map update-data) (completing (fn [acc v) (insert-data v)) nil (csv-maps file-name))

Ben Hammond10:04:19

(map-indexed can work as a transducer, so you can poke that in there too


ok thanks @ben.hammond I'll look into it.


hmmm I thought I memoize my DB call.. but when I do that it doesn't quite work 😞


but when I call it from the REPL it all works fine... it only fails when running from an uberjar (and straight away, not after a long time with an OOM error)


hmm maybe not..


moar debugging needed... my idea doesn't quite work...


got it working now... and it goes a lot faster 🐎 now... 🎉 so pleased about that.


had to put the memoize in the correct place of course... 😇

Ben Hammond13:04:05

my problem with memoize is that I never want to store everything for all time

Ben Hammond13:04:24

but you cannot give it an eviction policy


there is core.cache (?) for that I think

👍 8

in this case I don't know what to cache... but a LRU would be ok in my case.

Ben Hammond14:04:24

oooh I was not aware of that