Fork me on GitHub
#clojure-europe
<
2019-04-03
>
kardan06:04:31

Good morning

thomas07:04:30

morning friends and @jasonbell 😉

ordnungswidrig07:04:58

Aloha! :beach_with_umbrella:

ben.hammond08:04:47

another delightful morning inside Europe

thomas09:04:18

so.... clojure question.... for a change 😉

thomas09:04:27

I have a lazy function:

thomas09:04:10

and it read in a csv file (which can be big hence lazy) and then does an insert in de DB...

thomas09:04:27

but how do I force it to be realised?

thomas09:04:59

when I call that function in a doall I get closed stream exception

ben.hammond09:04:04

use something eager like a loop or a reduce

ben.hammond09:04:56

with-open and lazy sequences are contradictory; there's no getting away from it

ben.hammond09:04:46

you can use IReduceInit to segment your diffferent functionality. Something like

(defn csv-maps
  "generic low overhead way to turn (massive) csv file
   into lazily evaluated stream of maps"
  [ff & opts]
  (reify clojure.lang.IReduceInit
    (reduce [_ f init]
      (with-open [^BufferedReader r (jio/reader ff)]
        (let [hdr (first (apply csv/read-csv (.readLine r) opts))]
          (transduce (map (partial zipmap hdr))
                     (completing f)
                     init
                     (apply csv/read-csv r opts)))))))

ben.hammond09:04:16

you can have 'with-open` inside the reducible

ben.hammond09:04:58

and then when you call it from reduce or into everything is kept nice and eager

ben.hammond10:04:49

either that or you have to get into specific lifecycle state function; which is less fun than it sounds

ben.hammond10:04:21

so you end up writing your loop as something like

(transduce (map update-data) (completing (fn [acc v) (insert-data v)) nil (csv-maps file-name))

ben.hammond10:04:19

(map-indexed can work as a transducer, so you can poke that in there too

thomas10:04:56

ok thanks @ben.hammond I'll look into it.

thomas11:04:55

hmmm I thought I memoize my DB call.. but when I do that it doesn't quite work 😞

thomas11:04:27

but when I call it from the REPL it all works fine... it only fails when running from an uberjar (and straight away, not after a long time with an OOM error)

thomas11:04:00

hmm maybe not..

thomas11:04:24

moar debugging needed... my idea doesn't quite work...

thomas13:04:11

got it working now... and it goes a lot faster 🐎 now... 🎉 so pleased about that.

thomas13:04:41

had to put the memoize in the correct place of course... 😇

ben.hammond13:04:05

my problem with memoize is that I never want to store everything for all time

ben.hammond13:04:24

but you cannot give it an eviction policy

thomas14:04:30

there is core.cache (?) for that I think

thomas14:04:56

in this case I don't know what to cache... but a LRU would be ok in my case.

ben.hammond14:04:24

oooh I was not aware of that