Good morning

Morning

morning friends and @jasonbell 😉

Aloha! :beach_with_umbrella:

another delightful morning inside Europe

so.... clojure question.... for a change 😉

I have a lazy function:

and it read in a csv file (which can be big hence lazy) and then does an insert in de DB...

but how do I force it to be realised?

when I call that function in a doall I get closed stream exception

use something eager like a loop or a reduce

with-open and lazy sequences are contradictory; there's no getting away from it

you can use IReduceInit to segment your diffferent functionality. Something like

(defn csv-maps
  "generic low overhead way to turn (massive) csv file
   into lazily evaluated stream of maps"
  [ff & opts]
  (reify clojure.lang.IReduceInit
    (reduce [_ f init]
      (with-open [^BufferedReader r (jio/reader ff)]
        (let [hdr (first (apply csv/read-csv (.readLine r) opts))]
          (transduce (map (partial zipmap hdr))
                     (completing f)
                     init
                     (apply csv/read-csv r opts)))))))

you can have 'with-open` inside the reducible

and then when you call it from reduce or into everything is kept nice and eager

either that or you have to get into specific lifecycle state function; which is less fun than it sounds

so you end up writing your loop as something like

(transduce (map update-data) (completing (fn [acc v) (insert-data v)) nil (csv-maps file-name))

(map-indexed can work as a transducer, so you can poke that in there too

ok thanks @ben.hammond I'll look into it.

hmmm I thought I memoize my DB call.. but when I do that it doesn't quite work 😞

but when I call it from the REPL it all works fine... it only fails when running from an uberjar (and straight away, not after a long time with an OOM error)

hmm maybe not..

moar debugging needed... my idea doesn't quite work...

got it working now... and it goes a lot faster 🐎 now... 🎉 so pleased about that.

had to put the memoize in the correct place of course... 😇

my problem with memoize is that I never want to store everything for all time

but you cannot give it an eviction policy

there is core.cache (?) for that I think

in this case I don't know what to cache... but a LRU would be ok in my case.

oooh I was not aware of that

nice