This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2019-04-03
Channels
- # announcements (11)
- # aws (1)
- # beginners (57)
- # calva (14)
- # cider (22)
- # clara (23)
- # cljdoc (21)
- # cljs-dev (14)
- # clojure (76)
- # clojure-austin (3)
- # clojure-europe (32)
- # clojure-finland (1)
- # clojure-gamedev (5)
- # clojure-italy (3)
- # clojure-nl (17)
- # clojure-poland (4)
- # clojure-spec (23)
- # clojure-uk (29)
- # clojurescript (72)
- # cursive (6)
- # data-science (4)
- # datomic (2)
- # dirac (10)
- # duct (3)
- # emacs (1)
- # figwheel-main (10)
- # fulcro (86)
- # jackdaw (5)
- # juxt (2)
- # kaocha (2)
- # leiningen (16)
- # off-topic (47)
- # pathom (46)
- # protorepl (2)
- # reagent (4)
- # reitit (14)
- # ring-swagger (5)
- # shadow-cljs (43)
- # spacemacs (13)
- # sql (13)
- # tools-deps (6)
Aloha! :beach_with_umbrella:
another delightful morning inside Europe
and it read in a csv file (which can be big hence lazy) and then does an insert in de DB...
use something eager like a loop or a reduce
with-open
and lazy sequences are contradictory; there's no getting away from it
you can use IReduceInit
to segment your diffferent functionality.
Something like
(defn csv-maps
"generic low overhead way to turn (massive) csv file
into lazily evaluated stream of maps"
[ff & opts]
(reify clojure.lang.IReduceInit
(reduce [_ f init]
(with-open [^BufferedReader r (jio/reader ff)]
(let [hdr (first (apply csv/read-csv (.readLine r) opts))]
(transduce (map (partial zipmap hdr))
(completing f)
init
(apply csv/read-csv r opts)))))))
you can have 'with-open` inside the reducible
and then when you call it from reduce
or into
everything is kept nice and eager
either that or you have to get into specific lifecycle state function; which is less fun than it sounds
so you end up writing your loop as something like
(transduce (map update-data) (completing (fn [acc v) (insert-data v)) nil (csv-maps file-name))
(map-indexed
can work as a transducer, so you can poke that in there too
ok thanks @ben.hammond I'll look into it.
but when I call it from the REPL it all works fine... it only fails when running from an uberjar (and straight away, not after a long time with an OOM error)
my problem with memoize
is that I never want to store everything for all time
but you cannot give it an eviction policy
oooh I was not aware of that