Morning

moin moin

How are things in the Netherlands, @thomas ?

fine... and very misty at the moment @yogidevbear

ooh and no government yet... but that is ok.

morning

I get the feeling that no government can be a good thing

Good morn.

morning

mornin'

less gov = better gov, for the current given value of gov

keynes would have something to say about that 😉

Morning all.

morning!

Morning All… 🙂

Anyone got 5 minutes to help me with something that I should know better about..? I am trying to consume a 2.5Mn row CSV file and the approach I have been using for sub 250K row files is no longer cutting the mustard. I have a sense that two thing I don’t really grok, laziness and streaming, are going to help me here, but I am not really sure where to start… 😞 (I am using clojure.data.csv)

Current ingest looks like this:

(defn ingest-csv-data
  "Read in a CSV file from the filesystem based on a configured path :external-files-path"
  [csv-resource]
  (with-open [in-file (io/reader (io/file (str (:external-files-path ingest-cfg) csv-resource)))]
    (doall
     (csv/read-csv in-file))))

I am assuming that part of the problem is that doseq is not going to work with the laziness, that in order to doseq the collection (which is a lazy sequence of vectors) it will need to realise the whole collection first..? TLDR; I am running out of memory.

well

from my csv reading experience from before

take it with a pinch of salt though

i thought when you do with-open, you load the whole file into memory first?

so you will have memory problems at that point regardless?

but like i said i could be totally wrong

from my reading of doall that's going to force the sequence to be realised (i.e. the whole CSV file to be read in)

@maleghast you can use either laziness or streaming to help you with the OOM problem - have a look at the start of this for our lazy-tsv parser - https://gist.github.com/5207b5069bac7188fbb6dfce2d38c490

bingo, doall is the problem

so the in-file, is that just not evaluated then?

i dont really know much about readers to be fair

ill go do some googling

readers are a file-handle, maybe a buffer and some character interpretation atop the raw input-stream - they don't read the whole file

got ya thanks craig :thumbsup:

Thanks @mccraigmccraig @dominicm @guy and @iain.tatch

You were missing an i

@iain.tatch

I will take a look at @mccraigmccraig’s link-y / gist and let you know if I understand anything of it…

Thanks @dominicm 🙂

it has some custom line-end parsing behaviour in there @maleghast , which is why we aren't just using clojure.core/line-seq iirc, but it's illustrative of the mechanics of using lazy-seqs for processing large files

@mccraigmccraig - Yeah, I noticed the custom line-ending stuff…

I am reading through it now…

@mccraigmccraig - My reading of clojure.data.csv is that it already provides a “lazy” approach, i.e. that I don’t need__ to take the steps that you’ve taken here to create lazy seqs of chars, lines etc. to build the lazy collection of rows. I guess my question is more that if I want it to behave the way I think it should, do I have to do all the “work” inside the function that is “creating” the lazy collection of rows, in order to benefit from the laziness?

no @maleghast - just map / reduce over the seq that you get back from clojure.data.csv or whatever

it's the doall in your code above which was causing the whole file to be read into memory

without the doall the file would close before reading the seq

doing the work inside the lazy bit is a good way to think about it imo

yes.

Right, that’s kind of what I was asking - just doing it badly… If I did this:

(defn format-row
[row]
;;Do some data cleaning, data casting etc. here)

(defn insert-row
[row]
(sql/insert {:? row}, (:conn db)))

(defn csv-to-db [from]
  (with-open [reader (io/reader from)]
    (->> (read-csv reader)
         (map #(format-row) %)
         (map #(insert-row %))))))

would that work..?

could use a pmap for the insert-row bit 🙂 - but this ^ would be lazy and eval would need to be forced

Ok, and this is where the whole lazy thing hurts my brain - how do I enforce the eval?

i’d have csv-to-db be something more like with-csv-row and pass in a function which does the stuff

i.e. how do I get the actual__ data “out” on a row by row basis?

@glenjamin - *whoosh*

@maleghast reduce is good

as is doseq

@mccraigmccraig - well yes, I like both reduce and doseq, but I feel you are suggesting that they have powers I am unaware of, that I nonetheless believe that they do have… i.e. I need to level up a bit, don’t I?

😞

(defn with-csv-row [from action]
  (with-open [reader (io/reader from)]
    (doseq [row (read-csv reader)]
      (action row)))))

something like that

@glenjamin - Where action is a function that cleans up the row and runs the query?

but with balanced parens and correct function usage - i just typed that in slack

yes

so doseq only realises one row at a time from the lazy sequence of vectors that read-csv emits..?

Also, there’s nothing to stop me passing in the row and 2 functions, right, so that I can have a separate function for each of data clean / prep and db insert, right?

sounds to me like a good fit for transduce

(more for helping my brain not hurt than anything else, though I like to think__ I would keep side-effects to their own function(s))

@ben.hammond - Oh no, you’ve just invoked my third, super-secret achilles heel of Clojure-Dev-Imposter syndrome… Transduce / Transducers…

*WHOOSH*

@maleghast you could compose a bunch of functions defined independently to create action, then the csv function doesn’t need to know about the distinction

@glenjamin Good point, ok

🙂

and it should be straightforward to write reduce or transduce based variants of with-csv-row with the same signature

@glenjamin - That is only true if you’ve grok-ed transduce

(I imagine)

yeah, my point being “do this on each line of csv” is a decent interface to code to

i wrote a thing to import massive CSVs recently, I used semantic-csv’s transducer functions to process the rows

I think that I could write it with reduce instead, but I tend to think “reduce means reduction” and this use-case does not take one collection and produce a smaller one…

I realise that this is a simplistic view, but it helps me to keep my decision-making consistent, at the moment, while I level up from keen amateur to coding in clojure every day

if doseq will get the job done, in the parameters I need, I will stick to that for now and then get cleverer later… 🙂

@maleghast reduce is a swiss army knife. you can implement map and filter purely with reduce, for example. a better name for it would be accumulate imo

https://youtu.be/WkHdqg_DBBs

I found this to be an extremely helpful explanation

@sundarj - I know what you mean, I really do, I just find it easier, at the moment, at my level of grok, to keep reduce in the box that gets opened when I want to do something to a collection where the result is smaller than in size or different to in structure, or both from the original collection

@ben.hammond - thx, I will go look now…

@ben.hammond - It is a good video, thanks. I think I am going to have to watch it again, and possibly some more times, but more of the idea(s) make sense now than they did before, so thanks 🙂

I'd recommend all his videos. his pay channel represents pretty good value

since you can consume the whole thing in a couple of months

@ben.hammond - I will consider it, but I don’t watch videos when at home, we have a metred, satellite connection and watching YouTube(etc) videos is not really an option. This is the price I pay for living in the most beautiful spot ever. I happen to be in an AirBnB in London right now__ so I watched it 🙂

youtube-dl ftw (if not drm)

@ben.hammond that transducer video was the first time I properly understood what was going on inside a transducer and why (I've been happily using them before that, just not properly understood how it worked internally)

Video bookmarked. something I do need to wrap my head around.

@maleghast I also found this really useful; it’s reasoned from first principles: https://labs.uswitch.com/transducers-from-the-ground-up-the-essence/

Thanks @chrisjd

I will have a “proper” look later on, but this looks very helpful.