Fork me on GitHub
#clojure-nl
<
2018-04-30
>
thomas07:04:11

good moaning

thomas07:04:18

:rain_cloud:

vijaykiran08:04:22

So it takes ~19mins to parse a 8G CSV file on my computer

vijaykiran08:04:01

Yup, otherwise it will blowup my memory , isn’t it 🙂 ?

thomas08:04:20

good point...

thomas08:04:30

that might well be the reason it is so slow...

thomas08:04:02

otherwise you could read it into memory in one go... and that is a lot faster compared to lots of small reads

lmergen09:04:15

in my experience parsing all the fields is actually taking up the most time. even lightning fast C based CSV parsers have CPU as the bottleneck

lmergen09:04:36

disk reads from CSV are usually sequential and thus very fast