Why do you assume transduce holds everything in memory at once?

It's more or less like reduce

Eduction is built on top of transduce

maybe he means eager cos that's how transduce is advertised

yes, reduce is also eager. but that doesn't mean it will realize the entire input or hold everything in memory at once. Transducers know when to stop similar to how reduce knows to stop using a reduced value

I think reading the source might make more sense than speculate.

I mean that the result of the reduce will be held in memory all at once, whereas the eduction will only realise as much as has been asked for

so (take 10 (eduction (map identity) (range 100000)) would only realise the first 10 things, whereas (take 10 (transduce (map identity) my-conj (range 1000000)) would realise the whole result of processing the range and then the take would take from the fully realised thing.

correct.

but in your example you use run! over the entire result, so the eduction is not relevant there?

perhaps it is run! I'm not understanding. I thought run! would only have the one element in memory at a time that it was trying to process (unless the collection or collection producing function realised more than one)

run! is effectively just reduce, but you're reducing your entire eduction right. you're not lazily doing anything with your eduction. so in this case transduce or eduction boil down to the same thing

but reducing into a hash is going to take less memory than reducing into a seq of all the data. My understanding is that run! would not hold all of the seq in memory to do its work

but that if it was working on the result of transducing something into a vector then the whole vector would be in memory

yeah, you cannot lazily create a vector result.

but that's not an eduction/transducer problem?

not sure if I still follow :)

I'm not sure if I'm explaining myself badly or if my massive gap in knowledge is tripping me up

Or both 😉

about "laziness" (not the right term here imho), the eduction will be pulled in value by value, then if the input is realized? or not is another matter

(run! prn
      (eduction (map (fn [x]
                       (prn :x x)
                       (Thread/sleep 1000)
                       x))
                (range 10)))

The problem I'm trying to solve is that I need to transform data from a CSV and write it out as partitioned nippy files without blowing up memory

I agree that laziness isn't quite right

it's a pull based thing

(tm)

But eduction is only going to realise values as they are pulled

yes

an eduction is just partial application of xforms over something

And run! isn't going to hold them in memory

no

afaik

Unless I put it in an atom or something

it's like going over an iterator, item by item

it's the same as if you're just running over a lazy seq, no difference there

value by value (sounds better)

kinda sorta, without the cost of a lazy seq

I mean wrt to holding in memory

could be db rows, rs.next

But transduce would produce the whole vector which then run! would operate on

transduce is a bit like reduce, you could throw-away the accumulation every-time, but then why use transduce in the first place (if you mean using transduce instead of run!)

@otfrom My point with using transduce was: you're running over with a side effect. you could do the side effect in transduce instead, saving you the realisation of an eduction. But as I also pointed out, it may not be so important. mpenet has repeated this.

you never really realize an eduction, it never materializes, it's really just (sort of) an iterator

ok, that's a good point yes. no garbage from the eduction.

user=> (defn run!! [f xform coll] (transduce xform (fn ([]) ([x]) ([x y] (f y))) coll))
#'user/run!!
user=> (run!! prn (comp (drop 10) (take 1)) (range))
10
nil

the docstring probably gives a better description than me

I posted that above example to show the run! equivalent for transducers

but run! + eduction works equally well

eductions are awesome 🙂

it's the new juxt

but actually useful

I should create a company with that name maybe

Are you disputing the usefulness of juxt....?

;)

it's muscle flexing in most cases imho!

I usually use it with keywords (map (juxt :field-a :field-b) [{:field-a 1 :field-b 2}])

yeah I prefer select-keys, but I get your point

not really actually, different use

but yes, there are good uses for it. It's just quite rate

rare*

I use juxt all the time, but then I need to create a lot of vector from maps of data to go into excel or csv files, so select-keys doesn't work for me

most of my work is in and out of csv or excel

@borkdude I see what you are getting at with using transduce there now

yeah, but has mpenet has pointed out, the overhead from the eduction might be small enough not to make this an issue

esp as run! is just a reduce with a proc according to the docs

yes it will be very efficient

same here: vectors from maps, for some reason I do this fairly regularly

I not only use juxt, I use (apply juxt vec-of-keys) b/c I'm a monster who does (into [] (map #(friendly-key-lookup %) vec-of-keys) as well

thx for having the patience to go through this with me. I feel I understand a lot more of what is going on. 🙂

I remember a Clojure meetup in Amsterdam with the author of Midje doing a talk and somehow he needed matrix transposition. I just yelled: apply map vector. It's one of these things you just know ;)

it is indeed

and rarely need! I got to use it once, on a job interview 😛

got an offer for that one

I've used it a few times, but then transposing a matrix isn't entirely odd for me

especially not in the case of CSVs where you want to have a column instead of a row

Indeed