Maybe this helps? https://github.com/cemerick/clojure-type-selection-flowchart

Thanks, you’re right, it does. I forgot about that flowchart. Looks like I’ll need to use gen-class.

You can also split it into two things if that's OK in your scenario - e.g. return a vector of [(proxy ...) (fn [] (str sb))].

Also defprotocol creates an interface, so maybe that's usable. There's also definterface but it works only with AOT.

Or maybe you can even feed that proxy into reify. Dunno, haven't tested it.

Doesn't StringReader release the captured text with toString? Everything has a toString and perhaps PushbackReader's toString is not already very important?

Yes, you’re right. Just using toString is probably the simplest solution here, especially considering that this is only for the private use of a single namespace anyway.

Reduce is not lazy.

Reduce lazily consumes the seq you pass in though, right? Like, I'm only on the hook for the size of whatever reduce returns, not what gets inputted into it?

If your processing was truly lazy, you'd get a file closed exception, so your processing must be operating greedily, presumably because of the reduce. Also I would have thought ordered map would have to hold everything in memory in order to do the ordering? Anyway, memory usage will surely depend on how much 'reduction' is occurring in your reduce step.

> Reduce lazily consumes the seq you pass in though, right? No, reduce doesn't lazily consume, nor lazily return: E.g.

(->> (range)
     (reduce conj [])
     (take 10))

If the sequence you pass to reduce is lazy, then it will be consumed lazily.

(->> (range)
     (reduce (fn [acc x]
               (println "X" x)
               (conj acc x)) [])
     (take 10))

😄 yeah. I'm not sure there's a difference between (range) "lazily producing" and anything downstream "lazily consuming". That is, I don't think there's anything special about reduce that is "lazily consuming" here, right @U7S5E44DB?

Yeah exactly. Which is what @UJLF48QJC is looking for (reducing an arbitrarily large file)

> If your processing was truly lazy, you'd get a file closed exception, so your processing must be operating greedily Won't with-open keep it open until (in this case) (rows->schema ...) returns? But rows->schema itself is operating on lazily-loaded line by lazily-loaded line?

If your rows->schema returned a lazy result then no it wouldn’t. Eg try returning something that returns a lazy result (eg map) and see what happens.

ah, that makes sense then—`rows->schema` does not return a lazy result

To be a bit pedantic, even if it did return a lazy sequence (a map instead of a ordered-map for example), the file would still be fully consumed at the reduce step

For your other question, how to test it doesn’t blow up the memory in the future, I’m not sure how to do it (besides running the test in a memory constraint environment…), but there’s ways to test whether you’ve fully consumed a lazy-sequence.

(defn number-generator 
  ([] (number-generator 1))
  ([n]
   (if (< n 1000)
     (lazy-seq (cons n (number-generator (inc n))))
     (lazy-seq (throw (Exception. "Fully consumed"))))))

(take 999 (number-generator))  ;; works
(take 1000 (number-generator)) ;; throws exception "Fully consumed"

You could create a lazy-sequence that would throw an exception after a certain number of items have been consumed. It doesn’t help for your rows->schema function though, as you want everything to be consumed once you reach reduce

Has OP's question about whether the [header & rows] formulation "holds the head" been answered?

i don’t think so. it got wrapped up in lazy or not-lazy talk that the core question remained unanswered.

This isn't the most rigorous experiment, but memory usage remained constant when I ran

(let [[head & tail] (repeatedly rand)]
  (reduce max tail))

In this example, isn’t header simply a collection of column headings? It doesn’t hold a sequence of the other rows. It’s the destructured ‘rows’ that holds the head of a lazy sequence, but you then greedily process that so the sequence is fully realised by the end of the with-open block.

You have to pass the :readers option to edn/read-string. You can just pass it *data-readers*if you just want to use the current ones

oh ok

My guess is you have some sort of lazy/partially-realized data structure that the println is forcing evaluation of and making it serializable.

I tried running doall on the datastructure, but I guess if there are nested lazy-sequences a top-level doall won't realize everything?

somewhere nested inside you have a seq over an Iterable source

the more details you can reveal, the better

this object is being read in from a transit string using cognitect.transit/reader . I don't need laziness. It's a business object, let me try to obscure it a bit

it's roughly like this, arbitrarily nested:

(lazy-seq-1 :a-val (lazy-seq-2 :b-val (lazy-seq-3 :c-val)))

are you transforming what you read from transit? using eduction?

no, it's being read just like this with no further processing

(with-open [input-stream (ByteArrayInputStream. (.getBytes transit-str))]
     (transit/read (transit/reader input-stream :json opts)))

(type (<-transit-json-str (->transit-json-str (list 1 2 3))))
#_#<Class@14a50707 clojure.lang.LazySeq>

you can call vec on all your lazy seqs

Yeah, I was able to override the transit list handler to match what I wanted and now my value is serializable:

(def persistent-list-opts
  {:handlers {"list" (transit/read-handler (fn [v] (apply list v)))}})

thanks for your help, all