morning

Bore da

@paulspencerwilliams is your talk going to be recorded?

BTW for those of you on Linux if you ever want to do a screen recording, a talk or even pair with someone and you want to show your keypresses on screen, this is quite cool https://github.com/wavexx/screenkey

Looks like I'll be walking from KX to Camden this morning!

@agile_geek not sure, but it's only an intro. Might try and run it at Brum FP in slightly more detail and should be able to get that one recorded...

@paulspencerwilliams would be good if you can

Morning

morning

morning

Morning all 👋

Suup 🌴

Here's a Clojure question for a change. Is it just me who finds the file io in Clojure awkward? If I want to read a large file lazily I end up having to either: 1. use with-open somewhere at the top of a stack of calls that deal with each line of the file and line-seq in a function to process each line 2. write a custom function that opens a reader and recursively calls a closed over helper fn (with a lazy-seq wrapping the body) and manually calls .read conj'ing the results to its recursive call. E.g.

(defn read-messages [filename]
  (let [reader (io/reader filename)]
    (letfn [(lines []
              (lazy-seq
               (try
                 (if-let [line (.readLine reader)]
                   (cons line (lines))
                   (.close reader))
                 (catch Exception e (when reader (.close reader))))))]
      (lines))))

https://clojuredocs.org/clojure.core/line-seq ?

@jonpither see point 1

I end up with the with-open several levels higher up OR I have to pass the bulk of the app as a higher order fn doseq within the fn that uses line-seq. That also works but I struggle a bit with thinking that way although maybe that's a better approach as it means the fn to read the lines calls the fn stack to process them (inversion of control) but when I start nesting writes inside the fn that processes each line it can get ugly.

I don't think I'm expressing this issue well in here. I will prepare a gist or something tonight with examples of approaches I can think of and post it here to see what people think.

I vaguely remember the one time I had to deal with bigger files it wasn't straight forward.

I guess with small files you don't actually notice it when you do something wrong (eg not close file handle)

I've done it a number of times but with many months in between and it always takes me ages to figure out how to process them lazily in a sensible way. I seem to lose hours on this everytime. But again I am a bit slow sometimes.

@thomas with small files you can just process them eagerly and have the whole file in memory

true

@agile_geek never had a particular problem with the with-open idiom. I see the point to be forced to have it at the top of the computation, but that can always be hidden inside the the first layer that you consider “public interface”.

If you don’t mind the library, https://github.com/thebusby/iota approach is quite smart.

@reborg hmm, I think it's maybe just a hang up from my procedural programming days! The more I think about it the more I think I am favouring having a with-open and a line-seq in a doseq at the top of the computation and passing the body of the doseq the computation to perform on each line.

It's just that I go through this thought process everytime I need to process a file lazily and it never seems to come naturally to me.

@agile_geek perhaps https://github.com/ring-clojure/ring/blob/1.5.0/ring-core/src/ring/util/io.clj#L11?

@agile_geek: I’ve done quite a lot of I/O on large files… including a lot of reifying files as lazy seqs… I’ve come to the conclusion though that the best way is to avoid laziness & I/O altogether… Best option in my mind is to use a transducer if you can.

@rickmoynihan not sure I understand. How can you use a transducer to avoid I/O and laziness?

agile_geek: I/O is one of the main things transducers/reducers were intended to support… it’s just not that widely known.

@rickmoynihan any links to examples?

agile_geek: yeah hold on...

can’t remember the name of it

I've used core.async pipelines with transducers with a with-open at the top of the stack of fn's but I think I was just reading each line in a go loop from the file and putting to a channel

from memory

you don’t need to use channels - you can use the underlying java io methods with them

I'm still struggling to understand.

sorry still trying to find an example

OK - no hurry 😉

sorry I can’t find a better opensource example but you can take a look at this code where I was tying to benchmark whether it was worth writing a transducer based csv parser. Code was never meant to be used, but basically you just hook into CollReduce on the IOstream and write the rest as xforms e.g here’s a really crude CSV parser: https://github.com/RickMoynihan/transducer-csv/blob/master/src/transduce_csv/bench.clj#L36

basically you need something like this bit: https://github.com/RickMoynihan/transducer-csv/blob/master/src/transduce_csv/core.clj#L10

though I think ideally you’d move the .readline out into a transducer function too

there is another repo somewhere that does something similar… wracks brain

@agile_geek: let me know if it still doesn’t make sense

@agile_geek: Ahhh here’s the other repo… IIRC I discovered this after I’d done the above, but it’s basically the same idea: https://github.com/pjstadig/reducible-stream I think it would benefit from being wrapped up at a lower level of abstraction though… more like

Cheers

the big problem with laziness + IO is that the life cycle of the laziness is different from the resource life-cycle… i.e. how do you know when the consumer is done with the sequence? You don’t, hence you have to wrap it in a with-open and consume it eagerly somewhere, which means somewhere you have to treat it like it's eager. Transducers solve this, by actually being eager; but by separating the computational elements out (so it still feels lazy)… e.g. a transducer can know that it’s the final (take 10 ,,,) and close the stream after the items have been taken; and as a user you don’t need to bother wrapping it in with-open & doall anymore

interesting rickmoynihan although I don’t think reducers/xducer are solving all problems

theres no such thing as a silver bullet 🙂

But they do solve the resource issue with I/O - whilst letting you map/filter/etc over files

if you don’t want/need to reduce but just process a lazy-seq you still have that problem of closing the IO resource somewhere. If you reduce you have the possibility to hook up the .close at the end of the reduction which I find quite fine idea

was looking at this from reborg above https://clojurians.slack.com/archives/clojure-uk/p1483959035002035

was wondering what the hashmap equivalent would be (other than a database (key value or otherwise))

reborg: yeah but the problem with lazyseqs and closing the resource is why I’m saying use a (trans|re)ducer.... I think if you want a lazy-seq coupled to I/O you’re doing it wrong. lazy-seq + I/O has historically been used a lot, but Rich has been saying it’s bad since basically Clojure 1.0

@otfrom: thanks - I’d forgotten about iota.

the general theme i’m seeing is passing the data processing function into the IO code, rather than passing the IO reference into some data processing code

glenjamin: +1

I think the biggest issue with this approach is how to use it with things like ring… e.g. to my knowledge you can’t really return a transducer to ring

I'm wondering if konserve might be my huckleberry https://github.com/replikativ/konserve

rickmoynihan not sure there is a wrong/good, I suppose it depends on your requirements. Personally I’d say use lazy-seqs with IO if your app is fetching/processing/spitting out and you can’t afford to bring the thing in memory

a lot of CRDT stuff there looks very intersting

i think the key thing is that the lazy-seq should stay within a single function, and not get passed around

glenjamin: agreed - but then it’s not really like a normal lazy seq

@glenjamin I agree about the "passing the data processing function into the IO code" but I need a way to remember this! I think I feel a blog post coming on. Usually helps me solidify my thinking

reborg: You can totally do that… but I think the problem is that clojure.core doesn’t have transducible-IO library… Having done a huge amount of lazy-seq I/O I’m trying to slowly move away from it - because the costs of laziness are really high for large datasets and controlling resource cycle is a pain. You can use transducer/reducer based I/O and still not load everything into memory.

As soon as you reduce (xform or not) you are not loading everything into memory by design 🙂 Is only that not all problems can be solved with a reduce as the last form

reborg: not true…

You can e.g. reduce into a file or an outputstream

just need to implement CollReduce and friends

it basically entirely depends on the reducing function… e.g. here I transduce over a 1gb CSV file to count the lines… I don’t hold more than a line of the file in memory at any one point in time… Infact it’s actually much easier on memory/GC than the equivalent lazy-seq code: https://github.com/RickMoynihan/transducer-csv/blob/master/src/transduce_csv/bench.clj#L36

the reducing function could equally just write it to an outputstream (to do a file copy)

What I mean is that (reduce + (range 1e9)) is not going to load all the 1e9 items in memory at any given time. The head is consumed and garbage collected and yes, depending on the reducing function.

My point tho is that all suggestions in this thread are equally interesting.

reborg: completely agree… and like I said you can use lazy-seq’s for I/O - it’s something I’ve done a huge amount… however maintaing a code base with a lot of I/O done in that style makes me yearn for something better… e.g. hacks where you add (.close rdr) to the end of a lazy seq is far from ideal.

I implemented a lexer+parser with transducers once

It was a bit weird, but mostly worked

@agile_geek upgrade from laziness and go async - manifold streams have on-closed and on-drained callbacks which can be used for the resource close - i would certainly like to use a stream based file i/o lib, though i haven't cared enough to implement one yet. some stream info - https://github.com/ztellman/manifold/blob/master/docs/stream.md

mccraigmccraig: it’s definitely an option, and a lot of people seem to be moving that way… I do wonder if core.async chans could compete with the performance of a transducer based I/O solution though… also transducers can work with core.async chans too so not necessarily mutually exclusive solutions… e.g could probably wire a CollReduce reader through a transducer into a core.async channel pretty seamlessly

bigger issues are as you say having to be aware of the differences between blocking/non-blocking I/O - I’ve been assuming blocking I/O so far as nio is not supported by libraries I need

and whilst nio is trendy - I really don’t typically need to handle 100k concurrent connections on a single server

yes, without nio you are stuffed for async models - manifold will cheat for you and use a threadpool to manage blocking actions, but there's not a lot of point to using that feature if all your i/o ops are blocking

yeah - we’re pretty tied to a suite of parsers built on blocking I/O - async doesn’t make much sense for us

I really do wish the clojure I/O ecosystem was a little less fractured

rickmoynihan mccraigmccraig who has the best I/O approach in your opinion or do we need to come up with one?

otfrom: I think there’s necessarily basically two approaches to I/O on the JVM: async & blocking, so clojure needs at a minimum two approaches also. I think in an ideal world clojure would provide an io mechanism that extended CollReduce etc to readers out of the box and that backing I/O and effects with lazy-seq’s would be frowned upon or treated as deprecated/legacy. So yeah I think there’s work needed to unify this stuff - and that is probably "a new way”.

i/o covers a tonne of stuff @otfrom ... i've been happy with my network i/o options recently (via aleph), but i would have liked an async file i/o lib

converting async to blocking is easy though, so i think there only needs to be one async approach !

(with some blocking lip-gloss)

there are some problems though… I had thought that because you can call sequence on a transducer to return a lazy seq - you could potentially interop with lazy-seq stuff whilst having a transducer backed solution through this mechanism; but I’m not sure it has some problems as Seqable etc aren’t protocols - so might be doable but YMMV on that front. async -> blocking can be done as mccraigmccraig says; but if the abstraction has a cost (core.asyc chan’s seem to), I’m not sure you want to require everything to be expressed as an async thing… I think transducers might provide a way out though… as you can pick an async/blocking source/destination depending on your needs and providing you mix in the appropriate xforms write the bulk of the transducer with no knowledge of either.

also there’s been murmerings from cognitect that transducers will eventually be made to support the parallel cases of reducers too

you could e.g. do something like (def xf (comp (->AsyncFile “/blah/file.csv”) async-split-lines record->hash-map)) or (def xf (comp (->BlockingFile “/blah/file.csv”) split-lines record->hash-map)) But basically the user should decide.

I’ve not though too much about the async case though

ok lunch time!

well, i'm all for abstractions which let the user decide

I ended up with a function that took a reader and a transducer, and applied it: https://github.com/glenjamin/hand-parser/blob/master/src/hand/parser.clj#L5

similar effect to implementing CollReduce I think

lol readuction :thumbsup:

mccraigmccraig: totally agree that i/o covers a tonne of things though - so might not be possible to have one approach to rule them all

i recall being rather happy with that name

Anyone 'attending' Clojure Remote?

I thought the tickets were really expensive Tbh. Though there might be a good reason.

not sure when you saw the prices, but they got dropped recently to 425. Need to start writing some slides really!

thought i had a link for 10%, but not convinced it's working right