Fork me on GitHub
#clojure-uk
<
2021-02-19
>
jiriknesl07:02:17

Good morning

dharrigan08:02:38

Good Morning!

djm08:02:44

πŸ‘‹

mccraigmccraig10:02:58

mΓ₯ningΒ‘

folcon12:02:06

Any resources for working with large files / streaming in clojurescript? Specifically reading / building them for download client side? Should I be looking at core.async? This is within a reframe app if that helps?

alexlynham14:02:38

in our app we build some moderately sized dumps, we basically write them to an S3 bucket and give the receipient back a signed s3 link, but then we don't have to stream it directly to the client :thinking_face: not sure what we'd do if we had to stream it, probably do something that wasn't a standard http response

folcon16:02:07

Oh streaming in this sense is managing a local file stream to an output which can be downloaded, where the file is larger than the memory chome allows...

mccraigmccraig17:02:27

i've used the cordova file api , which i gather is based on the html 5 file api @folcon

mccraigmccraig17:02:17

if that's the api you are looking at, it's all promise-based, so you might find it easier to use something like funcool/promesa than core.async

folcon00:02:06

thanks πŸ˜ƒ... It's really hard to tell how popular core async is, some people swear by it and others seem to avoid it like the plague. I'll dig into using promesa, I was specifically thinking of core.async because it's transducer compatible, basically my thought process was build up a stack of xforms and then apply them. Perhaps I can do the same with promises? :thinking_face:

mccraigmccraig13:02:15

i use both of them - i like core.async for stream-like things (e.g. a stream of events, or a stream of file buffers), but i find it awkward for promise-like things (e.g. a single http request)

mccraigmccraig13:02:36

basically i really like the manifold programming model, which uses both promises (aka Deferred and Stream together - and it's fine to use core.async and promesa together to similar effect in cljs, although you won't get the automatic glue that manifold gives you (e.g. a reduce of a Stream returns a Deferred)

mccraigmccraig14:02:03

i'm currently working on a thing which will give you a manifold-like model, but cross-platform and with contextual programming benefits, but it's not going to be ready for a while yet

mccraigmccraig14:02:51

for file-handling my ideal API would be based around promises, and where content streaming is required, promises of core.async chans

mccraigmccraig14:02:05

e.g. file/exists? returns promise<bool>, file/content-stream returns promise<chan<buffer>>

folcon15:02:46

Ah, that might not be a good fit then, the files in question are file objects that you would get from within a browser context of an upload or drag-n-drop event, so perhaps core.async might be a better fit?

folcon15:02:05

Those events give you individual files or an array of files

mccraigmccraig18:02:31

not sure without more details - but if you've got more detail i'm happy to give you my take

folcon23:02:29

Sure πŸ˜ƒ... Let me know what details you need and if the explanation that I give below is insufficient πŸ˜ƒ... I've got lots of Files[0] and a stack of transforms that I run which I'm then using to create blobs[1]. I don't quite know what transforms I'm running until later, but sometimes I need information about the file contents to work that out. Then once I build the appropriate transform stack I then run all the transforms on the file contents to build a blob which I then use to trigger a download. My current approach is to just read the file contents and then use that to work everything out, but of course for large files you can't just do that because you'll crash the browser tab with an OOM. So I'm thinking can I build a xform of any transformations I want to run on the file and then apply them to the file in chunks, using slice, or just a stream. One potential bit of fiddliness is that I might have to do this more than once, as some intermediate steps towards building the xform may require me to run the current xform state across the whole file to work out what the next step is.:thinking_face: - [0]: https://developer.mozilla.org/en-US/docs/Web/API/File - [1]: https://developer.mozilla.org/en-US/docs/Web/API/Blob

mccraigmccraig12:02:40

well, those Blobs have a .stream() method which returns a ReadableStream - which is a stream of values very much like a core.async chan, so i would convert that ReadableStream into a core.async chan, and go from there with your xform stack ... the principle challenge will probably be in dealing with errors - iirc the ReadableStream has an error state which core.async chans don't - so you'll have to map the ReadableStream error-state to a distinguished error value on the chan

folcon12:02:00

Thanks for that, I'll give that a go πŸ˜ƒ, do you mind if I follow up at some point later if I hit an issue?

mccraigmccraig12:02:26

not at all πŸ™‚

1
folcon12:04:22

So @mccraigmccraig I've been trying this to a degree and one issue that I've been having is I'm working in re-frame, so I can't have long running processors just going and hogging the event queue. So I've been converting it all into batched steps where an event call does one chunk of work and then consumes a queue of work until it's completed and then continues on. I'm wondering if there's a better way to do this that still utilises core.async as then I can elegantly use xforms or perhaps I'm thinking about this the wrong way? Not that I'm expecting you to know that environment, but I figured I'd just ask just in case πŸ˜‰...

mccraigmccraig12:04:37

i've not worked with long-running CPU intensive processes in js-land ... done lots on the back-end though - the approach that has stuck best is to divide work into chunks and put messages describing those chunks onto a stream/chan, then xform the stream/chan (wherein the work is done) until finally reducing it for a result

mccraigmccraig12:04:06

we also use re-frame - i wouldn't use the re-frame event system for scheduling work though - maybe step outside to core.async and dispatch a re-frame event when you are done

mccraigmccraig12:04:39

and if you need some introspection into what's going on inside of a core.async process, define a re-frame cofx to fetch that info

folcon15:04:36

https://clojurians.slack.com/archives/C064BA6G2/p1617798039147000?thread_ts=1613754317.404500&amp;cid=C064BA6G2 Is there an example of this you can point to? I'm not really familiar with using cofx's as I've not really grokked what they're for and how they fit with everything else 😳...

folcon15:04:03

https://clojurians.slack.com/archives/C064BA6G2/p1617797946146800?thread_ts=1613754317.404500&amp;cid=C064BA6G2 I'm assuming with this you mean dispatch a reframe event that starts some long running work in core.async and then fire an event when it's done. Wouldn't there be contention between core.async and reframe? Not super sure how I can also track progress other than dispatching a progress update event to re-frame as well...

mccraigmccraig15:04:05

cofx are inputs to a re-frame event handler from outside of re-frame - they are input side-effects in the pure world of re-frame (while fx are output side-effects). if you need your re-frame event handlers to consider any data which isn't in the event then you want a cofx (it's quite consistent - the app-db itself is a cofx, while any modification to the app-db is an fx)

mccraigmccraig15:04:29

you could, of course, just dereference any old atom in your event-handler - but then your event-handler isn't pure and is hard to test, so instead you do the atom deref in a cofx and now your event-handler is pure and easy to test and the impurity is isolated in the cofx

mccraigmccraig15:04:02

so if you do some async stateful stuff outside of re-frame and you want to look in on it then you can either use a cofx (e.g. with an event dispatched on a timer) or you can dispatch re-frame events from your process

mccraigmccraig15:04:57

yes, there could be contention btw re-frame and whatever work you are doing - if that's going to be a serious issue, then you might want to look at web workers e.g. https://shadow-cljs.github.io/docs/UsersGuide.html#_web_workers

folcon14:05:42

@mccraigmccraig Currently going through these tips you gave me and trying to work it out πŸ˜ƒ... Finally got some time to think about this πŸ˜‰...

folcon14:05:09

As far as I can tell, the only way to get introspection into the core.async process (ie tracking how much progress into a long running task has been completed) is for it to fire an re-frame dispatch event as it goes writing to the app db where it is, and then having the application display that progress marker...

mccraigmccraig09:05:46

alternatively, keep an atom with progress/state in, and fetch from that state in a cofx, maybe from an event dispatched by a timer

jiriknesl20:02:28

Today, we have been discussing convetions in naming functions. Like having the same naming for the same functions. We have arrived to this:

;; dump - read all values from the storage
;; load - put all values to the storage
;; set! - set value
;; find - read 0..n values from the storage
;; fetch! throws an exception when cannot find, otherwise return 1 record
;; exists? - returns boolean
What’s your way to name functions?

seancorfield20:02:39

For me, dump and load are the wrong way round: I'm used to load meaning load-from-storage and store meaning store-to-storage -- I think we only use dump to mean display a complete in-memory structure (either on screen or write it to disk).

dharrigan20:02:22

I'm the same, dump to me always meanings writing to file, network, somewhere more long-term, persistent.

dharrigan20:02:32

load is to load into memory

folcon12:04:22

So @mccraigmccraig I've been trying this to a degree and one issue that I've been having is I'm working in re-frame, so I can't have long running processors just going and hogging the event queue. So I've been converting it all into batched steps where an event call does one chunk of work and then consumes a queue of work until it's completed and then continues on. I'm wondering if there's a better way to do this that still utilises core.async as then I can elegantly use xforms or perhaps I'm thinking about this the wrong way? Not that I'm expecting you to know that environment, but I figured I'd just ask just in case πŸ˜‰...

folcon14:05:42

@mccraigmccraig Currently going through these tips you gave me and trying to work it out πŸ˜ƒ... Finally got some time to think about this πŸ˜‰...