Fork me on GitHub
#clojure-uk
<
2021-05-20
>
djm05:05:35

๐Ÿ‘‹

dharrigan06:05:08

Maidin mhaith

djm06:05:01

I see that freenode is imploding. It had been a while since I was in #clojure on there (it was pretty quiet when I was last in). But looks like at least some people have moved over to libera now.

djm06:05:40

(Annoying that Slack won't let you type #foo without it linking to #foo, if that exists as a channel)

djm06:05:30

The downside to text-based communication - I can't tell if you're aware of what's going on, and that's a joke, or if you're unaware and are referring to a literal netsplit

dharrigan07:05:44

I'm aware, just making a somewhat-related injoke

dharrigan07:05:58

however, I'm on Freenode, and I'm not seeing a mass exodus in the channels I'm in

djm07:05:50

I'm not in many channels on freenode anymore anyway, and some of those that I'm in are fairly moribund these days

Aron08:05:41

freenode imploding? to me it seemed like a successful hostile takeover

djm08:05:04

Sure, followed by lots of staff and at least some users leaving.

djm08:05:51

It seemed like a lot of staff. No idea how many staff freenode has altogether these days.

Aron08:05:06

I see what you mean

Luis Thiam-Nye10:05:04

Good Morningโ„ข

folcon17:05:47

Hey, are there any tutorials or anything that really walk someone through core.async or promises in clojurescript? I'm really trying to understand what's going on here and it just ends up being a confusing mess. It's really quite frustrating because I can write async code in python et al really easily, but doing so in clojurescript just doesn't seem to click at all. I invariably get a channel or a promise and I can't get the value out of it... Actually is there some way to drive it in the REPL better? Part of my frustration is that it's very hard to tell if things are working at the REPL. I just get back a ManyToManyChannel, so I feel like I have very little visibility into how it's working.

mccraigmccraig18:05:55

what constructs would you use in python @folcon? not promises ?

folcon18:05:12

primarily multiprocessing

mccraigmccraig18:05:30

if i need to inspect something in the cljs repl i do something like (defn pval [p] (let [a (atom ::undefined)] (p/then p #(reset! a %)) a))

folcon18:05:53

I think my primary issue is not being clear what I'm working against and the different constructs that I've seen give me inconsistent results. For example, Using promises, I'm currently reading in a zip file and walking through the entries. I console.log the entry name and contents and they don't match. The entry name changes, but the contents seem to be permanently pointing to the first entry. Not really sure what's going on there. I'd prefer to use core.async, but I'm not sure how to work with it in a repl in clojurescript. As I mentioned previously I constantly get ManyToManyChannel being the evaluation result, so how do you evaluate functions and get the output to test your assumptions?

mccraigmccraig18:05:53

(since there's no deref available for js promises)

folcon18:05:39

Then bind a to some def and poke it?

mccraigmccraig18:05:02

yep... same idea works for core.async too

mccraigmccraig18:05:49

although unless you are talking promise-chans then chans are going to behave differently

mccraigmccraig18:05:05

if you are wanting to test the async outputs in unit-tests then async is your friend - https://clojurescript.org/tools/testing#async-testing

folcon18:05:50

I've not even gotten that far yet ๐Ÿ˜ƒ

mccraigmccraig18:05:04

what sort of async ops are you doing ?

folcon18:05:56

Got a bunch of xml files inside a zip I'm trying to read

mccraigmccraig19:05:32

so you are having to convert from a js promises/streams API into something cljs ?

folcon19:05:53

Getting lost trying to work with this and then mixing in streams and file api's stuff not working and having to convert to blobs etc...

folcon19:05:04

well that would be the ideal outcome

folcon19:05:14

I'm trying to read the file in chunks if at all possible

folcon19:05:38

I've gotten it working in the past where I get a massive string for the file contents, but that's really unhelpful when someone loads a giant file

mccraigmccraig19:05:52

it sounds entirely doable, but the mix of promises and streams in the js api probably presents some challenges mapping it over to cljs

mccraigmccraig19:05:05

we do loads of stuff like that on the backend, where we use manifold for both promises+streams, but i don't think we do anything like it in js

folcon19:05:13

I'm pretty sure it's doable, just currently not clear how to attack it ๐Ÿ˜ƒ...

mccraigmccraig19:05:14

my default line of attack would be to use promesa for promises, write something to convert between js streams and core.async chans, and write something which reduces a core.async chan to a promise - that would give you something which looks roughly like the manifold api

folcon19:05:08

Thanks for the tip though ๐Ÿ˜ƒ... I'll try it out!

mccraigmccraig19:05:58

oh, i hadn't seen <p! , that looks ok too

folcon19:05:23

Are there any good resources around working with streams in clojurescript?

folcon19:05:49

I'm writing a lot of ad-hoc code and it feels very inelegant ๐Ÿ˜ƒ...

mccraigmccraig20:05:36

don't know - I've never worked with js streams directly... but if you convert pipe then to/from core.async chans then you can just work with the chans

folcon20:05:28

Thanks anyway ๐Ÿ˜ƒ

folcon23:05:10

Finally figured it out, you're supposed to request a promise nested within the entry promise >_<...

folcon13:05:47

The zip lib, I couldn't work out where the data was, I kept reading the buffer expecting it to contain the data, instead you have to within the promise call a function that doesn't appear in the method list (had to read source to find it) which creates another promise which when it resolves gives the file contents...

mccraigmccraig13:05:01

oh, right, so there's a first "open the zipfile" step, then repeated "fetch" steps to get at the actual data ?

folcon14:05:43

It looks like opening the zipfile is more opening a table of contents? Then yes, fetching the contents... One thing I'm uncertain about is that a core async chan is supposed to have a buffer limit right? Is that going to be a problem for this kind of workflow where I could have a zip file with lots of entries and I can't queue them all up?

mccraigmccraig14:05:45

yeah, i think zip/tar are like a filesystem inside a file, with index/metadata pointing off to other offsets inside the file

mccraigmccraig14:05:41

one of the nice things about core.async (and also manifold and rx and js streams and vert.x etc) is backpressure - and core.async handles it very nicely. your >! operation will not "return" until buffer space is available, so your producer process will be "parked" until buffer space is available... symmetrically, your consumer process will park on <! until something is available

mccraigmccraig14:05:52

i use quotes because that's sync terminology, and it's all really async - the go macro transforms everything into callbacks, so >! gets transformed into a put! with a callback and <! gets transformed into a get! with a callback etc

mccraigmccraig14:05:19

but in summary, you don't really have to worry about filling buffers up - there aren't any buffers by default, and your produce will only go at the pace your consumers dictate. if you add some buffers in between producer and consumer then that lets your consumers lag a bit without stopping your producer. in this particular case, where you have control of both producer (whatever is parsing the zipfile and putting content onto a chan) and consumer (whatever is reading content from the chan) you may not need any buffers at all

folcon15:05:03

Does that still work in a js scenario?

mccraigmccraig15:05:05

yes, absolutely - it's not blocking, it's parking - it makes a continuation closure fn, which is registered as a callback, to be called when data is available

mccraigmccraig15:05:58

not a million miles from what js async/await does, or what the promesa/let macro does

folcon19:05:09

Ok that's very cool, going to have to try that out tomorrow...