Fork me on GitHub
#core-async
<
2016-04-07
>
royalaid17:04:34

That direct link seems to be 404ing @angusiguess

pre17:04:14

@angusiguess: Running start-supervisor throws “an invalid offset has been specified”. It would be nice to see your test, replacing simple-consumer test-check data with a real channel, so we can test this setup with a stream.

pre17:04:57

Great post.

angusiguess17:04:58

I'll polish things up a little bit more tonight, I have some more feedback to address.

pre17:04:33

No worries. I’m currently implementing an AWS data pipeline on core.async, and while we have a few nice abstractions, the error recovery is still a mystery. Your inputs are great.

angusiguess17:04:56

Excellent! I'm slowly working up to trying to write a library to codify some of this.

pre17:04:30

It would be nice to see your example devoid of any assumptions on client/consumer. It might as well be any external Queue. not just kafka.

royalaid17:04:29

Wondering if there is a proscribed way to do unit testing of stuff involving channels?

pre17:04:42

testing is hard, but I’ve been using repl and plain old clj.test with a consumer/producer.

royalaid17:04:40

I am doing a similar thing, just grabbing all of the contents of a channel with (take-while some? (repeatedly #(poll! testing-chan)))

royalaid17:04:04

then making some assertions about the returned results

angusiguess18:04:27

I've written a good few async tests. The small unit tests usually involve pumping individual signals through and checking the results.

angusiguess18:04:45

I've been working a little bit with property based tests too but still working out how best to do that.

angusiguess18:04:14

Using alts with a timeout and an assertion will fail the test rather than just hanging on blocking takes.

angusiguess18:04:27

I'm going to talk about it in a bit but another thing you can do, if you have a stateful go-loop is to dump that state and assert on it.

angusiguess18:04:33

But I like signals-in signals-out more.

johanatan20:04:44

@ghadi: we are talking about ~38 secs instead of 30 secs. I'm running on AWS micro and I think it's just not buff enough to handle the load

ghadi20:04:23

i don't know what you mean

pre22:04:40

@angusiguess: you might want to opensource your async.test lib

pre22:04:09

signals in —> signals out

angusiguess22:04:49

@pri: I’ll have to think about how to structure things a little more but I can definitely talk about some of it too.

pre22:04:09

check it in … happy to help.

angusiguess22:04:24

One of the other benefits of extracting these functions from our go-loop is that we can actually test the functions themselves too.

angusiguess22:04:43

I’ve updated the blog post and the code to address some feedback, the producer should now recover properly and keep reading.