Fork me on GitHub
#onyx
<
2016-10-18
>
yonatanel07:10:57

Implementing a plugin doesn't require both input and output, correct?

lucasbradstreet08:10:00

That's right, you can do them independently

asolovyov14:10:03

Hey all! I have a weird problem: my django app writes something to database, sends an event to Kafka about that and then commits transaction to database. My onyx app is so fast (obviously) that it reads stuff from kafka before it appears in database, which throws an error and kills the job. I wonder what's the best way to deal with that. It seems that there is no way I can pass anything from lifecycle handle-exception to the task, so there is no way to make 'retry this task 3 times before giving up', right? I can of course hardcode that inside of a task itself...

asolovyov14:10:05

yeah, but I don't want it to be stuck forever...

asolovyov14:10:20

like, die please if there is nothing in database in the next 10 seconds or so

lucasbradstreet14:10:19

Limiting retries are certainly a bit of a problem with the current design. Could you stick a timestamp on the message (I think Kafka can do timestamps now too) and make the flow conditions drop it after some time period?

yonatanel14:10:35

@asolovyov Why do you commit the transaction after sending to kafka? What happens if sending to kafka fails, or if committing fails after successfully sending to kafka?

michaeldrogalis15:10:41

Need to push back the release of the thing I mentioned yesterday, apologies. Needs a few more days.

michaeldrogalis15:10:22

We're basically coming out with some new test/single-node facilities. They're quite nice to use, but need to be completely finished before being useful. Hit a snag last night trying to complete it.

asolovyov16:10:41

@yonatanel: you're asking hard questions. Too hard for our old little django app :)

asolovyov16:10:20

@lucasbradstreet: that's a great idea! Thanks!

asolovyov16:10:26

I wonder though if I can fix django app...

yonatanel16:10:01

@asolovyov Maybe you can batch the input in X seconds windows and then delay a bit the processing of each window.

yonatanel16:10:35

Though I don't believe :onyx/batch-fn? supports time based batches, and I didn't see an option to link a window accumulation to a task.

michaeldrogalis16:10:14

You can control an upper bound on the amount of time to wait for a batch to be read with :onyx/batch-timeout, but I dont think thats what you want here.

asolovyov19:10:31

I think I've got an idea how to fix django, a hack, but whatever, I plan to throw it out sooner or later :)

michaeldrogalis20:10:27

Got things done a little quicker than we thought we would. Blog post tomorrow, but worth checking out today.

michaeldrogalis20:10:14

Happy to announce onyx-local-rt, an alternate runtime for Onyx that is pure and deterministic, and runs in ClojureScript: https://github.com/onyx-platform/onyx-local-rt

michaeldrogalis20:10:16

Almost all API features of the distributed runtime work, so you can switch your tests over to use the local-rt. Tests go from ~1200ms down to < 10s on my machine.

Travis20:10:06

Very cool, but why support clojurescript? Just wondering its purpose?

michaeldrogalis20:10:54

@camechis Half because "why not" since it all we mostly had to do was convert files from clj to cljc, half because we get a lot of people who like the programming model but don't need the parallelism or fault tolerance.

yonatanel21:10:18

Onyx in the browser as a teaching tool for example would be a nice excuse to learn some Om.

michaeldrogalis21:10:53

I almost want to switch learn-onyx over to use it so new comers don't need to futz around with ZooKeeper or BookKeeper.

yonatanel21:10:04

I don't recall doing anything except running the repl for learn-onyx.

michaeldrogalis21:10:35

It's pretty snappy, we've done a lot of work to make it smooth. Probably a good idea to just leave it alone.

michaeldrogalis21:10:13

But, anyhow. I'll write a longer post tomorrow. onyx-local-rt should be useful for testing, teaching, in-browser usage, and single-node usage.

Travis21:10:06

Very cool, might switch over to it soon