@michaeldrogalis: Actually, I guess it's the same as when you start a job on every node in the cluster, so never mind.

@dave.dixon: Every subscriber receives every event in the log in the same order.

You can idempotently start a job by suppling the job ID.

@dave.dixon: See the docstring for onyx.api/submit-job.

@michaeldrogalis: I have to first do onyx.api/kill-job then onyx.api/submit-job in each subscriber, right? I assume these don't step on each other if issued at different times from different nodes.

@dave.dixon: kill-job is also idempotent, yes. Once a job has been killed, it cannot be restarted. Would recommend having your trigger write out a UUID to the log to serve as the new job ID. Then your subscriber(s) can resubmit the same job content through a different ID, which avoids the situation where two subscribers do submit, kill, submit, and thus make no progress.

@michaeldrogalis: thx, makes sense.

Looks like Onyx was mentioned in this InfoWorld post on viable alternatives to Spark. Clojure platforms are often pretty niche compared to their Scala/Java counterparts. It’s awesome to see some recognition! http://www.infoworld.com/article/3101729/big-data/big-data-brawlers-4-challengers-to-spark.html

We're groooowing. 😄

Awesome!

@michaeldrogalis: If I'm sending a command like :restart-job via the log, do I need to implement extensions/apply-log-entry etc for :restart job? Currently getting error because this is not implemented, but maybe I'm missing something in how I should write to the log?

@dave.dixon: Yes, because all peers will consume all log messages, and need to know how to update their local replicas and state accordingly. all-log-entry, in your case, should just return the replica unaltered

fire-side-effects should be a no-op, same with replica-diff.

@michaeldrogalis: cool, that all worked, thanks for your help. Good exercise in understanding the internals of onyx.

@dave.dixon: You're welcome 🙂

Hi there! I’ve set up a lifecycle that injects a connection to the database for a window/trigger combination which persists the results of the workflow. I have a need now to keep track of “bad data” in the database as well but wanted to handle that in a flow condition. Can I reuse the connection from the lifecycle so that I don’t need to set up a separate connection for the flow condition?

@jholmberg: Yes. The connection object in the event map will be the same.

Thanks @michaeldrogalis, that will save me from having to set up an unneeded connection!

small typo: https://github.com/onyx-platform/learn-onyx/blob/master/test/workshop/jobs/challenge_6_0_test.clj#L135 s/deliver/delivered/g

@aaelony: I found some typos a couple weeks ago and submitted a PR; I believe that's the recommended thing to do, if you can.