This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-01-06
Channels
- # architecture (8)
- # aws (2)
- # beginners (156)
- # boot (163)
- # cider (22)
- # cljs-dev (2)
- # cljsrn (11)
- # clojars (6)
- # clojure (328)
- # clojure-austin (7)
- # clojure-dusseldorf (10)
- # clojure-italy (2)
- # clojure-russia (19)
- # clojure-spec (178)
- # clojure-uk (86)
- # clojurescript (81)
- # cursive (17)
- # datomic (33)
- # funcool (40)
- # hoplon (8)
- # jobs (5)
- # klipse (13)
- # leiningen (1)
- # luminus (21)
- # off-topic (140)
- # om (49)
- # om-next (4)
- # onyx (29)
- # planck (5)
- # protorepl (2)
- # re-frame (58)
- # reagent (2)
- # remote-jobs (4)
- # ring-swagger (16)
- # testing (1)
- # untangled (26)
- # yada (27)
@isaac I would recommend maintaining your own copy of the job data. It’s available in ZooKeeper, but it’s not meant to be stored there permanently.
However there are many different things that we want to hang off the txn's, e.g. send-email or generate-billing etc etc
The latter is preferable most of the time. If an unhandled user-level exception occurs in one job, it won’t disrupt the others from making progress.
@len Sorry, a bit confused. You’re asking about one job receiving some information, and turning around to start other jobs?
@len Would recommend having N long running jobs for each thing you’re doing.
Rather than launching one-off jobs.
It’s not, but operationally I think you’ll have an easier time not doing that.
Kafka is a pretty good in-between since you can backlog all of your intermediate work.
Does the :onyx/batch-size
setting of a task signify the amount of segments passed to that task? Or the amount of segments that task passes to downstream tasks? I'm curious what happens if you set varying batch sizes for different tasks in a job.
@aengelberg It controls the read-factor. I believe we have an open issue for making a separate parameter to control the write-factor.
It’s expected that multiple tasks within a job will have different :onyx/batch-size
values, but sometimes you don’t need that level of tuning and using the same value throughout works alright.
To be more specific, :onyx/batch-size
controls the number of segments that will be read from that peer’s Aeron channel before it begins processing them. It will wait at most :onyx/batch-timeout
milliseconds before giving up and processing them in the case where it doesn’t reach :onyx/batch-size
.
i’m trying to find docs on the onyx http api (for listing what jobs are running and such) my search skills seem weak today. can anyone point me in the right direction?
Anytime!