This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # beginners (114)
- # boot (10)
- # cljs-dev (18)
- # clojure (57)
- # clojure-russia (12)
- # clojure-spec (2)
- # clojure-uk (1)
- # clojurescript (62)
- # cursive (49)
- # datomic (3)
- # emacs (2)
- # mount (1)
- # off-topic (25)
- # onyx (58)
- # pedestal (5)
- # re-frame (5)
- # ring-swagger (1)
- # specter (8)
- # unrepl (2)
- # untangled (4)
- # vim (10)
- # yada (39)
ok, i’m thoroughly confused. what would be the best way to get an overview of the active jobs in the cluster, comparable to what onyx dashboard does ? or is the best approach to keep a record of jobs currently in the cluster, separately from onyx ?
but for that i need to actually join the cluster, which seems to be a non-trivial thing to do
Kinda depends on what you’re trying to do. Are you building your own dashboard?
or that i can query the cluster for currently running jobs, in case i need to do an upgrade
Joining and following the log isn’t very difficult, but overkill for something like that.
Here’s an implementation of a log subscriber https://github.com/onyx-platform/lib-onyx/blob/master/src/lib_onyx/log_subscriber.clj
It’s pretty much what the http replica server does without the nice rest endpoint.
that’s my problem — subscribing to the log is easy, joining the cluster is a different thing
it seems i’m not getting any other logs than ‘prepare join cluster’ if i don’t actually join it
this is what i was doing
(defn list-jobs [config] (let [logs (chan)] (println "subscribing to onyx") (go-loop  (let [x (<! logs)] (println "got log: " (pr-str x)))) (onyx.api/subscribe-to-log config logs) (Thread/sleep 15000)))
yep. now that you're here, i recall you mentioned that you're actually keeping track of jobs in a separate database at distributed masonry, is this correct ?
at this point, im actually considering feeding onyx its own log and ceate aggregates out of it... for science, of course :)
@lmergen Yeah, we have a separate database to track job status. The trick is that it’s only allowed to be updated by a log subscriber. That let’s us avoid having incorrect or stale information. We’re using a separate database because we need to integrate that with other information for Pyroclast. Additionally, we’re seamlessly migrating and tracking job status across Onyx tenancies.
You get a deterministic replica everytime you play the log from the beginning, so everytime you boot up your system, you can update your external database with any new information while it was down.
If you’re just making some CLI tools, the log subscriber by itself might be fine. If you have the replica, the active jobs are just
(keys (:allocations replica)) 🙂
i might take that approach later on. it makes sense i think, especially when doing a lot of migrations
i think its great how simple onyx has remained. its tempting to build out an ecosystem of tools around it (and there are some, like the dashboard), but the core has remained very simple
and it's very interesting to see the different approaches you can take with a few of these core facilities, such as log subscription
@lmergen Thanks 🙂 I wish our public tooling for this stuff was a bit more unified, but I can’t complain about the fact that most of it is still easy to build without needing changes to core.
i think it shows the various directions you can go, without forcing you into a single one
One bad habit I can’t drop is using
(get :keyword map) instead of
(get map :keyword). I have no idea where I picked it up but I do it all the time now.