ok, i’m thoroughly confused. what would be the best way to get an overview of the active jobs in the cluster, comparable to what onyx dashboard does ? or is the best approach to keep a record of jobs currently in the cluster, separately from onyx ?

i was thinking about aggregating the current log

but for that i need to actually join the cluster, which seems to be a non-trivial thing to do

https://github.com/onyx-platform/onyx-peer-http-query

right, that was plan B 🙂

that’s actually the recommended approach ?

Kinda depends on what you’re trying to do. Are you building your own dashboard?

nah, just a small layer of cli tools on top of onyx for my own internal use

so that i can integrate them in, say, monitoring tools

or that i can query the cluster for currently running jobs, in case i need to do an upgrade

Yea I think that would be easiest to integrate with

Joining and following the log isn’t very difficult, but overkill for something like that.

yep

okay, thanks

Here’s an implementation of a log subscriber https://github.com/onyx-platform/lib-onyx/blob/master/src/lib_onyx/log_subscriber.clj

Not much to it

oh!

It just connects to ZK and shoves events through the Onyx state machine

that doesn’t look too difficult indeed

It’s pretty much what the http replica server does without the nice rest endpoint.

but that doesn’t actually join the cluster, right ?

that’s my problem — subscribing to the log is easy, joining the cluster is a different thing

Why do you need to join the cluster?

it seems i’m not getting any other logs than ‘prepare join cluster’ if i don’t actually join it

What are you using now?

this is what i was doing

(defn list-jobs [config]
  (let [logs (chan)]
    (println "subscribing to onyx")

    (go-loop []
      (let [x (<! logs)]
        (println "got log: " (pr-str x))))

    (onyx.api/subscribe-to-log config logs)

    (Thread/sleep 15000)))

Yea that’ll just loop once, you need to recur

…

😄

right, i’ve had a long day

Heh no worries

thanks for your help @gardnervickers

:thumbsup:

yep, worked like a charm

@lmergen Ha, that one gave me a laugh.

classic case of thinking the problem is way more difficult than it actually was :)

Glad you got through it 🙂

yep. now that you're here, i recall you mentioned that you're actually keeping track of jobs in a separate database at distributed masonry, is this correct ?

is that just to make querying job status a bit faster ?

at this point, im actually considering feeding onyx its own log and ceate aggregates out of it... for science, of course :)

@lmergen Yeah, we have a separate database to track job status. The trick is that it’s only allowed to be updated by a log subscriber. That let’s us avoid having incorrect or stale information. We’re using a separate database because we need to integrate that with other information for Pyroclast. Additionally, we’re seamlessly migrating and tracking job status across Onyx tenancies.

right, that makes sense

You get a deterministic replica everytime you play the log from the beginning, so everytime you boot up your system, you can update your external database with any new information while it was down.

If you’re just making some CLI tools, the log subscriber by itself might be fine. If you have the replica, the active jobs are just (keys (:allocations replica)) 🙂

yes, derive state from onyx, rather than keeping a shadow database

i might take that approach later on. it makes sense i think, especially when doing a lot of migrations

Sounds right.

i think its great how simple onyx has remained. its tempting to build out an ecosystem of tools around it (and there are some, like the dashboard), but the core has remained very simple

and it's very interesting to see the different approaches you can take with a few of these core facilities, such as log subscription

i think rich would be proud :)

@lmergen Thanks 🙂 I wish our public tooling for this stuff was a bit more unified, but I can’t complain about the fact that most of it is still easy to build without needing changes to core.

i think it shows the various directions you can go, without forcing you into a single one

It’s simple - so long as you remember to recur that loop 😄

yeah well, that was simply stupid ;)

One bad habit I can’t drop is using (get :keyword map) instead of (get map :keyword). I have no idea where I picked it up but I do it all the time now.

Chalk me up for that one, too.

Same, probably caused by switching back and forth with erlang lately