I think they may actually be broken, if I remember correctly

unless I fixed it heh

You have a branch with some WIP comments

and there's also an issue about triggers firing

https://github.com/onyx-platform/onyx-local-rt/issues/4

√

Yeah, I had forgotten about that issue. @michaeldrogalis the triggers don’t even fire for the segment triggers

https://github.com/onyx-platform/onyx-local-rt/tree/fix-trigger-fires

Yeah, that's what I am seeing. I switched my window and triggers to elements, and still nothing fires.

I’ll fix it today. Thanks for the reminder

So the job completes with all the segments in the outbox, but no triggers ever fire, which seems to make sense given the issue + what you said.

No worries, I just was thinking I did something wrong until I finally started poking around the source and then saw the issue after that.

Thanks, look forward to a fix when you get a chance.

That’s strange, could have sworn I had all the triggers firing before I put it down.

Ill check it out tonight and see if there was a regression somewhere. Maybe I’m losing my marbles and didn’t have them working to begin with. o-o

@michaeldrogalis this test is enough to check that it’s not working

@lucasbradstreet Yeah, I mean before we cut the first release.

ah right

was mostly pointing you to something you could use to debug it

to save you some time

Oh. Misunderstood you there. Yeah

On that note, if anyone can figure out a way to make most of Onyx core’s tests run against local-rt, that’d be pretty awesome. That’s been the goal.

Segment trigger in onyx-local-rt worked for me some time ago

Perhaps I’m not losing it 😛

I have an Onyx job that continuously reads data from a web socket. What's the best way to safely tell it to shutdown? I have other jobs that read from Kafka and in those cases, I just send the :done sentinel across the topic. But with this particular job, I'm connecting to a 3rd party site and therefore don't have control of the producer.

@stephenmhopper onyx.api/kill-job

@michaeldrogalis Does that allow segments to flush from the system before killing the job?

@stephenmhopper It doesn’t, no.

Okay, I think it should work fine for the job I have that just reads from a websocket. But the jobs that are reading from Kafka are using onyx-kafka so I'm guessing that if I use onyx.api/kill-job on a job after it's sent an ack to Kafka but before it finishes processing, that it could potentially result in me skipping messages when I restart if I don't reset?

@stephenmhopper onyx-kafka will only ack messages that have completed their path through Onyx, no messages should be lost.

We can do that because Kafka has acknowledgement built in - can’t do the same for websockets because they lack that feature

after gc we loose info about any historical data right?

@mariusz_jachimowicz Some. Completed and killed jobs are cleared: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/log/commands/gc.clj#L17-L28

I think that people often want to see this historical data - summarize about what jobs was finished, killed, jobs duration. We could store this summarized data in some node in ZK rather than getting from replica. What do you think?

We need to periodically clear away completed jobs to keep the amount of data in ZooKeeper manageable and the memory size for the replica limited for long running Onyx clusters. Could definitely make a tool outside of Onyx core uses the log subscriber or queries on Onyx’s peer health endpoint to get that information, then do whatever with it.

@michaeldrogalis Thanks that's good to know