@michaeldrogalis: hi i’ll try to stick around on here all day. likely catch you in the next couple hours or otherwise in your morning much later on

I am going through the flat-workflow example and I am getting a schema validation error from my version of this entry in the catalog https://github.com/onyx-platform/onyx-examples/blob/0.6.x/flat-workflow/src/flat_workflow/core.clj#L30

the schema is looking for :onyx/medium Keyword which is not provided in the example

I've slightly adapted the example (and will try the original now) but the changes I have made should not affect this

I see :onyx/medium set for input / output stuff, but what should it be when specifying :onyx/type :function ?

never mind, I figured it out, I was transcribing and accidentally supplied :onyx/ident in the catalog entry where I should have supplied :onyx/fn

@robert-stuttaford: Hi

@noisesmith: Cool, glad you got it figured out. How have things been going in general?

I got hung up on kafka (bad error reporting, the overly verbose log output actually made it harder to figure out what was going on) which was a precondition to using onyx in my project. I have that sorted out now and am this close to actually using onyx in anger.

Kafka's a toughie to get a handle on.

Are you guys using replication and partitioning, too?

No, not really. I don't think my tech lead realized what he was signing up for when he told me to use kafka, and he left for vacation after speccing that.

Hahah

"we wanted a nice snack, so we built a high-throughput state of the art cookie factory, and set up a garage with a fleet of delivery vehicles"

so now we have this nice infrastructure, if we ever actually needed it...

Sounds about right. Almost seems like you could have used something like Durable Queue?

yeah

devops guy is still pushing for replacing kafka with sqs

What do people do when they want to test locally with SQS?

there's an in-memory impl

which of course has even worse performance (not that sqs is even meant to be fast in the best of cases)

Oh good. A few years ago I used it and that didn't exist. Nice.

In our use case we could use something pretty low tech actually. The client doesn't expect the data to come back immediately (lag times in the order of minutes are fine), and is smart enough to retry if it doesn't get a result after some period of time also.

At least you're onto the fun part of the project now 😛

classic back-burner batch job stuff

heh, true

onyx is looking nice so far - I have yet another type of logging in my repls

(stack traces, printlns, java logger output, timbre output, and now color-coded stack traces sometimes as well)

Hah

it's actually kind of useful because I can tell by the shape / color of the text which part of the app is talking, lol

0.7.0 should be out in about 3 weeks. We're shipping a very large performance increase.

oh, nice

any particular places where the api is changing?

The general public API is the same. We switched the plugin multimethod interface to a protocol though

Pretty straightforward migration. We're trying hard not to break backwards compatibility, but now is the time to do it

are zookeeper warnings about end of stream exceptions when running onyx.api/submit-job and failing to unregister MBean when doing the various shutdowns normal, or a sign I have something set up wrong?

I see the full expected results of the workflow, and I can re-run it (getting the same warnings, and still seeing the correct result)

https://www.refheap.com/106829

That's funny. I'm debugging a Zookeeper end-of-stream exception for another project right now. No, it's a sign that something is wrong.

It's not necessarily fatal, but it's not good. Something is stopping its listener too soon.

OK, thanks

That might be a shutdown order thing in your case

Make sure you take down the peers, then the group, then the env - in that order.

@michaeldrogalis: also, as I track the bug further, I notice unnamed anonymous functions in my stack, would pull requests that add names go anonymous functions eg. (fn it-does-foo [] ....) likely be accepted as prs?

eg. changing this to (fn clean-up-function [] ...) https://github.com/onyx-platform/onyx/blob/master/src/onyx/log/zookeeper.clj#L69

@noisesmith: Sure

cool, because that makes the stack traces just a little bit easier to understand

Nothing worse than seeing blah$what$ahhh$

right

oh wait, that would be the wrong name for that function - it's actually doing something totally different

but general idea stands

Oh, I didnt click the link - for some reason I thought that was just an example. But right, the approach is welcomed

@noisesmith: Btw, I dont know what the level of secrecy on your code is, but if you can get permission to add me to your repo, that's how I typically help diagnose road blocks with companies if that ever happens.

Gist's are working just as well these days, though

Most of the problems are recurring misunderstandings of how something works, so we can usually guess what's wrong pretty quickly.

@michaeldrogalis:

how much do you know about BlockingQueue?

Eh, little.

What seems to be the problem again, if you don't mind?

that’s what Datomic’s tx-report-queue is. i’ve no prior experience with it, but basically, that’s what we need to turn into an onyx input

so, the async chan we currently use is getting in the way (i think)

we write stuff to it and sometimes we see activity immediately, and sometimes we have to write a whole lot of stuff before it does anything

like there’s a disconnect between what’s going in and what’s coming out of the async chan

it’s a (chat 100) at the moment

as you might imagine, this is making testing unreasonably difficult to do

Right, yeah

as we can’t tell what the issue is - are flow conds failing, or is it just stuck in the nether?

Have you fully removed Onyx from the equation? e.g. are you just connecting the BlockingQueue to a core.async channel and testing there?

yeah, so separately we have the same async chan without a buffer as a logging output to show us what the txor is writing

and that prints as it happens

i don’t actually know what the root cause of this non-determinism is, so my intuition tells me to simplify. it makes sense to me that we can forego the async chan and just have the tx-queue produce inputs directly

Yeah, I would definitely do that

ok. i have to go play Lego for 10 mins and then put my 5yr old to bed. and then i’ll be back

Okay

@michaeldrogalis: and i’m back

I'm looking at the examples (starting with the "flat workflow" example because I think that's the simplest), and I'm wondering - if I want to adapt it to keep running and listening to jobs, would the sensible thing be to create a go block or future that repeatedly calls take-segments!, or is there an existing abstraction that should be used?

@noisesmith: You can use the Leiningen template to get to that sort of state

It packages everything up into a "dev environment" that can be reused

@robert-stuttaford: Hey, sorry I need 30 minutes for lunch. Ill brb

OK, cool - lein new onyx appname I assume?

lein new onyx-app your-app-name

Brb

aha

@michaeldrogalis: happy eats

i found clojure.core/seque which apparently lets you treat a BlockingQueue like a lazy sequence

@robert-stuttaford: Thanks! I am back now

@michaeldrogalis: turns out all I needed was to remove the :done from the queue of tasks that the example queues up.

... I think

Oh, that's what you were asking - yes. Make sure you drain the output channel though, otherwise puts will eventually block

nice (that was the cause of the issue that led to "... I think" above, so I am good

everything hunky dory

Common gotcha

thanks again for the help

No problem.

nice, looks like I'll have proof of concept working on one peer by EOD, can start doing the multiple process / multiple machine thing tomorrow with any luck...

after kafka, this is very easy

haha

Hah

sorry, michaeldrogalis, i’m on a call with Matt, talking new tasks for Onyx

@robert-stuttaford: No problem, I'm here for a few more hours.

We're probably in the worst 2 possible time zones to catch each other, heh.

yeah timezones are -ing killing me. i’ve got a guy in Novosibirsk, one in Samara (both in Russia) and then Matt in Chicago

and two young kids who have their own ideas about how long things should take 😁

That's a handful. If you want to send me a detailed email, I can try to give some advise.

ok, just about done with Matt

Also if you run out of steam, I can take a crack at it myself. Np.

the ultiimate solution is probably going to be dead simple. i just suck at BlockingQueues

Let us hope so. 😄

(I think it will be)

ok. are you able to talk voice at all, michaeldrogalis ?

or are you at work?

Workin' 😞

no worries

i think i might have to ask your help here... i’m a little out of my depth on this one

the goal: every time a (.take tx-r-q) has a transaction result, put all the (unrolled) datoms from :tx-data on the queue to the next task

it’s the same as what would happen for any other queue consumer

the queue is the originator of the action, rather than some imperative code in the input plugin reading from a known source of data

so i guess it’d look more like https://github.com/onyx-platform/onyx-kafka/blob/0.6.x/src/onyx/plugin/kafka.clj than https://github.com/onyx-platform/onyx-datomic/blob/0.6.x/src/onyx/plugin/datomic.clj

My suggestion, and I've done this a few times as in the Kafka source, is to background a thread that reads from the queue and puts it onto a channel. Let Onyx work with the channel when it tries to do reads.

You're on the right track there.

You might even want to reference https://github.com/onyx-platform/onyx/blob/master/src/onyx/plugin/core_async.clj

ok, so that’s basically what we’re doing now. but it’s producing jarring results

should it be possible to see a 1:1 input:processing relationship?

as we give it stuff, event by event, it does stuff?

perhaps we’re just doing it wrong

Right, you should be seeing that

Ill pull in your code tonight and check it out myself

i’ll save you some effort and produce a clean repo for you to muck around in

this gives us the double benefit of an open-sourceable trapperkeeper + onyx + datomic starter kit for whoever wants one

That'd be super, thanks

retweet fodder 😁

Need some more blog posts from other folks. 😛

hehe

blogging. wish i had time for that sort of thing

Reading - who has time for that. I just close my eyes and bang on my keyboard for 8 hours

hahaha

you have a very smart forehead!

thanks, as ever, michaeldrogalis - i’ll get that sample prepared and sent to you asap

it’ll assume you have a transactor with mbrainz db in it

Cool, thanks! Hope to get you over this roadblock as quick as we can

Easy enough

catch up with you soon!

K. Take it easy man. See ya!

now i’m going to go murder my pillow

Enjoy 😛 Catch you