This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2015-07-22
Channels
- # admin-announcements (22)
- # boot (109)
- # cider (119)
- # cljs-dev (19)
- # clojure (181)
- # clojure-berlin (3)
- # clojure-dev (10)
- # clojure-japan (10)
- # clojure-korea (2)
- # clojure-russia (105)
- # clojure-sg (1)
- # clojure-spain (2)
- # clojurescript (53)
- # code-reviews (2)
- # core-async (4)
- # datomic (2)
- # editors (21)
- # instaparse (5)
- # jobs (1)
- # ldnclj (12)
- # off-topic (2)
- # onyx (146)
@michaeldrogalis: hi i’ll try to stick around on here all day. likely catch you in the next couple hours or otherwise in your morning much later on
I am going through the flat-workflow example and I am getting a schema validation error from my version of this entry in the catalog https://github.com/onyx-platform/onyx-examples/blob/0.6.x/flat-workflow/src/flat_workflow/core.clj#L30
the schema is looking for :onyx/medium Keyword
which is not provided in the example
I've slightly adapted the example (and will try the original now) but the changes I have made should not affect this
I see :onyx/medium set for input / output stuff, but what should it be when specifying :onyx/type :function ?
never mind, I figured it out, I was transcribing and accidentally supplied :onyx/ident in the catalog entry where I should have supplied :onyx/fn
@noisesmith: Cool, glad you got it figured out. How have things been going in general?
I got hung up on kafka (bad error reporting, the overly verbose log output actually made it harder to figure out what was going on) which was a precondition to using onyx in my project. I have that sorted out now and am this close to actually using onyx in anger.
Kafka's a toughie to get a handle on.
Are you guys using replication and partitioning, too?
No, not really. I don't think my tech lead realized what he was signing up for when he told me to use kafka, and he left for vacation after speccing that.
"we wanted a nice snack, so we built a high-throughput state of the art cookie factory, and set up a garage with a fleet of delivery vehicles"
so now we have this nice infrastructure, if we ever actually needed it...
Sounds about right. Almost seems like you could have used something like Durable Queue?
devops guy is still pushing for replacing kafka with sqs
What do people do when they want to test locally with SQS?
there's an in-memory impl
which of course has even worse performance (not that sqs is even meant to be fast in the best of cases)
Oh good. A few years ago I used it and that didn't exist. Nice.
In our use case we could use something pretty low tech actually. The client doesn't expect the data to come back immediately (lag times in the order of minutes are fine), and is smart enough to retry if it doesn't get a result after some period of time also.
At least you're onto the fun part of the project now 😛
classic back-burner batch job stuff
heh, true
onyx is looking nice so far - I have yet another type of logging in my repls
(stack traces, printlns, java logger output, timbre output, and now color-coded stack traces sometimes as well)
it's actually kind of useful because I can tell by the shape / color of the text which part of the app is talking, lol
0.7.0 should be out in about 3 weeks. We're shipping a very large performance increase.
oh, nice
any particular places where the api is changing?
The general public API is the same. We switched the plugin multimethod interface to a protocol though
Pretty straightforward migration. We're trying hard not to break backwards compatibility, but now is the time to do it
are zookeeper warnings about end of stream exceptions when running onyx.api/submit-job
and failing to unregister MBean when doing the various shutdowns normal, or a sign I have something set up wrong?
I see the full expected results of the workflow, and I can re-run it (getting the same warnings, and still seeing the correct result)
That's funny. I'm debugging a Zookeeper end-of-stream exception for another project right now. No, it's a sign that something is wrong.
It's not necessarily fatal, but it's not good. Something is stopping its listener too soon.
OK, thanks
That might be a shutdown order thing in your case
Make sure you take down the peers, then the group, then the env - in that order.
@michaeldrogalis: also, as I track the bug further, I notice unnamed anonymous functions in my stack, would pull requests that add names go anonymous functions eg. (fn it-does-foo [] ....) likely be accepted as prs?
eg. changing this to (fn clean-up-function [] ...)
https://github.com/onyx-platform/onyx/blob/master/src/onyx/log/zookeeper.clj#L69
cool, because that makes the stack traces just a little bit easier to understand
Nothing worse than seeing blah$what$ahhh$
right
oh wait, that would be the wrong name for that function - it's actually doing something totally different
but general idea stands
Oh, I didnt click the link - for some reason I thought that was just an example. But right, the approach is welcomed
@noisesmith: Btw, I dont know what the level of secrecy on your code is, but if you can get permission to add me to your repo, that's how I typically help diagnose road blocks with companies if that ever happens.
Gist's are working just as well these days, though
Most of the problems are recurring misunderstandings of how something works, so we can usually guess what's wrong pretty quickly.
how much do you know about BlockingQueue?
Eh, little.
What seems to be the problem again, if you don't mind?
that’s what Datomic’s tx-report-queue is. i’ve no prior experience with it, but basically, that’s what we need to turn into an onyx input
so, the async chan we currently use is getting in the way (i think)
we write stuff to it and sometimes we see activity immediately, and sometimes we have to write a whole lot of stuff before it does anything
like there’s a disconnect between what’s going in and what’s coming out of the async chan
it’s a (chat 100) at the moment
as you might imagine, this is making testing unreasonably difficult to do
Right, yeah
as we can’t tell what the issue is - are flow conds failing, or is it just stuck in the nether?
Have you fully removed Onyx from the equation? e.g. are you just connecting the BlockingQueue to a core.async channel and testing there?
yeah, so separately we have the same async chan without a buffer as a logging output to show us what the txor is writing
and that prints as it happens
i don’t actually know what the root cause of this non-determinism is, so my intuition tells me to simplify. it makes sense to me that we can forego the async chan and just have the tx-queue produce inputs directly
Yeah, I would definitely do that
ok. i have to go play Lego for 10 mins and then put my 5yr old to bed. and then i’ll be back
@michaeldrogalis: and i’m back
I'm looking at the examples (starting with the "flat workflow" example because I think that's the simplest), and I'm wondering - if I want to adapt it to keep running and listening to jobs, would the sensible thing be to create a go block or future that repeatedly calls take-segments!
, or is there an existing abstraction that should be used?
@noisesmith: You can use the Leiningen template to get to that sort of state
It packages everything up into a "dev environment" that can be reused
@robert-stuttaford: Hey, sorry I need 30 minutes for lunch. Ill brb
OK, cool - lein new onyx appname
I assume?
lein new onyx-app your-app-name
@michaeldrogalis: happy eats
i found clojure.core/seque
which apparently lets you treat a BlockingQueue like a lazy sequence
@robert-stuttaford: Thanks! I am back now
@michaeldrogalis: turns out all I needed was to remove the :done
from the queue of tasks that the example queues up.
... I think
Oh, that's what you were asking - yes. Make sure you drain the output channel though, otherwise puts will eventually block
nice (that was the cause of the issue that led to "... I think" above, so I am good
everything hunky dory
Common gotcha
thanks again for the help
No problem.
nice, looks like I'll have proof of concept working on one peer by EOD, can start doing the multiple process / multiple machine thing tomorrow with any luck...
after kafka, this is very easy
sorry, michaeldrogalis, i’m on a call with Matt, talking new tasks for Onyx
@robert-stuttaford: No problem, I'm here for a few more hours.
We're probably in the worst 2 possible time zones to catch each other, heh.
yeah timezones are -ing killing me. i’ve got a guy in Novosibirsk, one in Samara (both in Russia) and then Matt in Chicago
and two young kids who have their own ideas about how long things should take 😁
That's a handful. If you want to send me a detailed email, I can try to give some advise.
ok, just about done with Matt
Also if you run out of steam, I can take a crack at it myself. Np.
the ultiimate solution is probably going to be dead simple. i just suck at BlockingQueues
Let us hope so. 😄
(I think it will be)
ok. are you able to talk voice at all, michaeldrogalis ?
or are you at work?
Workin' 😞
no worries
i think i might have to ask your help here... i’m a little out of my depth on this one
the goal: every time a (.take tx-r-q) has a transaction result, put all the (unrolled) datoms from :tx-data on the queue to the next task
it’s the same as what would happen for any other queue consumer
the queue is the originator of the action, rather than some imperative code in the input plugin reading from a known source of data
so i guess it’d look more like https://github.com/onyx-platform/onyx-kafka/blob/0.6.x/src/onyx/plugin/kafka.clj than https://github.com/onyx-platform/onyx-datomic/blob/0.6.x/src/onyx/plugin/datomic.clj
My suggestion, and I've done this a few times as in the Kafka source, is to background a thread that reads from the queue and puts it onto a channel. Let Onyx work with the channel when it tries to do reads.
You're on the right track there.
You might even want to reference https://github.com/onyx-platform/onyx/blob/master/src/onyx/plugin/core_async.clj
ok, so that’s basically what we’re doing now. but it’s producing jarring results
should it be possible to see a 1:1 input:processing relationship?
as we give it stuff, event by event, it does stuff?
perhaps we’re just doing it wrong
Right, you should be seeing that
Ill pull in your code tonight and check it out myself
i’ll save you some effort and produce a clean repo for you to muck around in
this gives us the double benefit of an open-sourceable trapperkeeper + onyx + datomic starter kit for whoever wants one
That'd be super, thanks
retweet fodder 😁
Need some more blog posts from other folks. 😛
blogging. wish i had time for that sort of thing
Reading - who has time for that. I just close my eyes and bang on my keyboard for 8 hours
you have a very smart forehead!
thanks, as ever, michaeldrogalis - i’ll get that sample prepared and sent to you asap
it’ll assume you have a transactor with mbrainz db in it
Cool, thanks! Hope to get you over this roadblock as quick as we can
Easy enough
catch up with you soon!
K. Take it easy man. See ya!
now i’m going to go murder my pillow
Enjoy 😛 Catch you