@michaeldrogalis @lucasbradstreet https://github.com/onyx-platform/onyx/pull/632

Great, thank you!

That looks great!

It is still rough around the edges, I think the user-guide should have more “flow” - but that’s just like my opinion 🙂 I’ll think more about it this weekend and come back with suggestions

first - hdfs-write-plugin 🙂

I'm sure you're right

hello friends 🙂

@lucasbradstreet: how're ya goin

G’day. Doing good. How’s it going over in Cognician land?

super good 🙂

it's "save on AWS costs" day over here

we were wondering if we could revisit the idea of possibly sticking http://docs.caudate.me/hara/hara-io-scheduler.html inside a task somehow so that we can ensure that only one instance starts via onyx/max-peers

because doing so allows us to avoid running a separate instance for it

I think it’ll be a bit awkward, because you’ll have to have it associated to a task

I assume you’re not just running an extra JVM because you need it to fail over to a different machine?

well, our onyx instances are homogenous

we could run another jvm on one of them, but then we need a good mechanism for choosing an instance, and dealing with failure modes

onyx already solves all those problems really well

via :onyx/max-peers 1

yeah, I figured that’s what you were going for.

the downside is that you’ll need it wired up to a task in your workflow, and I assume that task won’t send or receive anything (unless you piggy back it on one of your other tasks)

yeah. i guess its only input is the clock on the wall

ok. thanks. food for thought!

actually, i guess we could bring it in to the codebase and find some way to only run it on one of the instances, by inspecting the relationship of the instance to the auto-scaling group, or something

I have some other ideas, like following the log and only running if the replica says that you’re the host with the lowest IP, or something similarly determinstic

that would save the cost at least, even if it doesn't solve the failure modes

mmm

oh, that's interesting

That’s essentially the strategy I’m using to do some peer election in other ways

you’d have to code it up right though

so all instances would run the scheduler, but only the instance with the lowest IP would allow scheduled work to occur?

you’d probably be better off just using ZooKeeper plain and doing leader election via a Curator recipe

right

that solves instances going up and down

yeah, if one of them goes down and is no longer in the replica, the other peers may have to spring into action

It’s probably a bit hacky vs just writing up some leader election code with ZooKeeper though

i wonder if there's a clojure friendly scheduler that can track its state via ZK

cos we literally just need something to run exactly once every hour

i know, i'll delegate it! 🙂

thanks for the sanity check, Lucas

i hope things are going swimmingly for you and the team?

Everything is going great, thanks 😄

trying to get all my immigration stuff sorted out so I can join everyone in the US

awesome!

@vijaykiran: Fantastic work. My favorite part about having a single-page user guide is being able to use Ctrl+F again 😛

I'll review and get it merged tomorrow, along with those tutorials from earlier. I think the only thing left is to write a small script to automatically build that HTML file, which should only be 4-5 lines 🙂 I can take that piece.

@michaeldrogalis: btw, there's a asciidoctor-stylesheet-factory that you might want to check out for an easily tweaked stylesheet

@dominicm: Oh yeah, good idea. I played around with that a while back. There was some snazzy looking stuff in there.

On a somewhat-related note, I just managed to get the stylesheet factory to compile with libsass and foundation 5. That would open up space for it to be part of a boot compilation system.

Cool 🙂

will it be easier to work on my workflow then try to put it into production or is it better to start with a simple workflow put that into production then iterate on it?