Fork me on GitHub
#announcements
<
2019-09-02
>
schmee07:09:02

Announcing Daguerreo, a task execution library! > Daguerreo is a library to create workflows using tasks. It takes care of dependency resolution, parallellism, retries and timeouts and let’s you focus on your business logic. It shares the basic structure of other task execution libraries such as Airflow or Luigi, but unlike those it is meant to be embedded in your application rather than run as a standalone service. https://github.com/schmee/daguerreo/

thenonameguy11:09:35

Any plans to add checkpointing/handling process failures?

schmee12:09:11

Not sure what you mean by that, but I’m certainly open to feature requests/suggestions 🙂 If you’re interested, please open an issue describing what you’re looking for and maybe an example use-case! 🙏

schmee12:09:57

I guess checkpointing = restart the job from “in the middle” to avoid redoing a bunch of work?

thenonameguy12:09:59

yes, of course for that you will need a pluggable persistent storage for checkpoints

thenonameguy12:09:34

this looks like a single-process version of it, without any process-level execution guarantees (ie. if I kill the running DAG, I have to handle resuming it somehow)

schmee12:09:14

correct, because of that it is not really suited for really long-running workflows (e.g big data-crunching jobs)

schmee12:09:03

the sweet spot for Daguerreo in it’s current state is when you have lots of smaller tasks that interact with each other in complex ways, but the overall time of the job is “short”, for some definition of short

thenonameguy12:09:31

makes sense, beats a hardcoded AST for sure 🙂 thanks!

thenonameguy12:09:25

> correct, because of that it is not really suited for really long-running workflows (e.g big data-crunching jobs) but if this is intended then the comparison with Airflow and similar tools that fit this niche is unfair IMO

schmee12:09:12

the comparison is not meant to imply “Daguerreo matches the features of these libraries”, but “if you are familiar with these libraries, it will help you understand Daguerreo faster”. I’ll re-phrase it and add some more in-depth comparisons on how they differ, thanks for the feedback! 👍

leonoel08:09:05

I don't quite understand why did you choose not to rely on jvm thread interruption mechanism

leonoel08:09:26

if correctly implemented, blocking calls such as Thread/sleep support cancellation via Thread/interrupt, and polling is supported as well via Thread/isInterrupted

schmee18:09:14

@ the reason is partly that I don’t want to expose users to old-school Java threading details, and partly that there are lots of libraries that don’t handle interrupts properly anyway. So my choice was to make it explicit and force the user to handle it rather than relying on a potentially shaky, more implicit solution. The current solution can also be modified to handle interrupts if I ever want to go down that road (I think :D). So that the reasoning, whether it’s good or not I’m actually not sure myself 🙂

schmee18:09:44

I’m looking forward to playing with Missionary in an upcoming project btw, it looks really cool! 🙂

leonoel19:09:01

that makes sense, however I find useful to have a way to be actively notified of cancellation (in addition to passive polling), for instance by attaching a callback to the token

schmee19:09:02

agree, that’s something I overlooked in the initial release. Would an on-complete callback suffice you think? then you can check the completion status in the callback and handle it, or hook it up to something like Manifold can work with deffereds instead

leonoel19:09:33

a callback looks like the most flexible solution to me, you can hook up to anything supporting cancellation, including a nested job and even Thread/interrupt

leonoel19:09:07

BTW you can make it a non-breaking change if you implement it on the arity 1 of is-active? 😉

darwin14:09:44

Dirac v1.4.0 is out. Featuring initial Figwheel Main integration. Give it a spin: https://github.com/binaryage/dirac/tree/master/examples/figwheel-main