Fork me on GitHub
#onyx
<
2016-03-24
>
jarradhope13:03:39

https://github.com/onyx-platform/onyx-datomic it looks like functions read-datoms and read-index-range are suited for usage only with in-memory db, when read-log can be used with persistent db. And it so not obvious for me, i thought that this was supposed to be transparent?

lucasbradstreet13:03:38

@jarradhope: what makes you say that read-datoms and read-index-range can only be used with a mem db?

jarradhope14:03:59

if change the database to “datomic:mem..” nothing actually will happen, process seems to hang

lucasbradstreet14:03:45

with read-datoms?

gardnervickers14:03:00

The tests for read-datoms uses an in memory DB. What does your onyx.log say?

jarradhope14:03:00

sorry i mean the process hangs when I use the in-memory db with read-log

lucasbradstreet14:03:17

Right, you definitely can’t use in mem db with read-log because mem-dbs have no tx log

rasom15:03:45

Hi. I'm trying to configure workflow where onyx will read input datoms using onyx-datomic plugin. Is it possible to use read-datoms function with persistent db and avoid repeat read of the same datom after restarting job? Or maybe there is an example how it can be done, i will appriciate if someone help.

lucasbradstreet15:03:56

@rasom: That would be possible if we checkpointed to a global key, into ZooKeeper, like we do in read-log. Unfortunately that’s not a feature we currently support in read-datoms

rasom15:03:57

So the only option to avoid repeat read is to use read-log?

rasom15:03:17

can this be done via other onyx plugin? onyx-kafka, onyx-redis?

gardnervickers15:03:54

@rasom: are you wanting to start a job at a specific place in the datomic log?

gardnervickers15:03:21

Or recover if your input crashes

rasom15:03:23

@gardnervickers: The idea is to have a guaranty that input data will be read once.

gardnervickers15:03:33

Hmmm that might be an issue, as we (and all streaming platforms) need the ability to re-read from input if a machine crashes.

gardnervickers15:03:01

In order to guarantee at least once delivery

lucasbradstreet15:03:51

You can only achieve "exactly once" using a windowing task. Using a global checkpoint key would get you closer, but not to exactly once

rasom15:03:46

@gardnervickers: thanks, understood… @lucasbradstreet: is there an example how it can be done with windowing task?

michaeldrogalis15:03:39

@rasom: You can use onyx-examples to see how windows work: https://github.com/onyx-platform/onyx-examples

rasom16:03:09

@michaeldrogalis: thanks. I still don’t understand how i can achieve "exactly once" using a windowing task

michaeldrogalis16:03:38

@rasom: Right, so exactly once side-effects are definitionally impossible to achieve. But you can get exactly once state updates, presuming that your state updates are idempotent. Consider a stream of messages and maintaining a running count. Onyx knows how to maintain a consistent count even if there's a machine/network failure, you don't need to think about it. And that's essentially what Windowing buys you.

michaeldrogalis16:03:56

You can look at the state management section of the docs for explanation of how it works under the hood.

gardnervickers16:03:54

@rasom: If you could share what you’re trying to do I think we could offer more insight

rasom16:03:18

@michaeldrogalis, thanks, looks like I understood the idea