This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-03-24
Channels
- # admin-announcements (1)
- # aws (3)
- # beginners (52)
- # boot (150)
- # braid-chat (1)
- # braveandtrue (5)
- # bristol-clojurians (2)
- # cider (21)
- # cljs-dev (1)
- # cljsfiddle (1)
- # cljsjs (5)
- # cljsrn (4)
- # clojars (3)
- # clojure (236)
- # clojure-berlin (2)
- # clojure-czech (1)
- # clojure-madison (1)
- # clojure-russia (164)
- # clojure-sdn (1)
- # clojure-sg (2)
- # clojure-uk (64)
- # clojurescript (149)
- # core-async (31)
- # cursive (33)
- # datomic (2)
- # devcards (5)
- # funcool (3)
- # hoplon (142)
- # immutant (27)
- # juxt (7)
- # lein-figwheel (6)
- # liberator (6)
- # off-topic (4)
- # om (46)
- # onyx (26)
- # parinfer (5)
- # perun (56)
- # proton (6)
- # re-frame (19)
- # reagent (1)
- # remote-jobs (12)
- # ring-swagger (17)
- # slack-help (2)
- # spacemacs (11)
- # specter (1)
- # untangled (11)
- # yada (3)
https://github.com/onyx-platform/onyx-datomic it looks like functions read-datoms and read-index-range are suited for usage only with in-memory db, when read-log can be used with persistent db. And it so not obvious for me, i thought that this was supposed to be transparent?
@jarradhope: what makes you say that read-datoms and read-index-range can only be used with a mem db?
if change the database to “datomic:mem..” nothing actually will happen, process seems to hang
with read-datoms?
The tests for read-datoms uses an in memory DB. What does your onyx.log say?
sorry i mean the process hangs when I use the in-memory db with read-log
Right, you definitely can’t use in mem db with read-log because mem-dbs have no tx log
ah i see
Hi. I'm trying to configure workflow where onyx will read input datoms using onyx-datomic plugin. Is it possible to use read-datoms function with persistent db and avoid repeat read of the same datom after restarting job? Or maybe there is an example how it can be done, i will appriciate if someone help.
@rasom: That would be possible if we checkpointed to a global key, into ZooKeeper, like we do in read-log. Unfortunately that’s not a feature we currently support in read-datoms
@rasom: are you wanting to start a job at a specific place in the datomic log?
Or recover if your input crashes
@gardnervickers: The idea is to have a guaranty that input data will be read once.
Hmmm that might be an issue, as we (and all streaming platforms) need the ability to re-read from input if a machine crashes.
In order to guarantee at least once delivery
You can only achieve "exactly once" using a windowing task. Using a global checkpoint key would get you closer, but not to exactly once
@gardnervickers: thanks, understood… @lucasbradstreet: is there an example how it can be done with windowing task?
@rasom: You can use onyx-examples to see how windows work: https://github.com/onyx-platform/onyx-examples
Also refer to the docs: http://www.onyxplatform.org/docs/user-guide/latest/windowing.html
@michaeldrogalis: thanks. I still don’t understand how i can achieve "exactly once" using a windowing task
@rasom: Right, so exactly once side-effects are definitionally impossible to achieve. But you can get exactly once state updates, presuming that your state updates are idempotent. Consider a stream of messages and maintaining a running count. Onyx knows how to maintain a consistent count even if there's a machine/network failure, you don't need to think about it. And that's essentially what Windowing buys you.
You can look at the state management section of the docs for explanation of how it works under the hood.
@rasom: If you could share what you’re trying to do I think we could offer more insight
@michaeldrogalis, thanks, looks like I understood the idea