Fork me on GitHub
#onyx
<
2017-05-17
>
theblackbox09:05:57

Morning all, I'm working on an issue that has been safe to ignore but troubling me nonetheless - implementing a "buffer" for DB writes. My workflow is effectively "analyse msg, attach side-effects, assert side-effects, reason about side-effects, save side-effects" and it's the saving that has an issue: the possibility that a message creates the same side-effect. So I'm trying to implement a "buffer" that will allow the first msg through, whilst aggregating subsequent messages until such time that the system is working on an up to date DB entry so that it no longer creates side-effects but modifies them whereupon I would release the aggregated modifier. (I hope this makes sense)

theblackbox09:05:43

I'm just looking for some direction on how I might go about doing this... in other situations I "prime" the DB so that all fields exist and every side-effect is a modifier, but that doesn't seem correct here as it will quickly bloat. So instead I'm attempting to use a session window with a segment trigger to allow the first msg through while aggregating the others. This seems like a good way to tackle it, but I'm open to suggestions from more experienced devs on this subject. A part of me wonders if this is the perfect situation for flow-conditions? I went with window/trigger because of the need to accumulate state.

lmergen10:05:04

@theblackbox what database are you using ?

lmergen10:05:25

it sounds like you need an upsert operation ? or am i thinking too simplistic ?

theblackbox10:05:25

I'm using mongo

theblackbox10:05:12

and you are totally correct, excepting the async nature means I can't guarantee the side-effect is going to be an upsert

theblackbox10:05:57

that's what I mean by buffering until a segment is a modifier then simply aggregating the buffered creates into the modify

theblackbox10:05:45

I think I'm on the right path with window/trigger/state to getting a working solution, then I think I need to improve it by utilising a persistent queue - which I have available in the guise of Kafka

lmergen10:05:31

right, what you can also do is version your data and try to do a 'compare and swap' operation when storing it -- if the data changed in the meantime, you can redo the calculation

lmergen10:05:08

this is a form op optimistic locking, though, and if your chance of having conflicts is too high, it will not scale

theblackbox10:05:42

cheers, appreciate the advice

devth14:05:12

just got the Unfreezable type: class clojure.lang.Delay error when starting a job in a fresh environment / deploy. found relevant slack logs: https://clojurians-log.clojureverse.org/onyx/2017-01-04.html https://clojurians-log.clojureverse.org/datomic/2016-10-04.html onyx "0.9.15" onyx-datomic "0.9.15.0" datomic-pro "0.9.5561" restarted the job several times to see if it consistently repros - it does. the value is:

#object[clojure.lang.Delay 0x5bb71dc7 {:status :pending, :val nil}]
I didn't see any resolution on this issue in the Slack logs. should onyx-datomic workaround it by checking if tx-range returned a delay? this same job is running successfully in another staging env against a separate datomic db that has nearly identical data. this is the first time i've seen that error.

gardnervickers14:05:15

Does the problem persist across JVM restarts? I wonder if it's something particular with your dataset that can be used to help diagnose the problem with the Datomic team.

devth14:05:26

one sec, restarting my jvm

devth14:05:30

yeah it repro'd on a fresh jvm

devth14:05:53

wait - different error. couldn't communicate with datomic this time

devth14:05:35

Error communicating with HOST 0.0.0.0 or ALT_HOST datomic on PORT 4334 – i see this intermittently. trying again

devth14:05:37

ok confirmed the Delay issue repro'd

devth14:05:46

Unfreezable type: class clojure.lang.Delay

devth14:05:22

this is a fresh datomic db with schema installed and about 500 generated test entities.

devth14:05:37

i'll check in #datomic to see if they looked into it at all

devth14:05:21

would be useful to have some logging in onyx-datomic to verify tx-range is for certain returning the Delay

gardnervickers14:05:35

That's really good that you found a repro, I'm sure they'll be able to assist further.

devth14:05:48

what's the best way to view checkpoint data for a job? not seeing it in onyx-dashboard

gardnervickers14:05:08

I don't think the dashboard provides that yet. Are you trying to get at the actual checkpoint state or metrics on checkpointing.

devth14:05:26

actual state - which datomic tx was processed last

michaeldrogalis15:05:29

@devth If the Datomic team doesn’t have an update, it’s probably a good idea for us to work around it in onyx-datomic. Why that’s happening is still a mystery to me.

devth15:05:46

i'll post back if/when i hear anything.

michaeldrogalis19:05:01

Been doing a lot of design work on log based architectures the last few months. Not directly related to Onyx, but hopefully useful for this crowd: http://pyroclast.io/blog/2017/05/17/the-future-of-event-stream-processing.html

Travis19:05:09

nice, will read shortly

michaeldrogalis19:05:16

Thanks. Hope some of the insight is fresh. 🙂

lucasbradstreet19:05:16

If you’re also willing to vote it up on https://news.ycombinator.com/newest it’d be appreciated 🙂

nha20:05:02

Nice writeup, it seems we are increasingly ditching b-trees for logs. However it does not even hint at possible issues/how to resolve things like read-after-write, monotonicity etc. I would love to know what you guys think 🙂

michaeldrogalis21:05:41

@nha Only so much room in one post. 🙂 Future material.

michaeldrogalis21:05:34

Though, important to note that in the context of event steaming, I’m not trying to say that materialized views are replacements for any sort of database. In the full landscape of data storage engines, this is just one narrow case.