Fork me on GitHub

Afternoon, I’ve just pushed a PR on the onyx-twitter plugin for the tracking of tweets. Not sure what your process was in terms of versioning so I’ve just added -SNAPSHOT on my version until you’d seen it.


Hi, is any plan to use the new Datomic client library in the onyx-datomic plugin? does this move make sense to onyx rationale?


@nlessa Yeah, we should definitely upgrade to at least the newest version of the dependency we’re already using - no reason to stay behind. I didn’t get a chance to look at the new client library. Is it more or a less a mirror of what already existed with lighter footprint?


@jasonbell Thanks, looks good to me. Can you drop the SNAPSHOT bit and I’ll merge it? Our release infrastructure will bump the versions automatically.


@michaeldrogalis Yes, its like that .It seems that would be very useful to the scenario of several onyx peers using datomic as input/output. But I am not sure I am not missing some "but".


@nlessa No, just haven’t had time to bump it. If you exclude the dep and substitute your own, is it binary compatible?


I believe you still need a “peer server” operating in a different address space, which the Datomic client library connects to. The intent behind the client library is to provide a way to use datomic from short-lived processes (think AWS Lambda). Unless you’re scaling your Onyx peers very frequently, you’re likely to benefit from using the old Datomic peer library with it’s in-process cache.


@michaeldrogalis Thanks, just removed the SNAPSHOT and pushed again.


@jasonbell Thanks, merged. It’s out on version


Brilliant, thanks!


Had the blog post ready and waiting 🙂


@jasonbell When you give the Twitter API a filter query, does it adjust its throughput to give you just as many tweets than if you didnt filter at all?


Yes I believe so.


You’ll get a lot more useful data at the same velocity.


.sample and .filter are two very different API calls from what I remember.


I wish I knew about that ages ago, hah.


Well you’ve now removed me ever needing SpringXD in this life 🙂


That’s where the whole streaming workflow stuff started for me, at velocity.


Pretty interesting


Well the flow predicate stuff means I can pipe out specific tracks into S3 bucket and so on.


Yeah, I was going to suggest that instead of using tracking, but if the throughput increases that’s better.


Most of the stream is garbage.


The public stream is garbage


Far better off using the filter track instead.


Just have to be careful with the number of tracks you do as there’s a limit on the URL length to the API, 200 chars I think.


Ahh, good to know. That’s helpful 🙂


@michaeldrogalis thanks for the support and input, have a grand day.


What's considered the most idiomatic way to access output task results in onyx-local-rt considering there are no output plugins? Is it best to do something like (get-in env [:tasks :my-out-task :outputs]) or is it better to just do something like use the lifecycle to fit your use case (ex: inject a core.async chan to write output to)? Ideally, I'd want to push results rather than poll for them if I can avoid it.


@hugesandwich Use get-in. Its just a data structure, so stick it in an atom and use a watch if you want to stream changes.


OK, I'm using get-in for now. I build my workflows dynamically so the extra complication is I then need to be sure to keep a map of which tasks are output tasks to grab their results accordingly. On the server, it's obviously much easier.


Anyway, thanks again.


You have access to :onyx/type inside (get-in env [:tasks …]), side-mapping shouldnt be needed if you dont mind recomputing that every time.


yeah that's why I'm keeping my own map because my task map can get big, so I don't want the overhead of recomputing


Sorry to bother, one more thing regarding onyx-local-rt. From the looks of the issues and scanning some of the code + branches on github, it looks like the triggers are broken for now? Is that true? Just realized I had a workflow with a window and timer trigger that's never getting called, but same workflow works in onyx.


@hugesandwich Triggers work fine, but Timers intentionally arent implemented because there’s no notion of time in local-rt.