Fork me on GitHub
#onyx
<
2016-12-28
>
jasonbell13:12:34

Afternoon, I’ve just pushed a PR on the onyx-twitter plugin for the tracking of tweets. Not sure what your process was in terms of versioning so I’ve just added -SNAPSHOT on my version until you’d seen it.

nlessa14:12:52

Hi, is any plan to use the new Datomic client library in the onyx-datomic plugin? does this move make sense to onyx rationale?

michaeldrogalis15:12:01

@nlessa Yeah, we should definitely upgrade to at least the newest version of the dependency we’re already using - no reason to stay behind. I didn’t get a chance to look at the new client library. Is it more or a less a mirror of what already existed with lighter footprint?

michaeldrogalis15:12:51

@jasonbell Thanks, looks good to me. Can you drop the SNAPSHOT bit and I’ll merge it? Our release infrastructure will bump the versions automatically.

nlessa16:12:49

@michaeldrogalis Yes, its like that .It seems that would be very useful to the scenario of several onyx peers using datomic as input/output. But I am not sure I am not missing some "but".

michaeldrogalis16:12:28

@nlessa No, just haven’t had time to bump it. If you exclude the dep and substitute your own, is it binary compatible?

gardnervickers16:12:55

I believe you still need a “peer server” operating in a different address space, which the Datomic client library connects to. The intent behind the client library is to provide a way to use datomic from short-lived processes (think AWS Lambda). Unless you’re scaling your Onyx peers very frequently, you’re likely to benefit from using the old Datomic peer library with it’s in-process cache.

jasonbell16:12:55

@michaeldrogalis Thanks, just removed the SNAPSHOT and pushed again.

michaeldrogalis16:12:28

@jasonbell Thanks, merged. It’s out on version 0.9.15.1.

jasonbell17:12:09

Brilliant, thanks!

jasonbell17:12:11

Had the blog post ready and waiting 🙂

michaeldrogalis17:12:47

@jasonbell When you give the Twitter API a filter query, does it adjust its throughput to give you just as many tweets than if you didnt filter at all?

jasonbell17:12:20

Yes I believe so.

jasonbell17:12:42

You’ll get a lot more useful data at the same velocity.

jasonbell17:12:05

.sample and .filter are two very different API calls from what I remember.

michaeldrogalis17:12:16

I wish I knew about that ages ago, hah.

jasonbell17:12:09

Well you’ve now removed me ever needing SpringXD in this life 🙂

jasonbell17:12:34

That’s where the whole streaming workflow stuff started for me, at velocity.

michaeldrogalis17:12:24

Pretty interesting

jasonbell17:12:11

Well the flow predicate stuff means I can pipe out specific tracks into S3 bucket and so on.

michaeldrogalis17:12:46

Yeah, I was going to suggest that instead of using tracking, but if the throughput increases that’s better.

michaeldrogalis17:12:52

Most of the stream is garbage.

jasonbell17:12:33

The public stream is garbage

jasonbell17:12:45

Far better off using the filter track instead.

jasonbell17:12:17

Just have to be careful with the number of tracks you do as there’s a limit on the URL length to the API, 200 chars I think.

michaeldrogalis17:12:31

Ahh, good to know. That’s helpful 🙂

jasonbell17:12:46

@michaeldrogalis thanks for the support and input, have a grand day.

hugesandwich20:12:02

What's considered the most idiomatic way to access output task results in onyx-local-rt considering there are no output plugins? Is it best to do something like (get-in env [:tasks :my-out-task :outputs]) or is it better to just do something like use the lifecycle to fit your use case (ex: inject a core.async chan to write output to)? Ideally, I'd want to push results rather than poll for them if I can avoid it.

michaeldrogalis20:12:16

@hugesandwich Use get-in. Its just a data structure, so stick it in an atom and use a watch if you want to stream changes.

hugesandwich21:12:16

OK, I'm using get-in for now. I build my workflows dynamically so the extra complication is I then need to be sure to keep a map of which tasks are output tasks to grab their results accordingly. On the server, it's obviously much easier.

hugesandwich21:12:53

Anyway, thanks again.

michaeldrogalis21:12:10

You have access to :onyx/type inside (get-in env [:tasks …]), side-mapping shouldnt be needed if you dont mind recomputing that every time.

hugesandwich21:12:59

yeah that's why I'm keeping my own map because my task map can get big, so I don't want the overhead of recomputing

hugesandwich23:12:14

Sorry to bother, one more thing regarding onyx-local-rt. From the looks of the issues and scanning some of the code + branches on github, it looks like the triggers are broken for now? Is that true? Just realized I had a workflow with a window and timer trigger that's never getting called, but same workflow works in onyx.

michaeldrogalis23:12:04

@hugesandwich Triggers work fine, but Timers intentionally arent implemented because there’s no notion of time in local-rt.