has there been any work / thinking on rate limiting? i found https://clojurians-log.clojureverse.org/onyx/2016-07-06.html – sounds like maybe something that was being considered.

@devth Rate limit in which sense? Reading data into Onyx?

right

probably doesn't make sense to rate limit on the output side

@devth We don’t have any official support on that. If you want to put a design proposal together I’d be happy to take a look at it.

ok, sounds good. just starting to think about it

@devth The problem I see with Onyx controlling rate limiting at the ingest versus the user just doing it in their output or sync function is that they end up doing the same thing, backpressuring to your input source.

Rate limiting is also specific to what you’re writing to. Different systems have different performance/cost profiles for either the # of items inserted or the # of transaction made.

Hey folks! I have a weird question about lazy sequences processing in onyx. Let’s imagine I have a task which returns a segment containing lazy seq:

{:numbers (for [i (range 10)] i)}

(reduce + (:numbers segment))

@jetmind The segment is sent over the network to the next peer, probably on another machine, so it’s realized.

That’s what I thought. The reason I asked is because I have a flow condition which reports exception and a task name it appeared in to sentry and I see a downstraem task name there, but stack trace indicates error inside the lazy seq in the upstream task.

@jetmind Can you rephrase that? I’m a bit confused to what you’re describing.

@michaeldrogalis Basically what I’m saying is that our error tracking system shows stack trace which indicates the error happened while realizing a lazy sequence (which is created in task1) and it seems as if this seq is realized in the task that follows the task1 and not in the task1 itself. I get the name of the task I’m sending to the error tracking system when exception happens from (-> event :onyx.core/task-map :onyx/name) in the post-transform hook.

@jetmind There’s one thing that might make this possible, but I’d have to check.

We’re using Nippy to serialize some types of data between networked machines

It wouldn’t surprise me if Nippy had some tolerance for freezing/thawing lazy seqs.

ok, I feel less insane now)

I know that sounds weird, but I also recently learned that it propagates metadata too. Heh.

Haha, yeah we’ll figure it out

cool, thanks for your help!

Wow, it does

:lazy-seq       (repeatedly 1000 rand)
 :lazy-seq-empty (map identity '())

That’s in it’s example README.

As far as I understand it’s still realizing the seq before serialization, there’s no magic, right?

(def frozen (nippy/freeze {:lazy (for [i (range 3)] (do (println i) i))}))
0
1
2

I don’t have time to dig deeper, but I can try to help some more later. That’s certainly odd behavior.

Sure, thanks! I think I’ll just realize the seq manually in the first task, but it would be great to understand why this is happening in the first place.

I depend on the task name to get some more data to report feedback to related systems and this behavior breaks things

Totally makes sense. If possible, can you open an issue with a minimal reproduction?

I know that’s a bit harder with these sorts of system, but if you can trim it down at all I’d greatly appreciate it.

Sure, I’ll try to trim it down later

Thanks 🙂

Hi all, can i use resume-point in onyx 0.9.15.0?

@lellis No, resume points are a 0.10 feature.

And with 0.10 i can't use onyx-http right @michaeldrogalis?

@lellis Right. We still need to figure out an appropriate way to deal with unreliable HTTP targets. Frequent non-200's will cause the message stream to make poor progress since the 0.10 architecture deals with failure by rewinding input streams.

If you can shift dealing with that concern into your application, it’s simple to bring this plugin up to 0.10 compatibility.

I got thus error when upgrade to "0.10.0.0-beta12"

Exception in thread "main" java.lang.NoSuchFieldException: builder, compiling:(onyx/storage/s3.clj:36:61)