@lmergen Thanks! We’ll get it looked at asap - everyone in Seattle went outside all day. First day of sunshine in ages. 😛

@devth fixed and backported to 0.9.15.1

I am just stripping out the transaction functions for now.

@lucasbradstreet great, i see you fixed up my PR 🙂

for curiosity’s sake, where exactly is where the backpressure comes into play ?

as far as i can see, you basically have gotten rid of the separation between partitioning and the read step

In 0.10, a checkpoint barrier is pushed through the job, with each barrier initiating a checkpoint in each task

But you can emit quite a lot of the partitioning segments in that time

yes, i noticed that was happening — i emitted all the partitions, and the checkpointing didn’t “work”

Because a small amount of partitioning data will not fill a small buffer. But then these segments are exploded

ahh!

By the downstream task. So not very good backpressure or checkpointing

i see

So it was better to figure out how to partition it in the input task

Which I can if all peers agree on the min and max id

yes, i didn’t want to break backwards compatibility, but this is much better

Understood. Yeah, we kinda have to break in this case. Glad it'll be improved anyway

Thanks for the PR

i’ll be trying to get this to work with postgresql today — i noticed some oddities in passing the db configuration, i think the user gets lost somehow i’ll probably add a few more commits to this to fix that as well

Great. Would be happy to merge any more work that you can do on it

👍

Hi all! I got this error, but cant understand what can be, some tip?

ERROR [onyx.peer.peer-group-manager:235] - Error caught in PeerGroupManager loop. java.lang.AssertionError: Assert failed: (invariants/short-identifiers-correct %)

@lellis hmm. What version of onyx are you seeing this on?

We fixed an issue like this a while back

If you're on the latest version it should also print out the replica. It'd help a lot if you could give us the full stack trace and replica.

@lmergen @lucasbradstreet Really glad that we ditched the partition-keys task in onyx-sql as a two step process to do reading. That was always awkward and a waste of resources.