OMG it was stuck because out-chan is an atom! Why did it not throw an exception there?! 😞

Hmm, it definitely should have thrown an exception. Weird

i've been noticing as well that some plugin exceptions get 'eaten' by onyx sometimes without providing output

never really was able to figure the details out

Hmm. Not good

i'll do a deeper dive next time i encounter such a situation

I’ll try to reproduce the out chan issue tomorrow. If I can reproduce it, it should be easy to fix

I'll check if it's an Onyx or a core.async trouble

yeah seems like an Onyx bug

if I catch and print exception it propagates

I mean it occurs 🙂

yes, this is what i noticed as well

okay I'm almost done with onyx-http! Retry works, happy case works, the only thing which is not working yet is that exception from async-exception-fn is not propagated. This is because write-batch is not called and I'm not sure how to force it.

last in-flight-writes value I see is 1, so it obviously is not finished yet

completed? is called constantly though

I'll check it out -- that's a nasty bug. 😕

@lmergen @asolovyov At what stage of the lifecycle is the exception being squashed? Im having trouble reproducing it

it's been a long time since i worked on onyx-sql

but i believe it was during write-batch

Hmm. Ill try with a different plugin.

i know that i do change logging config a bit, perhaps it’s related to that ?

i’ll make sure to better inspect it next time i encounter it

Hi there, I’m new to Onyx and am running into an issue when trying to add S3 checkpointing to my job. I configured my peer config with the s3.storage option and all the subsequent configuration options (auth-type, bucket, region, etc) and it looks like my Onyx job can talk to the bucket I specified, but I am getting a 403 - Access Denied response. Has anyone encountered that before?

For context, the job is running in an AWS EC2 instance with an appropriate role (S3 full access) and the S3 bucket has granted permissions to that role to perform all S3 operations.

Okay, gonna keep trying to track it down anyway. These bugs are really annoying when they crop up

@forrest.thomas Hmm. It sounds like you don’t have the right permissions setup still, but I’m not sure what could be wrong.

i thought so as well, but when I use the AWS CLI from that EC2 I can read/write to the bucket

normally, I would look at the object ACL to see what was happening, but I don’t think I can do that in this case since Onyx is the creator of the checkpoint and uploads it from in-memory (as far as i can tell)

Is Onyx running in a container? Maybe it's not seeing the same AWS keys as your CLI process?

it’s running on the host

I also have an encryption policy set on the bucket that requires aes256. I set that configuration option as well. is it possible that is getting lost somehow?

what do you get if you try to access the file using the aws cli on the host ?

(or rather, try to touch a file)

I doubt it, but you may want to try using a fresh bucket with no encryption just to check it out

i can access the bucket and all of its contents using the AWS CLI

I think it’s encryption’s fault

I see the config option but I don’t think we’ve actually implemented it (looking at the code now)

I stand corrected 😛

ah

If you can test whether it works without encryption, I’m happy to add it today.

That was an oversight.

sure, let me try

one sec

yep

works fine now

All of the code is already there, we just don’t pass the setting through.

thats the one

OK cool, I’ll have a fix shortly.

thats awesome! thank you SO much!

I forgot to mention. I’m using Onxy 10, will this fix be for that version or just 11+?

I’d like to make it against 0.12, which we should be releasing today, but you should be able to upgrade to 0.12 pretty seamlessly.

There are some breaking changes in the last two versions, but it’s not too bad.

gotcha

OK, well, I guess I needed to upgrade eventually. 😛

Yeah

heh

thanks again!

No worries. I’ll let you know when there’s something to try.

:thumbsup:

Just wondering if anyone has any methods/best practices for doing blue/green or just general job upgrading methods

@lucasbradstreet I'm afraid I need more of your help to finish onyx-http 🙂 I have no idea why it doesn't call write-batches after some time even though completed? returns false...

I'm going to leave right now but any pointers are really welcome, I would be happy to finish it tomorrow 🙂

We're updating to Onyx 0.12.0 in anticipation of the S3 encryption fix. I've made the appropriate fixes for the breaking changes (we've been running 0.10.0), and we're testing against 0.12.0-alpha4. It appears that some window triggers are firing twice (I'm seeing two copies of the same state getting emitted), which is breaking a lot of our tests. When I try 0.11.0.1 it all works fine. Seems like a bug in 12?

What trigger type are you using?

This could be due to the new support for watermarks causing a new event-type “:watermark” which is not being handled correctly by whatever trigger is being used.

That one is a segment trigger

And 0.11 is ok. Hmm, I’ll check it out shortly. Thanks

Ok, great. Thanks!

@fellows could you please give 0.12.0-20171108.231118-16 a go?

will do

Looks like that has the same problem.

Are you using :trigger/fire-all-extents? true?

Also, could you check from the trigger firing whether it’s firing on :job-completed?

Check the event-type of the state-event. That will fire when the job completes, which is probably happening in your tests.

If you could also check the :extents key in the state-event from your sync or emit, that would help.

Definitely not using fire-all-extents?.

Did you happen to drop a :trigger/refinement :discarding from the trigger, but you didn’t add :trigger/post-evictor in?

Is the state-event available inside the event in the trigger/emit function?

No, we dropped an accumulating refinement.

ok, nothing changed there then

I tried both types of post-evictor, actually, and neither had an obvious effect.

state-event is the fourth argument to trigger/sync and trigger/emit

Ah, sorry, I've been calling that window-data.

I'll take a look.

no worries. Yeah, if you can get me more info about the scenario of the fires that’d help

Bingo. The first time it's :new-segment, the second time it's :job-completed

:extents is (1) the first time and nil the second

Cool, we did a better job of sealing on job-completed now. We do it like that because maybe you have a trigger set to fire on every 10 elements, but then you had one more element be added and you still want it to flush when you complete the job.

So that will trigger even if it's going to emit a state that's identical to the previous one? Is there a way to turn that off, or do I need to specifically check for the event-type every time?

There’s no way for it to know what the previous state was, otherwise the memory consumption would increase by default. You should ignore the job completed event, or dedupe yourself, or use an evictor and combine your outputs as they’re evicted/synced