@eelke Is there a stacktrace for your

Writing to closed publication. java.lang.IllegalStateException: Publication has been closed

Secondly, have you tried the normal S3 plugin? Why write your own? Asking out of interest.

Second question: We had the following issue https://github.com/onyx-platform/onyx-amazon-s3/pull/5 with the 0.9.x version

And the 0.10.x onyx version hasn't gone alpha yet

First we wrote our own using amazonica. But we ran into several issues, amongst which com.amazonaws.ResetException problems.

I can send you the code

Question one: No stacktrace. But message is logged here: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/messaging/aeron/publication_manager.clj#L37

What’s the lifecycle on the peer, do they restart after failure? Just wondering if one of the task peers has died for some reason. They job keeps going but nothing happens. I’ve hit that before.

@eelke onyx-s3 is out for 0.10 - https://clojars.org/org.onyxplatform/onyx-amazon-s3

I’m starting to create my first Onyx project using lein new onyx-app ... and I’m seeing for the first time calls to onyx.job.add-task rather than explicitly creating the data structures for the catalog and lifecycles as I’ve seen in the “learn-onyx” source code. Are there advantages to using add-task over just having the structure explicit in the source file?

@niamu the add-task structure ensures you don't make mistakes when you need both catalogs and lifecycles together. It also makes it a bit nicer when parametrising tasks, though you can do that part in different ways

That makes sense. I’m looking at pitching Onyx to the company I work at and one of the advantages I saw in Onyx was the data structure of the pipeline being relatively easy to understand without needing to know Clojure. Just making sure there aren’t any sharp edges I’d encounter continuing down that path and avoiding add-task.

@niamu Nope, purely there as development-time helpers.

The important thing is that no matter how you construct the job, it always needs to be an EDN data structure before it’s submitted.

I’ll continue with explicitly defining the data structure in my source files because I think that works better for my needs and limits the amount of Clojure knowledge needed to understand how the data pipeline works. Thanks!

Sure!