This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2017-06-28
Channels
- # beginners (33)
- # boot (58)
- # cider (21)
- # cljs-dev (197)
- # cljsrn (112)
- # clojure (136)
- # clojure-belgium (5)
- # clojure-dev (57)
- # clojure-greece (1)
- # clojure-italy (3)
- # clojure-russia (6)
- # clojure-spec (148)
- # clojure-uk (54)
- # clojurescript (29)
- # cursive (24)
- # datomic (36)
- # devops (4)
- # emacs (11)
- # figwheel (1)
- # graphql (18)
- # hoplon (6)
- # leiningen (2)
- # luminus (4)
- # off-topic (7)
- # om (4)
- # onyx (27)
- # precept (1)
- # protorepl (12)
- # quil (1)
- # re-frame (28)
- # reagent (10)
- # ring (9)
- # robots (1)
- # rum (2)
- # slack-help (5)
- # spacemacs (16)
- # sql (16)
- # untangled (16)
- # vim (3)
- # yada (2)
@eelke Is there a stacktrace for your
Writing to closed publication. java.lang.IllegalStateException: Publication has been closed
message? That’s where I’d be looking first.Secondly, have you tried the normal S3 plugin? Why write your own? Asking out of interest.
Second question: We had the following issue https://github.com/onyx-platform/onyx-amazon-s3/pull/5 with the 0.9.x version
First we wrote our own using amazonica. But we ran into several issues, amongst which com.amazonaws.ResetException
problems.
Question one: No stacktrace. But message is logged here: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/messaging/aeron/publication_manager.clj#L37
What’s the lifecycle on the peer, do they restart after failure? Just wondering if one of the task peers has died for some reason. They job keeps going but nothing happens. I’ve hit that before.
jasonbell: Looking at the dashboard all peers are still up after the :backpressure-on
message. On another note, in one instance when I restart the nodes with 100% CPU autoscaling creates new nodes and the job starts processing data again.
So how are you running all this? Within a container system of some form, Docker etc, or just on a machine? Anything that trips 100% CPU starts to concern me. What memory, message size and those kinds of things are you dealing with? I’m running out of ideas 🙂
@eelke onyx-s3 is out for 0.10 - https://clojars.org/org.onyxplatform/onyx-amazon-s3
I’m starting to create my first Onyx project using lein new onyx-app ...
and I’m seeing for the first time calls to onyx.job.add-task
rather than explicitly creating the data structures for the catalog and lifecycles as I’ve seen in the “learn-onyx” source code. Are there advantages to using add-task
over just having the structure explicit in the source file?
@niamu the add-task structure ensures you don't make mistakes when you need both catalogs and lifecycles together. It also makes it a bit nicer when parametrising tasks, though you can do that part in different ways
That makes sense. I’m looking at pitching Onyx to the company I work at and one of the advantages I saw in Onyx was the data structure of the pipeline being relatively easy to understand without needing to know Clojure. Just making sure there aren’t any sharp edges I’d encounter continuing down that path and avoiding add-task
.
@niamu Nope, purely there as development-time helpers.
The important thing is that no matter how you construct the job, it always needs to be an EDN data structure before it’s submitted.
I’ll continue with explicitly defining the data structure in my source files because I think that works better for my needs and limits the amount of Clojure knowledge needed to understand how the data pipeline works. Thanks!