This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-12-12
Channels
- # admin-announcements (1)
- # bangalore-clj (13)
- # beginners (149)
- # boot (123)
- # cider (7)
- # clojure (167)
- # clojure-brasil (3)
- # clojure-greece (1)
- # clojure-korea (2)
- # clojure-new-zealand (2)
- # clojure-russia (70)
- # clojure-sanfrancisco (3)
- # clojure-spec (84)
- # clojure-uk (36)
- # clojurescript (300)
- # code-reviews (242)
- # community-development (34)
- # core-async (4)
- # css (1)
- # cursive (37)
- # datascript (1)
- # datomic (20)
- # defnpodcast (1)
- # dirac (15)
- # events (7)
- # garden (12)
- # hoplon (100)
- # lein-figwheel (11)
- # off-topic (2)
- # om (69)
- # om-next (3)
- # onyx (86)
- # planck (14)
- # proton (4)
- # protorepl (1)
- # quil (2)
- # re-frame (53)
- # rum (3)
- # untangled (1)
- # vim (50)
1. lisp way of writing (c (b (a))) requires more eye tracking than (-> a b c). What do you think?
@mariusz_jachimowicz without the threading macro i wouldn't have used clojure
2. also formatting code in blocks way could also help code readability
3. @lucasbradstreet @michaeldrogalis what do you think about using this formatting styles? I could do that in many places if it is ok for you.
4. @lucasbradstreet @michaeldrogalis I see that often there is component that holds channel inside that is used by other components - for instance group-ch
is used by few components. It could be good to introduce Channels component that holds only common used channels between components.
It could lead into -> loose coupling -> easier testing components.
Hi all! Im trying build my production onyx system, and now im have these log error when submit job. Any tips?
16-12-12 13:34:55 MBPdeFrederico INFO [onyx.log.zookeeper:97] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-12-12 13:34:56 MBPdeFrederico INFO [onyx.log.zookeeper:126] - Stopping ZooKeeper client connection
16-12-12 13:34:56 MBPdeFrederico DEBUG [formiguinhas-service.onyx.core:77] - Successfully submitted job: 43484d37-20fd-490b-9811-44307945e66f
16-12-12 13:34:56 MBPdeFrederico DEBUG [formiguinhas-service.onyx.core:78] - Blocking on job completion...
16-12-12 13:34:56 MBPdeFrederico INFO [onyx.log.zookeeper:97] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-12-12 13:34:56 MBPdeFrederico INFO [onyx.log.zookeeper:97] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-12-12 13:34:56 MBPdeFrederico WARN [onyx.log.zookeeper:206] -
java.lang.Thread.run Thread.java: 745
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 617
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1142
...
clojure.core.async/thread-call/fn async.clj: 434
onyx.log.zookeeper/eval39151/fn/fn zookeeper.clj: 235
onyx.log.zookeeper/find-log-parameters zookeeper.clj: 203
onyx.log.zookeeper/find-log-parameters/fn zookeeper.clj: 204
...
onyx.log.zookeeper/eval39483/fn zookeeper.clj: 550
onyx.monitoring.measurements/measure-latency measurements.clj: 11
onyx.log.zookeeper/eval39483/fn/fn zookeeper.clj: 551
onyx.log.zookeeper/clean-up-broken-connections zookeeper.clj: 77
onyx.log.zookeeper/eval39483/fn/fn/fn zookeeper.clj: 554
onyx.log.curator/data curator.clj: 126
org.apache.curator.framework.imps.GetDataBuilderImpl$2.forPath GetDataBuilderImpl.java: 138
org.apache.curator.framework.imps.GetDataBuilderImpl$2.forPath GetDataBuilderImpl.java: 142
org.apache.curator.framework.imps.GetDataBuilderImpl.forPath GetDataBuilderImpl.java: 279
org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground GetDataBuilderImpl.java: 287
org.apache.curator.RetryLoop.callWithRetry RetryLoop.java: 107
org.apache.curator.framework.imps.GetDataBuilderImpl$4.call GetDataBuilderImpl.java: 291
org.apache.curator.framework.imps.GetDataBuilderImpl$4.call GetDataBuilderImpl.java: 302
org.apache.zookeeper.ZooKeeper.getData ZooKeeper.java: 1155
org.apache.zookeeper.KeeperException.create KeeperException.java: 51
org.apache.zookeeper.KeeperException.create KeeperException.java: 111
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /onyx/1/log-parameters/log-parameters
code: -101
path: "/onyx/1/log-parameters/log-parameters"
16-12-12 13:34:56 MBPdeFrederico WARN [onyx.log.zookeeper:207] - Log parameters couldn't be discovered. Backing off 500ms and trying again…
`Is the onyx-local-rt behavior defined for calling new-segment from withing tick or drain (e.g from a sync function that knows the runtime itself)
I'm starting to need event sourcing for my CQRS simulation in order to simulate emitting commands in response to events 😱
@mariusz_jachimowicz I don't really like that use of the threading macro. Nor of aligning arguments in that style:
- It's near impossible to get truly aesthetic alignment looking right everywhere
- and it's a pain to set up in every editor
- (get-in x [:foo :bar])
is more idiomatic than (-> x :foo :bar)
Does someone have an example of how I can use onyx-seq
to wrap a lazy seq as input? I'm running into issues because my current setup keeps trying to call .close
and things like that on a nil
reader instance. Since I'm just emitting a basic lazy seq, I don't really have a reader
@stephenmhopper: I assume you're adapting the plugin example code but don't want to use a reader? In that case you should drop the .close
call if there's nothing to actually close.
I did that
But it's still throwing this exception: CompilerException java.lang.IllegalArgumentException: No implementation of method: :close! of protocol: #'clojure.core.async.impl.protocols/Channel found for class: nil
What are you using for an output?
{:onyx/name :out
:onyx/plugin :onyx.plugin.core-async/output
:onyx/medium :core.async
:onyx/max-peers 1
:onyx/type :output
:onyx/batch-size 20}
Are the lifecycle calls for core async being called to inject the channel for your output?
@mariusz_jachimowicz I’m not interested in changing the coding style that Onyx uses, the primary reasoning being diff busting.
For the same reason that Clojure discourages style & whitespace changes - it’s significantly harder to backtrack through history to discover which patch caused a problem.
Re, point number 4 - the vast majority of channels have been removed from Onyx’s implementation in the next release.
The threading macro accessor style is limited to elements that implement IFn, right?
@yonatanel It’s not defined per say, but I’ve given it some thought. It’s a bit tricky because you need to stick segments directly into the inbox of the env, and that env would then need to be contained within an atom to make it work.
@stephenmhopper You can just implement your own lifecycles for onyx-seq if you don’t want to use something that’s Closable. Their actual lifecycles are quite small. Or you might be able to get away with not submitting the after-task-stop lifecycle to the job and keep the others
@yonatanel That being said, I don’t think there’s anything wrong with what you’re doing. It shouldn’t cause bad behavior.
@michaeldrogalis It's too weird to call a function that returns a new state within another one like that. I've decided to request all state changes to anything in my simulation though a channel instead.
@yonatanel That seems easier to think about, yes.
I thought om.next transact! could do it but its changes are immediate and doing nested transacts creates states that I don't want to reason about.
We are fast forwarding to deploy onyx in production, so that simulation development will be slowed down 😰
@yonatanel Seems positive overall? 😛
The whole thing is too complicated to just quickly throw a system together, but I've researched options and it's the best I found so...
I mean, I'm kinda glad it doesn't just do everything for you until one day you realize you're missing some guarantees or that the memory usage is unbounded or something like that.
Well, I figured out my issue from earlier. As it turns out, I was using the onyx-seq plugin definition, but was trying to use :onyx.plugin.core-async/reader-calls
instead of :onyx.plugin.seq/reader-calls
@stephenmhopper Nice, good find
Yeah, if I find some time and manage to make a working app at some point in the future, I'm going to make a project similar to learn-onyx
except everything will be broken and throwing errors and you have to figure out what's wrong
@stephenmhopper oops. Not our fault with the documentation, I assume?
nah, it was all me
@stephenmhopper I laughed at first, but that actually sounds quite helpful.
I'll call it troubleshoot-onyx
Onyx: hard mode
Yeah, it'd be nice to have
I always seem to name the last parameter of apply-state-update "update". It must have cost me 2 days in total already.
that's a good one
I was trying to pass 41 million segments through Onyx locally using a lazy sequence. The basic flow was: onyx-seq
-> custom function that does nothing -> core.async/output
. I was running this inside of a REPL using the with-test-env
setup. I set up a go-loop
inside of the with-test-env
call to read messages from the output channel and print them out if the value in the map was divisible by 100k. Things were going great until message ~21 million
And then this guy showed up in the REPL output: FATAL [onyx.messaging.aeron.publication-manager:37] - Writing to closed publication. java.lang.IllegalStateException: Publication has been closed
And the onyx.log
starting spitting out a ton of these WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
This event became more frequent to the point that it happened three times within a 100k segment interval and now things appear to be at a standstill
Any ideas on what's going on here?
Looks like you’re probably GC’ing hard
that's what I was afraid of
I'm not seeing anything in my code that would cause the JVM to hold references to segments. Is there something inherent in the way the with-test-env
setup works that would cause this?
Is there anything else in the log that is interesting?
Would it be worth the effort to add the plugin part to onyx-local-rt
? at least the protocol part so that I could plug some external functionality (e.g. execute the task in a AWS lambda) ?
@richiardiandrea I don’t think so, it would raise the complexity bar enough to diminish local-rt’s purpose - to be stateless and threadless. And then there’s ClojureScript incompatibility, most plugins target JVM projects. If you need a protocol, you can always wrap the function to add a segment to local-rt.
Build around the runtime, not inside it 🙂
Mmm...yeah maybe I did not think this through enough, need some hammock time, thanks a lot 🙂
@lucasbradstreet I'm not seeing anything interesting in the logs. There's a billion of these guys: WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
And then occasionally, something like this will show up to the party:
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
INFO [onyx.messaging.messenger-buffer:19] - Stopping Messenger Buffer
INFO [onyx.messaging.aeron:55] - Stopping Aeron Messenger
INFO [onyx.messaging.acking-daemon:94] - Stopping Acking Daemon
INFO [onyx.messaging.acking-daemon:69] - Stopped Ack Messages Loop
INFO [onyx.peer.virtual-peer:54] - Stopping Virtual Peer 35a643ac-db85-47b9-b870-aabd91e2b99f
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
INFO [onyx.peer.task-lifecycle:580] - Job ff05c726-3370-4b8f-b804-5aaa03406bd5 {:job-id #uuid "ff05c726-3370-4b8f-b804-5aaa03406bd5", :job-hash "761fe79b377b8b214f701a6335b9f365776c4df0f636b87ef85adc11354f7d2"} - Task :in - Peer 35a643ac-db85-47b9-b870-aabd91e2b99f - Stopping task lifecycle
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
Which version of Onyx? Conductor Service Exceptions are almost always GCs
maybe you can try increasing the timeout
Try '-Daeron.client.liveness.timeout=50000000000' as a jvm flag, or however you’d rather set the java property
okay, thank you!
(and I'm on 0.9.15)