Fork me on GitHub
#onyx
<
2016-12-12
>
mariusz_jachimowicz10:12:26

1. lisp way of writing (c (b (a))) requires more eye tracking than (-> a b c). What do you think?

yonatanel10:12:36

@mariusz_jachimowicz without the threading macro i wouldn't have used clojure

mariusz_jachimowicz10:12:23

2. also formatting code in blocks way could also help code readability

mariusz_jachimowicz10:12:41

3. @lucasbradstreet @michaeldrogalis what do you think about using this formatting styles? I could do that in many places if it is ok for you.

mariusz_jachimowicz13:12:07

4. @lucasbradstreet @michaeldrogalis I see that often there is component that holds channel inside that is used by other components - for instance group-ch is used by few components. It could be good to introduce Channels component that holds only common used channels between components. It could lead into -> loose coupling -> easier testing components.

lellis13:12:51

Hi all! Im trying build my production onyx system, and now im have these log error when submit job. Any tips?

16-12-12 13:34:55 MBPdeFrederico INFO [onyx.log.zookeeper:97] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-12-12 13:34:56 MBPdeFrederico INFO [onyx.log.zookeeper:126] - Stopping ZooKeeper client connection
16-12-12 13:34:56 MBPdeFrederico DEBUG [formiguinhas-service.onyx.core:77] - Successfully submitted job:  43484d37-20fd-490b-9811-44307945e66f
16-12-12 13:34:56 MBPdeFrederico DEBUG [formiguinhas-service.onyx.core:78] - Blocking on job completion...
16-12-12 13:34:56 MBPdeFrederico INFO [onyx.log.zookeeper:97] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-12-12 13:34:56 MBPdeFrederico INFO [onyx.log.zookeeper:97] - Starting ZooKeeper client connection. If Onyx hangs here it may indicate a difficulty connecting to ZooKeeper.
16-12-12 13:34:56 MBPdeFrederico WARN [onyx.log.zookeeper:206] - 
                                                 java.lang.Thread.run              Thread.java:  745
                   java.util.concurrent.ThreadPoolExecutor$Worker.run  ThreadPoolExecutor.java:  617
                    java.util.concurrent.ThreadPoolExecutor.runWorker  ThreadPoolExecutor.java: 1142
                                                                  ...                               
                                    clojure.core.async/thread-call/fn                async.clj:  434
                                   onyx.log.zookeeper/eval39151/fn/fn            zookeeper.clj:  235
                               onyx.log.zookeeper/find-log-parameters            zookeeper.clj:  203
                            onyx.log.zookeeper/find-log-parameters/fn            zookeeper.clj:  204
                                                                  ...                               
                                      onyx.log.zookeeper/eval39483/fn            zookeeper.clj:  550
                         onyx.monitoring.measurements/measure-latency         measurements.clj:   11
                                   onyx.log.zookeeper/eval39483/fn/fn            zookeeper.clj:  551
                       onyx.log.zookeeper/clean-up-broken-connections            zookeeper.clj:   77
                                onyx.log.zookeeper/eval39483/fn/fn/fn            zookeeper.clj:  554
                                                onyx.log.curator/data              curator.clj:  126
       org.apache.curator.framework.imps.GetDataBuilderImpl$2.forPath  GetDataBuilderImpl.java:  138
       org.apache.curator.framework.imps.GetDataBuilderImpl$2.forPath  GetDataBuilderImpl.java:  142
         org.apache.curator.framework.imps.GetDataBuilderImpl.forPath  GetDataBuilderImpl.java:  279
org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground  GetDataBuilderImpl.java:  287
                           org.apache.curator.RetryLoop.callWithRetry           RetryLoop.java:  107
          org.apache.curator.framework.imps.GetDataBuilderImpl$4.call  GetDataBuilderImpl.java:  291
          org.apache.curator.framework.imps.GetDataBuilderImpl$4.call  GetDataBuilderImpl.java:  302
                               org.apache.zookeeper.ZooKeeper.getData           ZooKeeper.java: 1155
                          org.apache.zookeeper.KeeperException.create     KeeperException.java:   51
                          org.apache.zookeeper.KeeperException.create     KeeperException.java:  111
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /onyx/1/log-parameters/log-parameters
    code: -101
    path: "/onyx/1/log-parameters/log-parameters"

16-12-12 13:34:56 MBPdeFrederico WARN [onyx.log.zookeeper:207] - Log parameters couldn't be discovered. Backing off 500ms and trying again…
`

Travis13:12:40

Are your peers connecting to zookeeper?

Travis13:12:44

Same onyx Id?

lellis13:12:48

same, 1 for both config files.

Travis13:12:10

Can you verify

/onyx/1
is in zookeeper?

lellis13:12:44

sure 1 let me see!

lellis13:12:33

where can i supose find these /onyx/1 @camechis ?

Travis13:12:04

Use either zookeeper shell or exhibitor if you have it

lellis14:12:09

i think im running zookeper embeed. Let me install standalone and connect to shell.

lellis14:12:53

Sorry about dummy questions @camechis but all these is new to me. 😄

Travis14:12:31

Are you only running one peer?

Travis14:12:38

Or are you running the peer and your submit job in separate processes

Travis14:12:54

If so you will need to run a real zookeeper

Travis14:12:08

So they can both connect and see the same thing

lellis14:12:45

Yeah im running in separate processes

Travis14:12:07

Yeah they both need to connect to the same zk

lellis14:12:59

ok, im gonna install zk. ty!

yonatanel14:12:21

Is the onyx-local-rt behavior defined for calling new-segment from withing tick or drain (e.g from a sync function that knows the runtime itself)

yonatanel14:12:49

I'm starting to need event sourcing for my CQRS simulation in order to simulate emitting commands in response to events 😱

dominicm15:12:17

@mariusz_jachimowicz I don't really like that use of the threading macro. Nor of aligning arguments in that style: - It's near impossible to get truly aesthetic alignment looking right everywhere - and it's a pain to set up in every editor - (get-in x [:foo :bar]) is more idiomatic than (-> x :foo :bar)

stephenmhopper15:12:56

Does someone have an example of how I can use onyx-seq to wrap a lazy seq as input? I'm running into issues because my current setup keeps trying to call .close and things like that on a nil reader instance. Since I'm just emitting a basic lazy seq, I don't really have a reader

gardnervickers15:12:20

@stephenmhopper: I assume you're adapting the plugin example code but don't want to use a reader? In that case you should drop the .close call if there's nothing to actually close.

stephenmhopper15:12:55

But it's still throwing this exception: CompilerException java.lang.IllegalArgumentException: No implementation of method: :close! of protocol: #'clojure.core.async.impl.protocols/Channel found for class: nil

gardnervickers15:12:09

What are you using for an output?

stephenmhopper15:12:35

{:onyx/name :out
              :onyx/plugin :onyx.plugin.core-async/output
              :onyx/medium :core.async
              :onyx/max-peers 1
              :onyx/type :output
              :onyx/batch-size 20}

gardnervickers15:12:08

Are the lifecycle calls for core async being called to inject the channel for your output?

michaeldrogalis16:12:42

@mariusz_jachimowicz I’m not interested in changing the coding style that Onyx uses, the primary reasoning being diff busting.

michaeldrogalis16:12:21

For the same reason that Clojure discourages style & whitespace changes - it’s significantly harder to backtrack through history to discover which patch caused a problem.

michaeldrogalis16:12:04

Re, point number 4 - the vast majority of channels have been removed from Onyx’s implementation in the next release.

gardnervickers16:12:14

The threading macro accessor style is limited to elements that implement IFn, right?

michaeldrogalis16:12:26

@yonatanel It’s not defined per say, but I’ve given it some thought. It’s a bit tricky because you need to stick segments directly into the inbox of the env, and that env would then need to be contained within an atom to make it work.

michaeldrogalis16:12:53

@stephenmhopper You can just implement your own lifecycles for onyx-seq if you don’t want to use something that’s Closable. Their actual lifecycles are quite small. Or you might be able to get away with not submitting the after-task-stop lifecycle to the job and keep the others

michaeldrogalis16:12:27

@yonatanel That being said, I don’t think there’s anything wrong with what you’re doing. It shouldn’t cause bad behavior.

yonatanel16:12:32

@michaeldrogalis It's too weird to call a function that returns a new state within another one like that. I've decided to request all state changes to anything in my simulation though a channel instead.

michaeldrogalis16:12:18

@yonatanel That seems easier to think about, yes.

yonatanel16:12:00

I thought om.next transact! could do it but its changes are immediate and doing nested transacts creates states that I don't want to reason about.

yonatanel16:12:08

exactly, what you said

yonatanel16:12:49

We are fast forwarding to deploy onyx in production, so that simulation development will be slowed down 😰

michaeldrogalis16:12:02

@yonatanel Seems positive overall? 😛

yonatanel16:12:41

The whole thing is too complicated to just quickly throw a system together, but I've researched options and it's the best I found so...

yonatanel16:12:02

I mean, I'm kinda glad it doesn't just do everything for you until one day you realize you're missing some guarantees or that the memory usage is unbounded or something like that.

yonatanel16:12:03

So yes, positive

stephenmhopper17:12:19

Well, I figured out my issue from earlier. As it turns out, I was using the onyx-seq plugin definition, but was trying to use :onyx.plugin.core-async/reader-calls instead of :onyx.plugin.seq/reader-calls

stephenmhopper17:12:20

Yeah, if I find some time and manage to make a working app at some point in the future, I'm going to make a project similar to learn-onyx except everything will be broken and throwing errors and you have to figure out what's wrong

lucasbradstreet17:12:21

@stephenmhopper oops. Not our fault with the documentation, I assume?

stephenmhopper17:12:28

nah, it was all me

michaeldrogalis17:12:47

@stephenmhopper I laughed at first, but that actually sounds quite helpful.

stephenmhopper17:12:48

I'll call it troubleshoot-onyx

stephenmhopper17:12:19

Yeah, it'd be nice to have

yonatanel18:12:18

I always seem to name the last parameter of apply-state-update "update". It must have cost me 2 days in total already.

yonatanel18:12:35

If you're looking for ideas for troubleshoot-onyx :)

stephenmhopper18:12:09

that's a good one

stephenmhopper19:12:00

I was trying to pass 41 million segments through Onyx locally using a lazy sequence. The basic flow was: onyx-seq -> custom function that does nothing -> core.async/output. I was running this inside of a REPL using the with-test-env setup. I set up a go-loop inside of the with-test-env call to read messages from the output channel and print them out if the value in the map was divisible by 100k. Things were going great until message ~21 million

stephenmhopper19:12:09

And then this guy showed up in the REPL output: FATAL [onyx.messaging.aeron.publication-manager:37] - Writing to closed publication. java.lang.IllegalStateException: Publication has been closed

stephenmhopper19:12:35

And the onyx.log starting spitting out a ton of these WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns

stephenmhopper19:12:33

This event became more frequent to the point that it happened three times within a 100k segment interval and now things appear to be at a standstill

stephenmhopper19:12:38

Any ideas on what's going on here?

lucasbradstreet19:12:28

Looks like you’re probably GC’ing hard

stephenmhopper19:12:43

that's what I was afraid of

stephenmhopper19:12:38

I'm not seeing anything in my code that would cause the JVM to hold references to segments. Is there something inherent in the way the with-test-env setup works that would cause this?

lucasbradstreet20:12:13

Is there anything else in the log that is interesting?

richiardiandrea20:12:03

Would it be worth the effort to add the plugin part to onyx-local-rt? at least the protocol part so that I could plug some external functionality (e.g. execute the task in a AWS lambda) ?

michaeldrogalis20:12:12

@richiardiandrea I don’t think so, it would raise the complexity bar enough to diminish local-rt’s purpose - to be stateless and threadless. And then there’s ClojureScript incompatibility, most plugins target JVM projects. If you need a protocol, you can always wrap the function to add a segment to local-rt.

michaeldrogalis20:12:27

Build around the runtime, not inside it 🙂

richiardiandrea20:12:27

Mmm...yeah maybe I did not think this through enough, need some hammock time, thanks a lot 🙂

stephenmhopper21:12:31

@lucasbradstreet I'm not seeing anything interesting in the logs. There's a billion of these guys: WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns And then occasionally, something like this will show up to the party:

stephenmhopper21:12:36

WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
INFO [onyx.messaging.messenger-buffer:19] - Stopping Messenger Buffer
INFO [onyx.messaging.aeron:55] - Stopping Aeron Messenger
INFO [onyx.messaging.acking-daemon:94] - Stopping Acking Daemon
INFO [onyx.messaging.acking-daemon:69] - Stopped Ack Messages Loop
INFO [onyx.peer.virtual-peer:54] - Stopping Virtual Peer 35a643ac-db85-47b9-b870-aabd91e2b99f
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns
INFO [onyx.peer.task-lifecycle:580] - Job ff05c726-3370-4b8f-b804-5aaa03406bd5 {:job-id #uuid "ff05c726-3370-4b8f-b804-5aaa03406bd5", :job-hash "761fe79b377b8b214f701a6335b9f365776c4df0f636b87ef85adc11354f7d2"} - Task :in - Peer 35a643ac-db85-47b9-b870-aabd91e2b99f - Stopping task lifecycle
WARN [onyx.messaging.aeron.publication-manager:79] - Aeron messaging publication error: io.aeron.exceptions.ConductorServiceTimeoutException: Timeout between service calls over 5000000000ns

lucasbradstreet21:12:43

Which version of Onyx? Conductor Service Exceptions are almost always GCs

lucasbradstreet21:12:48

maybe you can try increasing the timeout

lucasbradstreet21:12:09

Try '-Daeron.client.liveness.timeout=50000000000' as a jvm flag, or however you’d rather set the java property

stephenmhopper21:12:26

okay, thank you!

stephenmhopper21:12:47

(and I'm on 0.9.15)