Fork me on GitHub
#onyx
<
2016-02-15
>
robert-stuttaford04:02:35

hi Lucas simple_smile is there a way to pass arbitrary values to :flow/post-transform functions? i have yeller logging inside one, and i need to provide it with configuration

robert-stuttaford05:02:14

i suppose i could grab the catalog from the event and dig around in there to find the config i injected for tasks

lucasbradstreet05:02:36

@robert-stuttaford: yeah, I think I would tend to inject that config when I need it, rather than have it part of the segment.

robert-stuttaford06:02:37

yup. that’s what i’ve done; remove it from the segment. just wondering about the pattern for /post-transform, as it doesn’t support the :onyx/params capability

lucasbradstreet06:02:55

Well, if you really need it at that point, I'd inject it into the event map via a lifecycle and then grab it back out in post-transform. But I would tend to inject it in whatever task that is handling the transformed messages instead

robert-stuttaford06:02:20

it’s the error flow cond which all tasks flow to

robert-stuttaford06:02:28

so injecting it via the task is not ideal

robert-stuttaford06:02:44

so i could use an :all lifecycle to inject config, but then how do i pass the config to the lifecycle simple_smile

lucasbradstreet06:02:54

Ok so what I'm trying to say is that you should inject it as a param to your handle-error task

robert-stuttaford06:02:27

that’s simply (constantly true) right now

lucasbradstreet06:02:28

Since that's what handles all the exceptions, right?

robert-stuttaford06:02:38

the post-transform is what does the yeller logging

robert-stuttaford06:02:56

i can’t remember why we set it up this way simple_smile

lucasbradstreet06:02:06

IMO that's the wrong way to go about it.

robert-stuttaford06:02:41

i updated the snippet above. you’re saying we can drop the post-tranform and do the logging inside the pred?

robert-stuttaford06:02:49

and then use onyx/params to pass config to the pred

lucasbradstreet06:02:57

Post transform is only really for transformation. Where it's flowing to is where you want to do any side effects

lucasbradstreet06:02:25

What is the tag-error task doing atm?

robert-stuttaford06:02:55

it just adds a :highstorm/error true tag to the transaction

robert-stuttaford06:02:03

so that we can know which txes caused issues later

lucasbradstreet06:02:50

Ok that makes sense. I guess I can see why you want to log it in the task throwing the exception so that you have the full exception object

lucasbradstreet06:02:55

@robert-stuttaford: so what I’m thinking is you build a lifecycle like this:

lucasbradstreet06:02:02

by returning the info you want in a map from the lifecycle function, you can inject the yeller config into the event map

lucasbradstreet06:02:29

Note that the lifecycle map has the config in this case

lucasbradstreet06:02:53

make sure to turn off the params injection though, because you wouldn’t want that on all the tasks

robert-stuttaford06:02:45

what is inject-conn returning?

lucasbradstreet06:02:08

The other thing worth considering is that if you setup a timbre appender that logs to yeller, it should all of these exceptions to yeller anyway

lucasbradstreet06:02:21

Lifecycle functions always return maps

lucasbradstreet06:02:35

see the second link

robert-stuttaford06:02:06

ok. let me get my head around all of this and i’ll come back if i have questions

robert-stuttaford06:02:21

so the root cause is this:

robert-stuttaford06:02:22

(defn log-segment-error [config {:keys [datom tx] :as segment} exception-obj] (if (yeller/env-is-yeller-env? config) (yeller/report config exception-obj {:environment (:environment config) :custom-data segment}) (log/error datom "\n\n" (pprint-str tx) "\n\n" exception-obj)))

robert-stuttaford06:02:34

when yeller is enabled, log to yeller. when not, log to disk (for dev)

robert-stuttaford06:02:13

i’ll shout in a mo once i’ve tried all this on

robert-stuttaford06:02:43

so it’ll put whatever i put in the map (in the example you provided, at :datomic/conn) into the top of the event map?

robert-stuttaford06:02:50

so that i can just read it right out of the event

lucasbradstreet06:02:21

Yep it just gets merged into the event map

robert-stuttaford07:02:48

jobs are stopping cleanly .. and quickly! hooray!

robert-stuttaford07:02:00

and the error handling appears to be working as well

robert-stuttaford07:02:35

testing the yeller appender now

robert-stuttaford07:02:25

clojure.lang.ExceptionInfo: Unfreezable type: class yeller.timbre_appender$make_yeller_appender$fn__7913 [email protected]

robert-stuttaford07:02:30

nippy is choking on the appender

lucasbradstreet07:02:20

Hmm. Are you putting it in a segment?

lucasbradstreet07:02:34

I assume you're not trying the event map approach yet?

robert-stuttaford07:02:07

ok will have to rethink that

lucasbradstreet07:02:19

Oh if it's an onyx/param you might have it in the wrong order

lucasbradstreet07:02:36

I mean your function might think it's a segment

robert-stuttaford08:02:32

hey Lucas, i’m getting this every now and then as i use the reset: Missing file for cnc: /var/folders/5h/lrqzt7gx61gg_zkqt47pmfzr0000gn/T/aeron-robert/cnc

robert-stuttaford08:02:48

my understanding was that this is all automatically managed now?

lucasbradstreet08:02:20

On startup or shutdown?

lucasbradstreet08:02:28

Embedded media driver, right?

robert-stuttaford08:02:43

this is locally in dev

lucasbradstreet08:02:01

You're sure the previous instance is completely shut down first?

robert-stuttaford08:02:37

i’ll admit it might not be

lucasbradstreet08:02:11

That's my first thought - that the previous one is deleting it as the next one is startup up

robert-stuttaford08:02:21

ah, that sounds likely

lucasbradstreet08:02:33

Maybe we should drop the deletion on shutdown now. We do it to clear the space

robert-stuttaford08:02:05

doing a fresh run now. pretty close to having everything stitched up on the error handling front. then i can move to test servers and work on getting two nodes processing tasks, which is the last issue i have

robert-stuttaford08:02:38

it’s happening on a fresh jvm startup

lucasbradstreet08:02:45

Ok interesting. Actually throws an exception and won't start up?

robert-stuttaford08:02:15

java.lang.IllegalStateException: Missing file for cnc: /var/folders/5h/lrqzt7gx61gg_zkqt47pmfzr0000gn/T/aeron-robert/cnc
java.lang.IllegalStateException: Could not initialise communication buffers
     clojure.lang.ExceptionInfo: Error in component :messaging-group in system onyx.system.OnyxPeerGroup calling #'com.stuartsierra.component/start

lucasbradstreet08:02:20

And definitely starting a media driver huh?

lucasbradstreet08:02:49

i.e. embedded-media-driver? true

robert-stuttaford08:02:19

i just checked, i’m not

robert-stuttaford08:02:24

how’d that change...

robert-stuttaford09:02:54

just tested a transactor death. yeller+timbre is working!

lucasbradstreet09:02:28

Nice! In that case you don’t need to inject the lifecycles, right? Just use timbre to log.

robert-stuttaford09:02:37

well, we still need to tell it whether to log to yeller or not

robert-stuttaford09:02:06

Michael said that lifecycles only manages what happens when errors occur, we still need flow conditions to actually log to yeller

robert-stuttaford09:02:13

or do i misunderstand things?

robert-stuttaford09:02:30

i can control whether the appender is included or not based on env, so that’s ok

lucasbradstreet09:02:06

You setup yeller as a timbre appender right? I just mean you can use timbre's info, warn and error from your flow conditions

lucasbradstreet09:02:25

Rather than having to inject in the yeller config

robert-stuttaford09:02:49

we are using the yeller custom-data arg to include the segment

robert-stuttaford09:02:16

(yeller/report config exception-obj {:custom-data segment})

robert-stuttaford09:02:24

which is hella useful simple_smile

lucasbradstreet09:02:45

Yeah fair enough. That's kinda handy

robert-stuttaford15:02:30

so, we’re using logback. as soon as i provide an :onyx/config key to peer config, all our logback config stops working

robert-stuttaford15:02:32

i’d like to keep the previous setup - onyx.log for onyx, and our own two logfiles (hs.log and error.hs.log) via logback, but continue to use the yeller-timbre appender. right now, with :onyx/config set, all the onyx logging is coming to stdout instead, and, weirdly, so too is all the stuff usually destined for the logback files.

robert-stuttaford15:02:30

also, with the yeller appender included, i am now getting exceptions on this line

robert-stuttaford15:02:56

clojure.lang.ExceptionInfo: Unfreezable type: class clojure.lang.Delay [email protected]

robert-stuttaford15:02:03

testing with the appender disabled to verify

robert-stuttaford15:02:08

ok. the delay error is still happening. i’ll try to track down why. definitely want to figure the log file config thing out, though simple_smile

lucasbradstreet17:02:18

Huh. That delay issue again.

lucasbradstreet17:02:11

For the logback stuff, maybe timbre is messing with it. Could you post your onyx log config?

robert-stuttaford17:02:56

it’s literally just this:

robert-stuttaford17:02:01

{:appenders {:yeller-appender (yeller/timbre-appender config)}}

lucasbradstreet17:02:37

Ah, so you’ve basically switched off all the other appenders

robert-stuttaford17:02:28

i’ll update-in + merge

lucasbradstreet17:02:57

I don’t think that’ll help. Basically if you supply a config it replaces the existing one automatically

lucasbradstreet17:02:26

Oh, I thought it should merge

lucasbradstreet17:02:41

Ah, the rotor one is only used if you don’t supply one

robert-stuttaford17:02:44

true. i can’t merge

robert-stuttaford17:02:56

cool, i’ll duplicate that stuff and include yeller

robert-stuttaford17:02:36

i wonder what the delay issue is? it’s a fresh onyx/id, so it can’t be old ZK state

robert-stuttaford17:02:44

the onyx/id is based on git sha now

lucasbradstreet17:02:46

Yeah, you’ve seen that one before IIRC

robert-stuttaford17:02:02

yup. you did something clever and it went away

lucasbradstreet17:02:21

can you post the stack trace?

lucasbradstreet17:02:30

I don’t know if we did anything last time

robert-stuttaford17:02:33

i sure can, in about 20

lucasbradstreet17:02:35

I thought it was something in your code

robert-stuttaford17:02:26

just getting the logging fixed

robert-stuttaford17:02:02

it’s quite late for you, isn’t it?

lucasbradstreet17:02:10

Yeah, almost time to sleep

robert-stuttaford17:02:33

ok. please don’t stick around on my account simple_smile let’s catch up tomorrow!

lucasbradstreet18:02:53

No worries. G’night!