Fork me on GitHub
#onyx
<
2016-08-08
>
aaelony16:08:34

hi - I'm finally able to give onyx a test run... I've gone through learn-onyx, and am now attempting to read a large file simply as a pass through, via onyx-seq. Just to start, I want to see things are flowing correctly even before I alter the deftest conditions to use real data from the file, rather than the segments defined in the test file. (run-tests) is giving me {:test 1, :pass 0, :fail 0, :error 1, :type :summary}, so my guess is that the file isn't being read, but I'm not yet sure where to find the error messages (I'm running it from emacs cider). I also have the onyx-dashboard running on localhost:3000, but it doesn't yet see the job. How do I configure that? Code is here: https://www.refheap.com/121997

aaelony16:08:10

any thoughts appreciated

dignati16:08:43

Can you post the content of your onyx.log file? You should find it at the root directory your project.

aaelony17:08:57

sure, I don't see anything that looks like an error in there though

aaelony17:08:29

another note, I started the project via lein new onyx-app bigfiletest -- +docker and then added things for onyx-seq

dignati17:08:15

Hm, that log file looks okay to me, too. It seems strange that you don't see an exception when the test report indicates one error

aaelony17:08:33

that's right. Perhaps cider is burying it...

aaelony17:08:26

aha! yes, it's in cider

aaelony17:08:50

it doesn't show up automatically in the window, but scrolling around it suddenly appears

dignati17:08:59

I think the dashboard is not showing you the job because the zookeeper server is being started and stopped every time you run a test

aaelony17:08:51

any hints on what works best to develop/debug jobs using the dashboard?

aaelony17:08:38

I'm seeing the error now, I should be able to fix it...

dignati17:08:53

You would need to run a zookeeper server (should be easy when you are using docker anyway), use that server instead of the embedded one for onyx and set a new tenancy-id every time you run a test

dignati17:08:07

It's probably easier to debug by keeping an eye on the repl output and onyx.log file

aaelony17:08:16

sounds good, I'll look into that. Many thanks for your help

dignati17:08:28

Sure, happy to help 🙂

Travis17:08:22

@aaelony: I have the dashboard running as well and I can see everything but the job in a production like env. I have posted an issue to the github issue tracker

aaelony17:08:43

cool, @camechis I'll check it out. My docker-beta is acting up, but will get that sorted out too...

jmv19:08:01

hi, i'm playing around with onyx and i'm not sure how to get the whole pipeline using onyx-twitter and onyx-es to run in the repl. i've used the onyx-app template and copied over the code reloading bit. i am able to start the system using (go). when i submit a job it says it was successful but nothing seems to happen. any tips on how i can debug?

michaeldrogalis19:08:06

@jmv: Are there any errors in the log file?

jmv19:08:34

no it looks like it starts everything up and then shuts it down

mccraigmccraig19:08:33

i'm seeing some odd behaviour with onyx-kafka 0.9.9.0 and kafka 0.10.0.0 - it seems to be retrieving wrong consumer offsets and replaying a load of messages - does that ring match any known problems ?

michaeldrogalis19:08:39

@mccraigmccraig: We haven't tested the plugin with Kafka 0.10 yet. I'm not sure what changes went into the next round of Kafka.

michaeldrogalis19:08:10

@jmv: Would need to see more of the set up to know what's going on. A guess at first glance is that there's nothing keeping the peers up. Perhaps the process is submitting the job, then immediately tearing the whole test environment down.

mccraigmccraig19:08:11

ah... it looks like the broker itself is recording offsets, judging by what's on disk, so there may be nothing in zk

mccraigmccraig19:08:13

so i guess i should revert to kafka 0.9 and see

michaeldrogalis19:08:30

Yeah, check it out and let us know.

jmv19:08:24

i'm kicking it off from the repl so i'm not sure if the environment would be taken down

michaeldrogalis19:08:10

@jmv: How're you invoking it from the repl?

michaeldrogalis19:08:30

@jmv: Would recommend with with-test-env for these kinds of things. It's still kind of hard to tell what's going on, depending on what go is doing.

michaeldrogalis19:08:14

@jmv: Okay. Do the logs indicate that your virtual peers started after invoking go?

jmv20:08:04

yeah it started 4

michaeldrogalis20:08:50

@jmv: Can you post the entire onyx.log file? Seems like something that would stick out.

michaeldrogalis20:08:58

Did your workflow have more than 4 tasks?

aaelony20:08:05

I have gzip compressed files. I see https://github.com/onyx-platform/onyx-seq/blob/0.9.x/src/onyx/tasks/seq.clj#L11-L14 has

(defn inject-in-reader [event lifecycle]
  (let [rdr (FileReader. (:buffered-file-reader/filename (:onyx.core/task-map event)))]
    {:seq/rdr rdr
     :seq/seq (map (partial hash-map :val) (line-seq (BufferedReader. rdr)))}))
should I just use a new function I have that does
:seq/seq  (-> rdr java.util.zip.GZIPInputStream. java.io.BufferedReader. line-seq)
(with the :val thing as well)... or is there a more elegant way to make this available in the general plugin?

michaeldrogalis20:08:18

@jmv: It's strange that none of the peers echo that they're starting to work on tasks. You should either see that, or a message in the logs that says the job didn't have enough peers to get off the ground. Are you on Onyx 0.9.9? I don't admittedly use the template or starter much, so Im not immediately familiar with the code.

michaeldrogalis20:08:43

Maybe your logs are getting redirected somewhere else after that line. Or there might be a problem with the template. Sorry Im a bit sporadic, in the middle of something.

michaeldrogalis20:08:57

@aaelony: That's how I would do it.

jmv20:08:10

yeah i'm trying 0.9.9. i also have the starter at 0.8.4 and that seems to start the sample tasks ok. and no worries, i'm doing other stuff too so take your time

jmv20:08:17

and thanks for helping at all!

michaeldrogalis20:08:57

@jmv: Thanks 🙂 There were some binary incompatible changes between 0.8 and 0.9 btw, be careful there to stay on 0.9

jmv20:08:50

ah good to know

mccraigmccraig21:08:57

this is interesting @michaeldrogalis ... the onyx-kafka consumer offset problem i'm seeing is there with kafka 0.9.0.1 too, but it's only in development and not in production - production is correctly resuming after the last kafka log offset processed (production is on kafka 0.10.0.0, i've tried both 0.10.0.0 and 0.9.0.1 in development)

mccraigmccraig21:08:04

that presumably means i have a problem with reset not following :largest... since my dev environment creates new jobs every reload, whereas production generally reloads the same job

michaeldrogalis21:08:32

@mccraigmccraig: Okay, thanks. Can you open an issue on onyx-kafka? We're a bit fragmented this week. Lucas is out and Gardner is in town with me working on something.

michaeldrogalis21:08:39

Will get it checked out as soon as we can.

mccraigmccraig21:08:32

@michaeldrogalis: sure, i'll see if i can get a bit more detail too - i'm also a bit fragmented this week 😉