This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-08-08
Channels
- # admin-announcements (2)
- # beginners (35)
- # boot (353)
- # capetown (1)
- # cider (1)
- # cljs-dev (41)
- # cljsjs (3)
- # cljsrn (3)
- # clojure (118)
- # clojure-austin (12)
- # clojure-russia (17)
- # clojure-spec (21)
- # clojure-taiwan (1)
- # clojure-uk (91)
- # clojurescript (80)
- # clojurex (1)
- # cloverage (3)
- # datomic (66)
- # devcards (2)
- # events (2)
- # garden (6)
- # hoplon (54)
- # jobs-rus (1)
- # keechma (1)
- # lein-figwheel (4)
- # leiningen (3)
- # luminus (3)
- # off-topic (7)
- # om (4)
- # onyx (53)
- # other-languages (17)
- # proton (7)
- # protorepl (4)
- # re-frame (123)
- # reagent (1)
- # ring (6)
- # rum (2)
- # spacemacs (1)
- # specter (21)
- # testing (1)
- # untangled (1)
- # yada (42)
Primes. 😄
hi - I'm finally able to give onyx a test run... I've gone through learn-onyx, and am now attempting to read a large file simply as a pass through, via onyx-seq
. Just to start, I want to see things are flowing correctly even before I alter the deftest conditions to use real data from the file, rather than the segments defined in the test file. (run-tests)
is giving me {:test 1, :pass 0, :fail 0, :error 1, :type :summary}
, so my guess is that the file isn't being read, but I'm not yet sure where to find the error messages (I'm running it from emacs cider). I also have the onyx-dashboard running on localhost:3000, but it doesn't yet see the job. How do I configure that? Code is here: https://www.refheap.com/121997
Can you post the content of your onyx.log file? You should find it at the root directory your project.
another note, I started the project via lein new onyx-app bigfiletest -- +docker
and then added things for onyx-seq
Hm, that log file looks okay to me, too. It seems strange that you don't see an exception when the test report indicates one error
it doesn't show up automatically in the window, but scrolling around it suddenly appears
I think the dashboard is not showing you the job because the zookeeper server is being started and stopped every time you run a test
You would need to run a zookeeper server (should be easy when you are using docker anyway), use that server instead of the embedded one for onyx and set a new tenancy-id every time you run a test
@aaelony: I have the dashboard running as well and I can see everything but the job in a production like env. I have posted an issue to the github issue tracker
cool, @camechis I'll check it out. My docker-beta is acting up, but will get that sorted out too...
hi, i'm playing around with onyx and i'm not sure how to get the whole pipeline using onyx-twitter and onyx-es to run in the repl. i've used the onyx-app template and copied over the code reloading bit. i am able to start the system using (go)
. when i submit a job it says it was successful but nothing seems to happen. any tips on how i can debug?
@jmv: Are there any errors in the log file?
i'm seeing some odd behaviour with onyx-kafka 0.9.9.0 and kafka 0.10.0.0 - it seems to be retrieving wrong consumer offsets and replaying a load of messages - does that ring match any known problems ?
@mccraigmccraig: We haven't tested the plugin with Kafka 0.10 yet. I'm not sure what changes went into the next round of Kafka.
@jmv: Would need to see more of the set up to know what's going on. A guess at first glance is that there's nothing keeping the peers up. Perhaps the process is submitting the job, then immediately tearing the whole test environment down.
ah... it looks like the broker itself is recording offsets, judging by what's on disk, so there may be nothing in zk
so i guess i should revert to kafka 0.9 and see
Yeah, check it out and let us know.
@jmv: How're you invoking it from the repl?
@jmv: Would recommend with with-test-env
for these kinds of things. It's still kind of hard to tell what's going on, depending on what go
is doing.
@jmv: Okay. Do the logs indicate that your virtual peers started after invoking go
?
@jmv: Can you post the entire onyx.log file? Seems like something that would stick out.
Did your workflow have more than 4 tasks?
I have gzip compressed files. I see https://github.com/onyx-platform/onyx-seq/blob/0.9.x/src/onyx/tasks/seq.clj#L11-L14 has
(defn inject-in-reader [event lifecycle]
(let [rdr (FileReader. (:buffered-file-reader/filename (:onyx.core/task-map event)))]
{:seq/rdr rdr
:seq/seq (map (partial hash-map :val) (line-seq (BufferedReader. rdr)))}))
should I just use a new function I have that does :seq/seq (-> rdr java.util.zip.GZIPInputStream. java.io.BufferedReader. line-seq)
(with the :val thing as well)... or is there a more elegant way to make this available in the general plugin?@jmv: It's strange that none of the peers echo that they're starting to work on tasks. You should either see that, or a message in the logs that says the job didn't have enough peers to get off the ground. Are you on Onyx 0.9.9? I don't admittedly use the template or starter much, so Im not immediately familiar with the code.
Maybe your logs are getting redirected somewhere else after that line. Or there might be a problem with the template. Sorry Im a bit sporadic, in the middle of something.
@aaelony: That's how I would do it.
yeah i'm trying 0.9.9. i also have the starter at 0.8.4 and that seems to start the sample tasks ok. and no worries, i'm doing other stuff too so take your time
@jmv: Thanks 🙂 There were some binary incompatible changes between 0.8 and 0.9 btw, be careful there to stay on 0.9
cool, thanks @michaeldrogalis
this is interesting @michaeldrogalis ... the onyx-kafka consumer offset problem i'm seeing is there with kafka 0.9.0.1 too, but it's only in development and not in production - production is correctly resuming after the last kafka log offset processed (production is on kafka 0.10.0.0, i've tried both 0.10.0.0 and 0.9.0.1 in development)
that presumably means i have a problem with reset not following :largest
... since my dev environment creates new jobs every reload, whereas production generally reloads the same job
here's my catalog for the task - https://www.refheap.com/470754c4f1daab1e0f2610f71
@mccraigmccraig: Okay, thanks. Can you open an issue on onyx-kafka? We're a bit fragmented this week. Lucas is out and Gardner is in town with me working on something.
Will get it checked out as soon as we can.
@michaeldrogalis: sure, i'll see if i can get a bit more detail too - i'm also a bit fragmented this week 😉
Thanks ^^