Fork me on GitHub
#onyx
<
2016-04-13
>
jeroenvandijk08:04:06

I’m trying to update a 0.8.11.0 generated app to 0.9.1 (as exercise and because I like the generated app in structure). Looking at the changelog I only need to update

:onyx/id
=>
:onyx/tenancy-id
. However I get this error (in the logs) when running the one test:
Assert failed: (invariants/active-job-invariant %)
. The test succeeds, but I wonder what it means. I saw that putting some delay in the tests actually gets rid of the error

lucasbradstreet09:04:42

hi @jeroenvandijk. Hmm, that’s interesting.

lucasbradstreet09:04:08

We really need to upgrade the template to 0.9. So you just updated the generated app to 0.9.1 and ran the test?

jeroenvandijk09:04:19

Yeah and i changed the onyx/id to onyx/tenancy-id in the configs

jeroenvandijk09:04:44

i can do it again and push it to github if it’s helpful

lucasbradstreet09:04:52

OK, cool. I’ll look into it. That shouldn’t happen but I wonder if there’s an element of timing with it.

lucasbradstreet09:04:58

I’m going to do that anyway, so don’t worry about it

lucasbradstreet09:04:21

hmm, it works fine for me

lucasbradstreet09:04:33

I did upgrade to 0.9.3 though

lucasbradstreet09:04:47

Oh wait nevermind

jeroenvandijk09:04:01

my previous “fix” (adding a print statement) doesn’t seem to help anymore

lucasbradstreet09:04:15

lein ancient didn’t upgrade it because the test failed

jeroenvandijk09:04:19

Ok (Thread/sleep 10) gives me no background errors

jeroenvandijk09:04:36

This test

(deftest onyx-dev-job-test
  (let [id (java.util.UUID/randomUUID)
        config (load-config)
        env-config (assoc (:env-config config) :onyx/tenancy-id id)
        peer-config (assoc (:peer-config config) :onyx/tenancy-id id)]
    ;; Be sure to set the peer count (5 here) to a number greater than
    ;; the amount of tasks in your job.
    (with-test-env [test-env [5 env-config peer-config]]
      (let [job (build-job :dev)
            {:keys [write-lines]} (get-core-async-channels job)
            _ (validate-enough-peers! test-env job)
            {:keys [job-id]} (onyx.api/submit-job peer-config job)]
        ;; FIXME When running test get this error "Assert failed: (invariants/active-job-invariant %)"
        (Thread/sleep 10)
        (feedback-exception! peer-config job-id)
        (let [results (take-segments! write-lines)]
          (is (= 4 (count results)))
          (is (= :done (last results))))))))

lucasbradstreet09:04:16

could you post the stack trace for me?

lucasbradstreet09:04:45

I think I’m going to have some problems reproducing it locally

lucasbradstreet09:04:58

@jeroenvandijk: Could you please clone onyx from [email protected]:onyx-platform/onyx.git and lein install from branch reproduce-invariant-failure ?

lucasbradstreet09:04:08

You’ll need to switch the project to use 0.9.4-SNAPSHOT

lucasbradstreet09:04:24

once that’s done it should log the offending replica to onyx.log

jeroenvandijk09:04:24

ok same error, will look into the log

jeroenvandijk09:04:43

it’s a log with just the output after running lein test once

lucasbradstreet09:04:41

@jeroenvandijk: could you pull one more time and run it again? I’m going to need see what the peer log entries are so I can see how we got here simple_smile

lucasbradstreet09:04:44

Great, it is the only thing I thought it could be. Looks like you found a bug.

jeroenvandijk09:04:16

I don’t get the error on Circle CI (just tried)

lucasbradstreet09:04:26

Yeah, it’s definitely a timing related issue. It’s caused by this https://github.com/onyx-platform/onyx/issues/493 which I’ve been meaning to fix

lucasbradstreet09:04:14

I think the peer is getting an extra chance to write an exhaust-input log entry out before it stops processing the job

jeroenvandijk09:04:57

Ah ok, I’m still to new to understand that, but great that you found it so quickly simple_smile

lucasbradstreet10:04:24

I’ll get a fix out soon. Thanks very much

lucasbradstreet10:04:51

Could you give it one pull / lein install and tell me if this fix works?

aspra10:04:15

+1 I am getting the same error some times when running the same job. I am using onyx 0.9.0

jeroenvandijk10:04:33

@lucasbradstreet I’ve run it twice, no exceptions

lucasbradstreet10:04:41

Great. I’ll get 0.9.4 out now then

lucasbradstreet11:04:30

@jeroenvandijk, @aspra: 0.9.4 is released with that fix. All the plugins are going through the release process now, but it’s safe to use 0.9.3 plugins with them for now

lucasbradstreet11:04:45

I’ll do a new template release upgraded to 0.9.4 shortly

aspra11:04:14

@lucasbradstreet thanks will upgrade

aspra11:04:45

@lucasbradstreet after I upgraded the error has not appeared again!

jeroenvandijk12:04:04

Do people run the Onyx-dashboard against an in-jvm development cluster? It doesn’t seem to work well. When running against the cluster generated by docker-compose it works fine

lucasbradstreet12:04:12

@jeroenvandijk: Yeah, it should work fine though if your ZooKeeper goes down in between runs you may have to restart the dashboard.

lucasbradstreet13:04:32

By the way I’ve upgraded the template to 0.9.4. It’s mostly to bump it to 0.9.4 but there’s a fix to the metrics in there too. Those were broken.

michaeldrogalis13:04:58

Managed to get 3/4 of the Onyx team in the same city. Now we just need to get in the same room. 😛

michaeldrogalis13:04:27

@jeroenvandijk @aspra Thanks for the bug report. simple_smile

jeroenvandijk14:04:22

@lucasbradstreet: Thanks for the upgrade of the template. I ran it over my existing application succesfully

jeroenvandijk14:04:59

The template doesn’t contain the Component repl functions. Is that on purpose? I think this is useful to add

jeroenvandijk14:04:30

I have copied the version here https://github.com/onyx-platform/onyx-starter/blob/0.9.x/env/dev/user.clj, but I suspect I did something wrong as I cannot connect with the dashboard properly when running (go)

gardnervickers14:04:41

The with-test-env macro ends up replacing a lot of the functionality that the user namespace used to offer

lucasbradstreet14:04:05

Yeah, generally we use with-test-env + clojure.tools.namespace.repl/refresh

jeroenvandijk14:04:31

Ah ok, will have a look. I saw something about it in the readme

lucasbradstreet14:04:02

@jeroenvandijk: maybe try adding a sleep to the _test file before the feedback exception call to keep the environment up long enough to connect to it with your dashboard?

jeroenvandijk14:04:05

i'm trying to start a cluster via the repl so I can submit jobs to it. I’m not trying to connect with the test env

lucasbradstreet14:04:19

Ah k, which is perfectly valid too

jeroenvandijk14:04:38

still need to use with-test-env for that?

lucasbradstreet14:04:51

Nope, though I think I would still tend to, personally.

lucasbradstreet14:04:05

At least for dev

jeroenvandijk14:04:20

ok will have a look. Probably good to know how that works anyway

lucasbradstreet14:04:40

What you’re doing is fine too, and worth getting the hang of

lucasbradstreet14:04:37

onyx.test-helper has an overall component that you could use to help you out with a component/repl reloaded type workflow https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/test_helper.clj#L106

lucasbradstreet14:04:44

That’s what’s used by with-test-env anyway

lucasbradstreet14:04:14

You could put that into your user.clj if you wish

jeroenvandijk14:04:17

Ah cool, that’s what I need simple_smile

jeroenvandijk14:04:51

Ah great, now i can connect with the dashboard. Probably some subtle difference

zamaterian15:04:19

Btw the onyx-seq README refers to :seq/elements-per-segment which is now deprecated

michaeldrogalis15:04:49

@zamaterian: Thanks, Ill get to that shortly. simple_smile

michaeldrogalis15:04:34

@zamaterian: Fixed, thanks for the heads up. ^^