I am using the datomic read-log plugin and getting this exception

Handling uncaught exception thrown inside task lifecycle - killing this job. -> Exception type: clojure.lang.ExceptionInfo. Exception message: Unfreezable type: class clojure.lang.Delay

any ideas where I could look ?

Also this might be related, running the dashboard, this is all in local dev mode with an embedded zookeeper, I get these messages on the dashboard console

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /onyx/2/log-parameters/log-parameters
    code: -101
    path: "/onyx/2/log-parameters/log-parameters"

17-01-04 10:31:29 mung WARN [onyx.log.zookeeper:207] - Log parameters couldn't be discovered. Backing off 500ms and trying again...

How onyx coordinate between multiple input peers?

@len I believe we saw the Delay thing pop up before. @robert-stuttaford Do you recall what was happening there?

It gets 2 segments from datomic and then blows up

@len Re: dashboard. The dashboard needs a ZooKeeper connection to hold onto permanently. Local-dev uses embedded ZooKeeper and trashes the state and connection everytime.

Yeah, that’s familiar. Let me try searching the Slack logs.

What version of onyx-datomic and Onyx are you using?

yeah i start the dashboard after the local peers, says connected

[org.onyxplatform/onyx-datomic "0.9.15.0"]

Im not exactly sure where we landed with this one to be honest: https://clojurians-log.clojureverse.org/datomic/2016-10-04.html

thanks thats interesting - I have the latest datomic and will wait for @robert-stuttaford to achieve enlightenment 🙂

terribly sorry, but i believe we never got to the bottom of that

I think the dashboard log error is because no jobs have completed yet

is there a workaround @robert-stuttaford ?

so, it usually only happened because we were mucking about somehow - restarting things in the wrong order, that sort of thing

we haven’t had it in a long while

@len The dashboard message you’re seeing is unrelated. It’s having trouble connecting to a valid ZK tenancy.

@robert-stuttaford Are you still rebooting the peers nightly?

it sounds like you have a fairly reproducible case, @len. i’d put something together for Lucas to chew on

no, we stopped that a while ago 🙂

Right, I remember now. We couldn’t figure out exactly where the Delay was coming from because we can’t see Datomic’s source. 😛

yep. it was something to do with trying to read outside of the available range, or something

i believe we filed a bug with Datomic actually

Strange how it only comes up sometimes

by ‘we’ i mean we told them about it over in #datomic and one of the guys said ‘oh, right, cool, we’ll look into that'

I was thinking that its because I am starting at the beginning of time

I just got back to it now so will try some things and let you know

i do know that the 3 txes in an ‘empty’ db don’t show up in tx-range

the ones at the epoch

but nil as a start-t is documented as valid

good luck 🙂

Thanks @robert-stuttaford 🙂

ok so if I start the datomic log with a recent t then all is fine and I dont get the Delay

@len What is “recent” in your case?

I started with t from (last (d/tx-range (d/log conn) nil nil))

now working with t from the 1st jan

it looks like the initial txns in datomic from when we started last year august have the issue, not sure which ones, I will only need the t from when we start in production so looks ok for me

Okie doke. I really wish we had some visibility into that one slice of code. 😕

yeah that would help

ok so still getting those log errors from the dashboard - any pointers where to look ?

Can you describe your set up once more? You said you’re running embedded ZooKeeper. Is that through with-test-env?

this is a simple app from the lein template

just running in dev mode

so configs as per the template

starting the dashboard and connecting to 127.0.0.1:2188

dashboard start looks good

java -jar onyx-dashboard-0.9.15.0.jar 127.0.0.1:2188
=================================
Starting Dashboard components ...
Starting Sente
Starting Channels
Starting ZKClient
Trying connect ZK 5s ...
Starting Deployments
ZK connection state: CONNECTED
Starting HTTP Server
Http-kit server is running at 

Okay, that’s helpful. So you’re leaving that running and not taking it down, right?

I’m only asking because doing that would also stop the ZooKeeper server, which is required for the dashboard.

i restart it after the restart of the dev peers yeah

its only when I select the tenancy that I see the errors in the console

Are you sure the tenancies match?

yeah the dropdown is populated in the UI

I just select from that list

Sorry, a few more questions. Which version of the dashboard are you running?

Everything is 0.9.15.x I presume?

0.9.15 yes

although the template brings 0.9.10 ?

for the peers - should I bump that ?

Yeah those versions need to be upgraded. The template project.clj was too difficult to get into our auto-release process.

will do and will let you know thanks

lib-onyx to 0.9.15 as well ?

Yeah. Keep me posted, we’ll get you up and running.

ok upgraded all to 0.9.15 and still the same scenario with the dashboard

mmm ok switched to the old tenancy id and i get data

Okay. So the next thing you can do is pop open a ZooKeeper console and verify that the error is valid. It’s taking it can’t find a znode on /onyx/2/log-parameters. You can check if that exists with zkCli, then ls /onyx/2

If you were hopping between versions, you might want to start with a brand new tenancy ID.

how can i clear everything out ?

I am new to zookeeper as well 🙂

rmr /onyx in zkCli

👍

ok cleaned out and restarted everything and found and issue on my side - had differents tennacy ids for the env and peer configs !

all looks good now thanks for the assists

@len For sure 🙂