Fork me on GitHub
#onyx
<
2016-06-18
>
aaelony00:06:52

hi - me again. We have Twitter keys now and am trying the Twitter example

aaelony00:06:05

Getting another error:

peer_1       | Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /onyx/1/ledgers/cookies/172.17.0.4:3196 

aaelony00:06:26

that's the only error I can find... This is using the https://github.com/onyx-platform/onyx-twitter-sample example

michaeldrogalis00:06:55

@aaelony: Did the peer fail to start up due to that exception?

aaelony00:06:47

peer_1       | 16-Jun-18 00:11:12 5c9dcffe2327 INFO [onyx.log.failure-detector] - Starting peer failure detector 

aaelony00:06:00

peer_1       | 16-Jun-18 00:11:12 5c9dcffe2327 INFO [onyx.log.failure-detector] - Starting peer failure detector
peer_1       | 16-Jun-18 00:11:12 5c9dcffe2327 FATAL [onyx.system] -
peer_1       |                                       twit.core.main
peer_1       |                                                  ...
peer_1       |                                      twit.core/-main              core.clj:   69
peer_1       |                                      twit.core/-main              core.clj:   79
peer_1       |                             lib-onyx.peer/start-peer              peer.clj:    9
peer_1       |                                   onyx.api/start-env               api.clj:  316
peer_1       |                                   onyx.api/start-env               api.clj:  319
peer_1       |                 onyx.system.OnyxDevelopmentEnv/start            system.clj:   88
peer_1       |                        onyx.system/rethrow-component            system.clj:   80
peer_1       |                    onyx.system.OnyxDevelopmentEnv/fn            system.clj:   89
peer_1       |              com.stuartsierra.component/start-system        component.cljc:  163
peer_1       |                                                  ...
peer_1       |             com.stuartsierra.component/update-system        component.cljc:  129
peer_1       |             com.stuartsierra.component/update-system        component.cljc:  135
peer_1       |                                  clojure.core/reduce              core.clj: 6544
peer_1       |                                                  ...
peer_1       |          com.stuartsierra.component/update-system/fn        component.cljc:  139
peer_1       |                com.stuartsierra.component/try-action        component.cljc:  117
peer_1       |                                   clojure.core/apply              core.clj:  648
peer_1       |                                                  ...
peer_1       |                      com.stuartsierra.component/fn/G        component.cljc:    5 (repeats 2 times)
peer_1       |            onyx.state.bookkeeper.BookieServers/start        bookkeeper.clj:  108
peer_1       |                                    clojure.core/mapv              core.clj: 6618
peer_1       |                                  clojure.core/reduce              core.clj: 6544
peer_1       |                                                  ...
peer_1       |                                 clojure.core/mapv/fn              core.clj: 6627
peer_1       |               onyx.state.bookkeeper.BookieServers/fn        bookkeeper.clj:  109
peer_1       |            onyx.state.bookkeeper.BookieMonitor/start        bookkeeper.clj:   75
peer_1       |                   onyx.state.bookkeeper.Bookie/start        bookkeeper.clj:   37
peer_1       |      org.apache.bookkeeper.proto.BookieServer.<init>     BookieServer.java:   87
peer_1       |      org.apache.bookkeeper.proto.BookieServer.<init>     BookieServer.java:   95
peer_1       |   org.apache.bookkeeper.proto.BookieServer.newBookie     BookieServer.java:  107
peer_1       |           org.apache.bookkeeper.bookie.Bookie.<init>           Bookie.java:  466
peer_1       | org.apache.bookkeeper.bookie.Bookie.checkEnvironment           Bookie.java:  362
peer_1       | org.apache.bookkeeper.bookie.Cookie.writeToZooKeeper           Cookie.java:  213
peer_1       |                org.apache.zookeeper.ZooKeeper.create        ZooKeeper.java:  783
peer_1       |          org.apache.zookeeper.KeeperException.create  KeeperException.java:   51
peer_1       |          org.apache.zookeeper.KeeperException.create  KeeperException.java:  119
peer_1       |            org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /onyx/1/ledgers/cookies/172.17.0.4:3197
peer_1       |     code: -110
peer_1       |     path: "/onyx/1/ledgers/cookies/172.17.0.4:3197" 

aaelony00:06:58

peer_1       | org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /onyx/1/ledgers/cookies/172.17.0.4:3197
peer_1       |     code: 0
peer_1       |                                          clojure.lang.ExceptionInfo: Error in component :bookkeeper in system onyx.system.OnyxDevelopmentEnv calling #'com.stuartsierra.component/start
peer_1       |      component: #<Bookie Servers>
peer_1       |       function: #'com.stuartsierra.component/start
peer_1       |         reason: :com.stuartsierra.component/component-function-threw-exception
peer_1       |         system: <#C051WKSP3>.system.OnyxDevelopmentEnv{:monitoring #<NoOp Monitoring Agent>, :logging-config #<Logging Configuration>, :bookkeeper #<Bookie Servers>, :log #<ZooKeeper Component>}
peer_1       |     system-key: :bookkeeper
peer_1       |
peer_1       | Exception in thread "main" org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /onyx/1/ledgers/cookies/172.17.0.4:3197
peer_1       | 	at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:366)
peer_1       | 	at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:466)
peer_1       | 	at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:107)
peer_1       | 	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:95)
peer_1       | 	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:87)
peer_1       | 	at onyx.state.bookkeeper.Bookie.start(bookkeeper.clj:37)
peer_1       | 	at onyx.state.bookkeeper.BookieMonitor.start(bookkeeper.clj:75)
peer_1       | 	at onyx.state.bookkeeper.BookieServers$fn__16430.invoke(bookkeeper.clj:109)
peer_1       | 	at clojure.core$mapv$fn__6953.invoke(core.clj:6627)
peer_1       | 	at clojure.lang.PersistentVector.reduce(PersistentVector.java:341)
peer_1       | 	at clojure.core$reduce.invokeStatic(core.clj:6544)
peer_1       | 	at clojure.core$mapv.invokeStatic(core.clj:6618)
peer_1       | 	at clojure.core$mapv.invoke(core.clj:6618)
peer_1       | 	at onyx.state.bookkeeper.BookieServers.start(bookkeeper.clj:108)
peer_1       | 	at com.stuartsierra.component$fn__6831$G__6825__6833.invoke(component.cljc:5)
peer_1       | 	at com.stuartsierra.component$fn__6831$G__6824__6836.invoke(component.cljc:5)
peer_1       | 	at clojure.lang.Var.invoke(Var.java:379)
peer_1       | 	at clojure.lang.AFn.applyToHelper(AFn.java:154)
peer_1       | 	at clojure.lang.Var.applyTo(Var.java:700)
peer_1       | 	at clojure.core$apply.invokeStatic(core.clj:648)
peer_1       | 	at clojure.core$apply.invoke(core.clj:641) 

aaelony00:06:02

hmmm, not sure where /tmp/bookkeeper_journal and /tmp/bookkeeper_ledger are located

aaelony00:06:58

some container perhaps

gardnervickers00:06:36

It's on your docker host

aaelony00:06:25

what do you mean by host? it's not docker nor docker-machine

gardnervickers00:06:50

Your using docker-compose up right?

aaelony00:06:16

on a mac that is using linux

aaelony00:06:24

well, a mac that uses docker-machine

gardnervickers00:06:28

Where are your containers running

aaelony00:06:31

and the container is linux

gardnervickers00:06:54

Ok yea then you can clear the /tmp directory on your docker-machine host

gardnervickers00:06:14

Coming home from the gym, I can help more in a few minutes

aaelony00:06:12

$  docker-machine ssh aa-machine
                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___ \ __| | ___   ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.11.2, build HEAD : a6645c3 - Wed Jun  1 22:59:51 UTC 2016
Docker version 1.11.2, build b9f10c9
[email protected]:~$ ls -hl /tmp
lrwxrwxrwx    1 root     root          13 Jun 17 23:00 /tmp -> /mnt/sda1/tmp/
[email protected]:~$ df -kh /tmp
Filesystem                Size      Used Available Use% Mounted on
/dev/sda1                17.9G      3.3G     13.7G  19% /mnt/sda1 

aaelony00:06:25

looks fine?

gardnervickers00:06:50

Your using an unmodified clone of the onyx Twitter sample?

aaelony00:06:18

basically, just cloned the repo then did:

Set your twitter API keys in docker-compose.yaml
    lein do clean, uberjar; docker build -t peerimage .
    docker-compose up
    docker-compose run --entrypoint=java peer "-cp" "/opt/peer.jar" "twit.core" "submit-job" "trending-hashtags" "-p" ":docker"
 

aaelony00:06:41

(with real Twitter API keys... hopefully)

gardnervickers00:06:02

try docker-compose rm -v

gardnervickers00:06:19

then docker-compose up

gardnervickers00:06:27

-v drops the volumes

aaelony00:06:30

$  docker-compose rm -v
WARNING: Not including one-off containers created by `docker-compose run`.
To include them, use `docker-compose rm --all`.
This will be the default behavior in the next version of Compose.

No stopped containers

gardnervickers00:06:49

docker-compose kill first

aaelony00:06:03

$  docker-compose kill
Killing onyxtwittersample_peer_1 ... done
Killing onyxtwittersample_mysql_1 ... done
Killing onyxtwittersample_zookeeper_1 ... done

gardnervickers00:06:11

alright then drop the containers and their volumes

gardnervickers00:06:15

that’ll get you into a clean state

gardnervickers00:06:18

docker-compose rm -v

aaelony00:06:46

yeah, after the kill, did docker-compose rm -v, then docker-compose up

aaelony00:06:28

it output a bunch of messages to the screen, but now is stuck after:

aaelony00:06:59

is that normal?

michaeldrogalis00:06:08

That's a clean start.

aaelony00:06:28

so I should see the data somewhere?

gardnervickers00:06:53

so the peers booted up?

gardnervickers00:06:06

now you have to submit a job, the cluster is currently in a waiting state

gardnervickers00:06:22

docker-compose run --entrypoint=java peer "-cp" "/opt/peer.jar" "twit.core" "submit-job" "trending-hashtags" "-p" ":docker”

aaelony00:06:24

do I need a new window to submit a job?

gardnervickers00:06:39

Yea if you cancel the window with docker compose the containers will come down

aaelony00:06:52

I should have run it as a daemon I guess

gardnervickers00:06:56

for future reference, -d starts the docker-compose setup in “daemon” mode

gardnervickers00:06:17

so once that comes back, you should see a job id if everything was successful

gardnervickers00:06:36

then you can look in the Trending table in the onyx db that was started up

aaelony00:06:37

looks good

aaelony00:06:11

what's the best way to view the Trending table? hunting for that...

gardnervickers00:06:21

if your on a mac, mycli is pretty good

gardnervickers00:06:25

brew install mycli

aaelony00:06:44

is that an sql client?

gardnervickers00:06:47

mycli -h 192.168.99.101 -u admin -p “mypass"

aaelony00:06:36

that just hangs

gardnervickers00:06:58

uhh try it with 192.168.99.100

gardnervickers00:06:06

i dont know where your docker machine is

aaelony00:06:07

$  docker-machine ls
NAME         ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
aa-machine   *        virtualbox   Running              v1.11.2

aaelony00:06:32

nevermind... I'm in

aaelony00:06:38

there was a trailing dash

aaelony00:06:21

Trending is empty 😞

gardnervickers00:06:45

And your peer logs look fine?

aaelony00:06:58

how can I tell?

aaelony00:06:27

I think I need to read for docs and get back to you

gardnervickers00:06:37

No this is a docker-compose thing

gardnervickers00:06:53

docker-compose logs

aaelony00:06:53

yeah, I mean learn about docker more

aaelony00:06:16

$ docker-compose logs | grep error peer_1 | # A fatal error has been detected by the Java Runtime Environment: peer_1 | # An error report file with more information is saved as:

aaelony00:06:28

but yeah...

aaelony00:06:38

thanks for your help

aaelony00:06:47

it's just a docker thing I need to read up on

gardnervickers00:06:10

Can I get the output from that, that’s most likely my problem

aaelony00:06:46

ok, enough for now.. got to run. thanks for your help.. just need to study up on docker 🙂