Fork me on GitHub
#onyx
<
2016-06-10
>
Drew Verlee00:06:02

aww the talk is at 9:00 pst. well, ill watch the recording.

michaeldrogalis01:06:02

@drewverlee: It moved pretty fast. It's probably better in recorded form, there's a ton of information.

gardnervickers01:06:14

Hey folks, as I was saying in the talk earlier we’re looking to make this project https://github.com/onyx-platform/onyx-twitter-sample a community repository showcasing how to do different things in Onyx. If we could come up with some interesting examples I think it might be fun to have a weekly or bi-weekly community driven hangout were we implement some new functionality as an Onyx job in that repository.

lucasbradstreet13:06:53

@devth that would be a good idea

michaeldrogalis13:06:14

@devn: I'll push that out now, thanks!

michaeldrogalis13:06:19

I'm half entertaining the idea of removing the blog post and repo altogether, but that seems a bit wasteful.

devth13:06:17

yeah, makes sense

michaeldrogalis13:06:14

Done. Thanks for letting us know.

devth13:06:48

great, np!

devth14:06:49

when using core async for local testing, can I only opt to use the output side? i'm trying to setup a test that reads datoms from an in-mem database via read-log (that still has a log, right?) and outputs directly to core.async. the output chan doesn't seem to be getting anything.

devth14:06:22

(-> base-job
        (add-task (datomic-task/input :read-datomic job-settings))
        (add-task (core-async-task/output :out job-settings)))

devth14:06:35

:workflow [[:read-datomic :out]]

michaeldrogalis14:06:39

@devth: Yes, you can only use core.async on the input or output side.

devth14:06:31

that's what I assumed. before the test runs, i transact 10 datoms into the in-mem db

lucasbradstreet14:06:37

I would suspect that read-datomic is not working ok

lucasbradstreet14:06:47

What type of datomic plugin are you using?

michaeldrogalis14:06:03

You can add :onyx/fn to any task. You might want to drop a ::spy function that prints out a segment and returns it to your input task to debug

lucasbradstreet14:06:08

If it's read-log then it's because in mem doesn't support a Tx log

devth14:06:27

lucasbradstreet: oh. yeah it's read-log

devth14:06:35

:onyx/plugin :onyx.plugin.datomic/read-log

lucasbradstreet14:06:37

Cool. That's a datomic limitation

devth14:06:26

ok, guess i'll need to rethink how to test this. the main purpose was to look at the shape of a segment that read-log outputs so that I could start working on a transform fn

lucasbradstreet14:06:39

We should probably validate the conn string doesn't include mem://

lucasbradstreet14:06:13

I'll get that done. @devth yeah, in our CI tests we spin up a datomic container

devth14:06:40

makes sense. I'll switch to datomic:free for tests

devth14:06:55

@michaeldrogalis: thanks for the ::spy tip - sounds very useful

lucasbradstreet14:06:31

datomic plugin now checks whether it’s being used with mem instances

devth14:06:41

nice! that was fast 🙂

lucasbradstreet14:06:34

better to just fix this stuff

devth14:06:19

feedback on the lein template: it'd be helpful if the generated basic-test included

(feedback-exception! peer-config job-id)

lucasbradstreet14:06:05

@devth: thank you. I believe we used to do that. Must have gotten lost in the latest refactor

devth14:06:14

I adapted basic-test for my datomic use case, but it silently failed until I figured out I needed to add this. then it helpfully told me datomic wasn't on the class path (forgot to (:require [onyx.plugin.datomic]))

lucasbradstreet14:06:24

paging @gardnervickers about the above feedback-exception! issue

devth14:06:26

yep, you used to because I used the generator a few days ago, then again yesterday 🙂

lucasbradstreet14:06:44

yeah, we usually recommend you tail the log too, but feedback-exception! is a lot nicer in general use

devth14:06:44

oh nice. i didn't realize onyx.log existed. all kinds of good stuff in there

devth14:06:54

(this is my first time using onyx 🙂 )

lucasbradstreet14:06:20

Yeah, that’s kind of the trade-off of making feedback-exception! so helpful!

devth14:06:31

so, my datomic/read-log test just hangs when I run it. I have a :onyx/fn ::spy and

(defn spy [segment]
  (prn "datomic segment" segment)
  segment)
but I don't see anything in the logs. after 60 seconds onyx.log says:
16-Jun-10 08:52:37 transcend.local INFO [onyx.messaging.aeron.publication-pool] - Publication  GC'd after non use within 60000
16-Jun-10 08:52:37 transcend.local INFO [onyx.messaging.aeron.publication-manager] - Stopped publication manager at: , stream-id: 1
but the test continues to hang. do I need to specify a datomic/log-end-tx? was hoping to just sample some of the log.

lucasbradstreet14:06:00

sounds like nothing was read, and nothing happened. You shouldn’t need to specify a log-end-tx

devth14:06:25

hm. i'm pointing to a local datomic:free transactor that I know has datoms in it (my webapp runs off it)

lucasbradstreet14:06:30

Yeah, that’s strange. Did you supply a log-start-tx?

devth14:06:31

task looks like

(s/defschema DatomicInputTask
  {:datomic/uri s/Str
   :onyx/batch-size s/Num})

(defn spy [segment]
  (prn "datomic segment" segment)
  segment)

(s/defn input
  [task-name :- s/Keyword
   task-opts :- DatomicInputTask]
  {:task {:task-map (merge
                        {:onyx/name task-name
                         :onyx/plugin :onyx.plugin.datomic/read-log
                         :onyx/type :input
                         :onyx/medium :datomic
                         :onyx/fn ::spy
                         ;; :datomic/log-start-tx <<OPTIONAL_TX_START_INDEX>>
                         ;; :datomic/log-end-tx <<OPTIONAL_TX_END_INDEX>>

                         :checkpoint/force-reset? true
                         :onyx/max-peers 1
                         :onyx/doc "Reads a sequence of datoms from the d/log API"}
                        task-opts)
            :lifecycles [{:lifecycle/task task-name
                          :lifecycle/calls :onyx.plugin.datomic/read-log-calls}]}
     :schema {:task-map DatomicInputTask}})

lucasbradstreet14:06:15

You shouldn’t need to

lucasbradstreet14:06:50

That looks good. I assume you’re supplying the uri as part of your task opts?

devth15:06:07

yep. I verified it throws an exception when I provide an invalid uri

devth15:06:16

(then I gave it a valid uri)

devth15:06:32

just saw a

||  [{:type org.apache.bookkeeper.bookie.BookieException$InvalidCookieException
||    :message ""
||    :at [org.apache.bookkeeper.bookie.Bookie checkEnvironment "Bookie.java" 352]}]

devth15:06:57

I also killed my webapp to make sure I wasn't surpassing the 2 peer limit on the free transactor

lucasbradstreet15:06:14

that should certainly work

lucasbradstreet15:06:44

Ah, you may want to temporarily turn off bookkeeper startup if you’re not using windowing / state management yet

devth15:06:11

ok. yeah consistently getting

org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
    code: 0
                                         clojure.lang.ExceptionInfo: Error in component :bookkeeper in system onyx.system.OnyxDevelopmentEnv calling #'com.stuartsierra.component/start
     component: #<Bookie Servers>
      function: #'com.stuartsierra.component/start
        reason: :com.stuartsierra.component/component-function-threw-exception
        system: <#C051WKSP3>.system.OnyxDevelopmentEnv{:monitoring #<NoOp Monitoring Agent>, :logging-config #<Logging Configuration>, :bookkeeper #<Bookie Servers>, :log #<ZooKeeper Component>}
    system-key: :bookkeeper
now

lucasbradstreet15:06:47

you can set this in your env config: :onyx.bookkeeper/server? false

lucasbradstreet15:06:03

that will solve that for now

lucasbradstreet15:06:12

You will lose the ability to do windowing/state management though.

lucasbradstreet15:06:48

That is a problem with frequent restarts with BookKeeper

lucasbradstreet15:06:12

Sorry, I have to run for a bit. Hope that helps. read-log should work like you’re trying to use it

devth15:06:41

thanks for the help! I disabled bookkeeper and re-ran. still hangs. i'll keep playing with it

lucasbradstreet15:06:11

Yeah, I think that’s an unrelated issue, though it may get in the way of a solution

devth15:06:59

arg. I was running tests from a repl. apparently that doesn't work? when I lein test instead I get a bunch of segments 😖

lucasbradstreet15:06:33

That should work, but maybe you had started up a bunch of jobs and one of them was a bad one or something

devth15:06:40

it still hangs indefinitely but at least I'm seeing something

lucasbradstreet15:06:42

It should work just fine at the repl

devth15:06:06

can i tell it to stop after consuming N segments, or timeout?

lucasbradstreet15:06:11

Yeah it'll hang I definitely because it'll never see a done because it's essentially a streaming job unless you set an end-tx

devth15:06:17

makes sense.

devth15:06:33

i guess I can set an end-tx

gardnervickers15:06:45

@devth: Nice catch on adding feedback-exception to job submission. Thanks!

devth15:06:01

gardnervickers: happy to help!

aaelony22:06:51

looks like the meetup example in the onyx-template changed recently. Any reason why?

aaelony22:06:02

I'm getting a bookkeeper error on the meetup data example now that I wasn't getting last week...

peer_1       | org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
peer_1       |                                                                      bookieHost: "172.17.0.7:3196"
peer_1       |                                                                      journalDir: "/tmp/bookkeeper_journal/1_3196"
peer_1       |                                                                      ledgerDirs: "1\t/tmp/bookkeeper_ledger/1_3196"
peer_1       |                                                                      ] is not matching with [4
peer_1       |                                                                      bookieHost: "172.17.0.6:3196"
peer_1       |                                                                      journalDir: "/tmp/bookkeeper_journal/1_3196"
peer_1       |                                                                      ledgerDirs: "1\t/tmp/bookkeeper_ledger/1_3196"
peer_1       |                                                                      ]
peer_1       |     code: 0
peer_1       |                                          clojure.lang.ExceptionInfo: Error in component :bookkeeper in system onyx.system.OnyxDevelopmentEnv calling #'com.stuartsierra.component/start
peer_1       |     data: {:reason :com.stuartsierra.component/component-function-threw-exception, :function #'com.stuartsierra.component/start, :system-key :bookkeeper, :component <#C051WKSP3>.state.bookkeeper.BookieServers{:env-config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :log <#C051WKSP3>.log.zookeeper.ZooKeeper{:config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :monitoring <#C051WKSP3>.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config <#C051WKSP3>.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config nil}, :server nil, :conn #object[org.apache.curator.framework.imps.CuratorFrameworkImpl 0x264651eb "org.apache.curator.framework.imps.CuratorFrameworkImpl@264651eb"], :prefix "1"}}, :system <#C051WKSP3>.system.OnyxDevelopmentEnv{:monitoring <#C051WKSP3>.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config <#C051WKSP3>.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config nil}, :bookkeeper <#C051WKSP3>.state.bookkeeper.BookieServers{:env-config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :log nil}, :log <#C051WKSP3>.log.zookeeper.ZooKeeper{:config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :monitoring <#C051WKSP3>.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config <#C051WKSP3>.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config nil}, :server nil, :conn #object[org.apache.curator.framework.imps.CuratorFrameworkImpl 0x264651eb "org.apache.curator.framework.imps.CuratorFrameworkImpl@264651eb"], :prefix "1"}}}
peer_1       |