aww the talk is at 9:00 pst. well, ill watch the recording.

@drewverlee: It moved pretty fast. It's probably better in recorded form, there's a ton of information.

Hey folks, as I was saying in the talk earlier we’re looking to make this project https://github.com/onyx-platform/onyx-twitter-sample a community repository showcasing how to do different things in Onyx. If we could come up with some interesting examples I think it might be fun to have a weekly or bi-weekly community driven hangout were we implement some new functionality as an Onyx job in that repository.

Maybe there should be a DEPRECATED notice or something on http://www.onyxplatform.org/jekyll/update/2016/03/21/Engraver-A-Tool-for-Managing-Onyx-Clusters.html ?

@devth that would be a good idea

@devn: I'll push that out now, thanks!

I'm half entertaining the idea of removing the blog post and repo altogether, but that seems a bit wasteful.

yeah, makes sense

Done. Thanks for letting us know.

great, np!

when using core async for local testing, can I only opt to use the output side? i'm trying to setup a test that reads datoms from an in-mem database via read-log (that still has a log, right?) and outputs directly to core.async. the output chan doesn't seem to be getting anything.

(-> base-job
        (add-task (datomic-task/input :read-datomic job-settings))
        (add-task (core-async-task/output :out job-settings)))

:workflow [[:read-datomic :out]]

@devth: Yes, you can only use core.async on the input or output side.

hm, ok.

that's what I assumed. before the test runs, i transact 10 datoms into the in-mem db

I would suspect that read-datomic is not working ok

What type of datomic plugin are you using?

You can add :onyx/fn to any task. You might want to drop a ::spy function that prints out a segment and returns it to your input task to debug

If it's read-log then it's because in mem doesn't support a Tx log

lucasbradstreet: oh. yeah it's read-log

:onyx/plugin :onyx.plugin.datomic/read-log

Cool. That's a datomic limitation

ok, guess i'll need to rethink how to test this. the main purpose was to look at the shape of a segment that read-log outputs so that I could start working on a transform fn

We should probably validate the conn string doesn't include mem://

I'll get that done. @devth yeah, in our CI tests we spin up a datomic container

makes sense. I'll switch to datomic:free for tests

@michaeldrogalis: thanks for the ::spy tip - sounds very useful

@devth: Surely 🙂

Nice catch @lucasbradstreet

datomic plugin now checks whether it’s being used with mem instances

nice! that was fast 🙂

better to just fix this stuff

feedback on the lein template: it'd be helpful if the generated basic-test included

(feedback-exception! peer-config job-id)

@devth: thank you. I believe we used to do that. Must have gotten lost in the latest refactor

I adapted basic-test for my datomic use case, but it silently failed until I figured out I needed to add this. then it helpfully told me datomic wasn't on the class path (forgot to (:require [onyx.plugin.datomic]))

paging @gardnervickers about the above feedback-exception! issue

yep, you used to because I used the generator a few days ago, then again yesterday 🙂

yeah, we usually recommend you tail the log too, but feedback-exception! is a lot nicer in general use

oh nice. i didn't realize onyx.log existed. all kinds of good stuff in there

(this is my first time using onyx 🙂 )

Yeah, that’s kind of the trade-off of making feedback-exception! so helpful!

so, my datomic/read-log test just hangs when I run it. I have a :onyx/fn ::spy and

(defn spy [segment]
  (prn "datomic segment" segment)
  segment)

16-Jun-10 08:52:37 transcend.local INFO [onyx.messaging.aeron.publication-pool] - Publication  GC'd after non use within 60000
16-Jun-10 08:52:37 transcend.local INFO [onyx.messaging.aeron.publication-manager] - Stopped publication manager at: , stream-id: 1

sounds like nothing was read, and nothing happened. You shouldn’t need to specify a log-end-tx

hm. i'm pointing to a local datomic:free transactor that I know has datoms in it (my webapp runs off it)

Yeah, that’s strange. Did you supply a log-start-tx?

nope

task looks like

(s/defschema DatomicInputTask
  {:datomic/uri s/Str
   :onyx/batch-size s/Num})

(defn spy [segment]
  (prn "datomic segment" segment)
  segment)

(s/defn input
  [task-name :- s/Keyword
   task-opts :- DatomicInputTask]
  {:task {:task-map (merge
                        {:onyx/name task-name
                         :onyx/plugin :onyx.plugin.datomic/read-log
                         :onyx/type :input
                         :onyx/medium :datomic
                         :onyx/fn ::spy
                         ;; :datomic/log-start-tx <<OPTIONAL_TX_START_INDEX>>
                         ;; :datomic/log-end-tx <<OPTIONAL_TX_END_INDEX>>

                         :checkpoint/force-reset? true
                         :onyx/max-peers 1
                         :onyx/doc "Reads a sequence of datoms from the d/log API"}
                        task-opts)
            :lifecycles [{:lifecycle/task task-name
                          :lifecycle/calls :onyx.plugin.datomic/read-log-calls}]}
     :schema {:task-map DatomicInputTask}})

You shouldn’t need to

That looks good. I assume you’re supplying the uri as part of your task opts?

yep. I verified it throws an exception when I provide an invalid uri

(then I gave it a valid uri)

just saw a

||  [{:type org.apache.bookkeeper.bookie.BookieException$InvalidCookieException
||    :message ""
||    :at [org.apache.bookkeeper.bookie.Bookie checkEnvironment "Bookie.java" 352]}]

I also killed my webapp to make sure I wasn't surpassing the 2 peer limit on the free transactor

that should certainly work

Ah, you may want to temporarily turn off bookkeeper startup if you’re not using windowing / state management yet

ok. yeah consistently getting

org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
    code: 0
                                         clojure.lang.ExceptionInfo: Error in component :bookkeeper in system onyx.system.OnyxDevelopmentEnv calling #'com.stuartsierra.component/start
     component: #<Bookie Servers>
      function: #'com.stuartsierra.component/start
        reason: :com.stuartsierra.component/component-function-threw-exception
        system: <#C051WKSP3>.system.OnyxDevelopmentEnv{:monitoring #<NoOp Monitoring Agent>, :logging-config #<Logging Configuration>, :bookkeeper #<Bookie Servers>, :log #<ZooKeeper Component>}
    system-key: :bookkeeper

you can set this in your env config: :onyx.bookkeeper/server? false

that will solve that for now

You will lose the ability to do windowing/state management though.

That is a problem with frequent restarts with BookKeeper

Sorry, I have to run for a bit. Hope that helps. read-log should work like you’re trying to use it

thanks for the help! I disabled bookkeeper and re-ran. still hangs. i'll keep playing with it

Yeah, I think that’s an unrelated issue, though it may get in the way of a solution

arg. I was running tests from a repl. apparently that doesn't work? when I lein test instead I get a bunch of segments 😖

That should work, but maybe you had started up a bunch of jobs and one of them was a bad one or something

it still hangs indefinitely but at least I'm seeing something

It should work just fine at the repl

ok

can i tell it to stop after consuming N segments, or timeout?

Yeah it'll hang I definitely because it'll never see a done because it's essentially a streaming job unless you set an end-tx

makes sense.

i guess I can set an end-tx

@devth: Nice catch on adding feedback-exception to job submission. Thanks!

gardnervickers: happy to help!

looks like the meetup example in the onyx-template changed recently. Any reason why?

I'm getting a bookkeeper error on the meetup data example now that I wasn't getting last week...

peer_1       | org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4
peer_1       |                                                                      bookieHost: "172.17.0.7:3196"
peer_1       |                                                                      journalDir: "/tmp/bookkeeper_journal/1_3196"
peer_1       |                                                                      ledgerDirs: "1\t/tmp/bookkeeper_ledger/1_3196"
peer_1       |                                                                      ] is not matching with [4
peer_1       |                                                                      bookieHost: "172.17.0.6:3196"
peer_1       |                                                                      journalDir: "/tmp/bookkeeper_journal/1_3196"
peer_1       |                                                                      ledgerDirs: "1\t/tmp/bookkeeper_ledger/1_3196"
peer_1       |                                                                      ]
peer_1       |     code: 0
peer_1       |                                          clojure.lang.ExceptionInfo: Error in component :bookkeeper in system onyx.system.OnyxDevelopmentEnv calling #'com.stuartsierra.component/start
peer_1       |     data: {:reason :com.stuartsierra.component/component-function-threw-exception, :function #'com.stuartsierra.component/start, :system-key :bookkeeper, :component <#C051WKSP3>.state.bookkeeper.BookieServers{:env-config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :log <#C051WKSP3>.log.zookeeper.ZooKeeper{:config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :monitoring <#C051WKSP3>.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config <#C051WKSP3>.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config nil}, :server nil, :conn #object[org.apache.curator.framework.imps.CuratorFrameworkImpl 0x264651eb "[email protected]"], :prefix "1"}}, :system <#C051WKSP3>.system.OnyxDevelopmentEnv{:monitoring <#C051WKSP3>.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config <#C051WKSP3>.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config nil}, :bookkeeper <#C051WKSP3>.state.bookkeeper.BookieServers{:env-config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :log nil}, :log <#C051WKSP3>.log.zookeeper.ZooKeeper{:config {:onyx/tenancy-id "1", :onyx.bookkeeper/server? true, :onyx.bookkeeper/delete-server-data? true, :onyx.bookkeeper/local-quorum? true, :onyx.bookkeeper/local-quorum-ports [3196 3197 3198], :zookeeper/address "zk:2181", :zookeeper/server? false, :zookeeper.server/port 2181}, :monitoring <#C051WKSP3>.monitoring.no_op_monitoring.NoOpMonitoringAgent{}, :logging-config <#C051WKSP3>.static.logging_configuration.LoggingConfiguration{:file "onyx.log", :config nil}, :server nil, :conn #object[org.apache.curator.framework.imps.CuratorFrameworkImpl 0x264651eb "[email protected]"], :prefix "1"}}}
peer_1       |