Fork me on GitHub
#onyx
<
2016-12-05
>
agile_geek09:12:29

I want to submit an Onyx talk to Devoxx this year but I need to do a bit more collaboration with @jasonbell to build my experience. Current role doesn't relaly have a use case for Onyx unfortunately!

yonatanel13:12:04

Is the onyx-local-rt runtime itself (whatever is returned from (api/init job) etc) data only and can be in Om versioned state? (it means it will have time travel)

Travis14:12:06

Hey guys, we are getting this error all of a sudden but not really sure what is causing this?

Exception in thread "main" clojure.lang.ExceptionInfo: empty String {:original-exception :java.lang.NumberFormatException}
	at onyx.compression.nippy$fn__10918$fn__10919.invoke(nippy.clj:33)
	at taoensso.nippy$read_custom_BANG_.invokeStatic(nippy.clj:1052)
	at taoensso.nippy$read_custom_BANG_.invoke(nippy.clj:1049)
	at taoensso.nippy$thaw_from_in_BANG_.invokeStatic(nippy.clj:1218)
	at taoensso.nippy$thaw_from_in_BANG_.invoke(nippy.clj:1098)
	at taoensso.nippy$thaw$thaw_data__10761.invoke(nippy.clj:1330)
	at taoensso.nippy$thaw.invokeStatic(nippy.clj:1356)
	at taoensso.nippy$thaw.invoke(nippy.clj:1279)
	at onyx.compression.nippy$zookeeper_decompress.invokeStatic(nippy.clj:56)
	at onyx.compression.nippy$zookeeper_decompress.invoke(nippy.clj:55)
	at onyx.log.zookeeper$fn__16795$fn__16797$fn__16798.invoke(zookeeper.clj:564)
	at onyx.log.zookeeper$clean_up_broken_connections.invokeStatic(zookeeper.clj:77)
	at onyx.log.zookeeper$clean_up_broken_connections.invoke(zookeeper.clj:75)
	at onyx.log.zookeeper$fn__16795$fn__16797.invoke(zookeeper.clj:561)
	at onyx.monitoring.measurements$measure_latency.invokeStatic(measurements.clj:11)
	at onyx.monitoring.measurements$measure_latency.invoke(measurements.clj:5)
	at onyx.log.zookeeper$fn__16795.invokeStatic(zookeeper.clj:560)
	at onyx.log.zookeeper$fn__16795.doInvoke(zookeeper.clj:558)
	at clojure.lang.RestFn.invoke(RestFn.java:445)
	at clojure.lang.MultiFn.invoke(MultiFn.java:238)
	at onyx.test_helper$feedback_exception_BANG_.invokeStatic(test_helper.clj:24)
	at onyx.test_helper$feedback_exception_BANG_.invoke(test_helper.clj:13)
	at onyx.test_helper$feedback_exception_BANG_.invokeStatic(test_helper.clj:19)
	at onyx.test_helper$feedback_exception_BANG_.invoke(test_helper.clj:13)

gardnervickers14:12:41

@camechis Is this happening in your peer or job submit process?

Travis14:12:10

the error is being reported back in the submit job process

gardnervickers14:12:10

That’s an exception from your job being deserialized from Zookeeper into the job submit process.

Travis14:12:32

i think we may have found it

Travis14:12:01

there is an uncaught exception in our logs in a lifecycle that bubbled up. Just was buried in the logs so we missed it before

yonatanel16:12:35

So far I've made a simple om.next app with onyx-local-rt, hard-coded job, textarea with env-summary, and buttons for commands and time travel: Reset, <-Revert, Tick->, Drain and Stop. Does that interest anyone? It's just the beginning and instead of the sample job from onyx-local-rt readme I'm gonna implement my own aggregations.

michaeldrogalis16:12:05

@yonatanel Almost! It has a few compiled functions inside of it.

yonatanel16:12:21

Funny to see the lockstep behavior when setting batch-size to 1

michaeldrogalis16:12:58

@yonatanel That’s intriguing. 🙂 Curious to get a look when you’re ready, sounds brilliant.

yonatanel16:12:28

@michaeldrogalis It works anyway. I don't actually need this to be data only, but at least not shared among instances

yonatanel16:12:42

not anything mutable anyway

yonatanel16:12:45

Let's hope my employers feel the same

michaeldrogalis16:12:25

Heh. Is it a demo for them or something?

yonatanel16:12:23

It's a dev tool for me, and might become a debugging tool for production

yonatanel16:12:45

For instance if we don't understand why some data is the way it is, if command sourcing is in place I could just step through them.

michaeldrogalis16:12:48

Awesome. Definitely interesting in seeing it if/when you want to put it out there.

yonatanel17:12:31

That's not fun if you're inexperienced in cljs: referencing :cljs.core/identity instead of clojure.core/identity

michaeldrogalis17:12:28

@yonatanel I actually didn’t know there was a difference. cljs == JS identity?

yonatanel17:12:59

frankly i have no idea. I just needed an identity task to do aggregation on original segments

yonatanel17:12:08

I think it behaves the same as clojure, but the fully qualified name is different and that's what onyx wants

michaeldrogalis17:12:32

Huh. I’ll keep my eyes out for it sometime.

stephenmhopper17:12:12

Is there a well defined way for pulling data from APIs via HTTP into Onyx? I see an onyx-http output plugin. What's the preferred way to do something similar for gathering input?

michaeldrogalis17:12:16

@stephenmhopper For the most part, Onyx is optimal for input streams that are durable and can be replayed verbatim. What’s the nature of the API you’re working with? Is it a REST API? Is there a sequence of calls you want to make across it?

stephenmhopper17:12:41

That makes sense

stephenmhopper18:12:00

I'll be polling endpoints periodically and want to process any new events from those endpoints

michaeldrogalis18:12:00

Sometimes it’s a good idea to stream in a sequence of API call descriptions through Kafka, and use a downstream tasks to invoke the actual call.

stephenmhopper18:12:28

So I'm thinking of storing them to an intermediate storage location first like a DB of sorts

stephenmhopper18:12:30

Kafka might work too

michaeldrogalis18:12:00

Sure, yeah that works just as well. You get the idea.

yonatanel18:12:42

Is trigger/sync event param supposed to be nil in any scenario?

michaeldrogalis18:12:35

@yonatanel It should never be, no. You see it happening?

yonatanel18:12:19

Yes. I have this:

(defn ^:export dump-aggregate [event window trigger opts state]
  (println "dump-aggregate")
  (println "state")
  (pprint state)
  (println "event")
  (pprint event))
;...
:triggers
   [{:trigger/window-id :
     :trigger/on :onyx.triggers/segment
     :trigger/threshold [1 :elements]
     :trigger/refinement :onyx.refinements/accumulating
     :trigger/sync ::dump-aggregate}]
And event is printed as nil all the time

michaeldrogalis18:12:47

Hum. That’s a bug for sure.

yonatanel18:12:15

I'll try to debug it. Too rusty on the browser right now :)

yonatanel18:12:59

Obviously I know nothing about this. Do you know how I get rid of this error?

Compiling "resources/public/js/compiled/onyx_cqrs_example.js" from ["src"]...
Failed to compile "resources/public/js/compiled/onyx_cqrs_example.js" in 2.047 seconds.
----  Could not Analyze  resources/public/js/compiled/out/onyx/spec.cljc  ----

  No such namespace: clojure.future, could not locate clojure/future.cljs, clojure/future.cljc, or Closure namespace "clojure.future" 

yonatanel18:12:15

I'm using lein checkouts feature to use my own onyx-local-rt

yonatanel18:12:34

Oh, I'm using figwheel and onyx-local-rt probably isn't configured for that or clojurescript build. Maybe I can learn from onyx-blueprint

michaeldrogalis18:12:01

@yonatanel Bring in the dependency [clojure-future-spec “1.9.0-alpha14”]. We flipped this to a dev dependency in core last release since we can’t use Clojure-1.9-alpha yet

michaeldrogalis18:12:11

Sorry, we forgot to document that I think

michaeldrogalis18:12:49

I’m itching to move the whole local-rt into core, but I want to see 1.9 come out properly first.

lucasbradstreet18:12:47

Could we do a manual release of onyx local rt? I think that'd solve the issue

yonatanel18:12:54

I thought onyx-local-rt uses an onyx branch which fixes it

lucasbradstreet18:12:50

It's on main line onyx now

michaeldrogalis18:12:53

Ill cut a release now

michaeldrogalis18:12:30

Odd, that dependency is already in local-rt

lucasbradstreet18:12:43

I seem to remember about being able to do Clojure version reader conditions just like we can do cljc vs clj

lucasbradstreet18:12:51

Any idea of that ever made it in?

yonatanel18:12:54

I see [clojure-future-spec "1.9.0-alpha13"]

lucasbradstreet18:12:00

It is but did we do a release?

lucasbradstreet18:12:24

I got in on circle and building but didn't add it to the release process

michaeldrogalis18:12:35

Ah. Yeah Ill cut it now

yonatanel18:12:01

If i'm not mistaken, the wanted dependency should come earlier in the list, before onyx.

michaeldrogalis18:12:38

By list, you mean in :dependencies?

lucasbradstreet18:12:40

I don't think that matter

michaeldrogalis18:12:46

Yeah I dont believe it does.

lucasbradstreet18:12:52

Either in deps or in require

michaeldrogalis18:12:56

[org.onyxplatform/onyx-local-rt “0.9.15.0”] is out

yonatanel18:12:59

I'm getting a nasty "Unable to resolve symbol: import in this context, compiling:(clojure/core_instant18.clj:11:1)". Not sure why though

colinhicks18:12:03

this defines the clojure.future ns for ClojuresScript, only. The clojure-future-spec dep is Clojure-only.

michaeldrogalis18:12:22

@yonatanel As a result of upgrading?

michaeldrogalis18:12:59

Another reason I’m chomping at the bit for 1.9 😕

colinhicks18:12:09

Yep. Over in CLJS land, spec is implemented in the latest, non-alpha builds. Makes things much nicer.

yonatanel18:12:19

Perhaps, but also due to using latest clojure and cljs and om. Happens with this:

:dependencies [[org.clojure/clojure "1.9.0-alpha14"]
                 [org.clojure/clojurescript "1.9.293"]
                 [org.omcljs/om "1.0.0-alpha47"]
                 [org.onyxplatform/onyx-local-rt "0.9.15.0"]]
Doesn't happen with these defaults from lein new figwheel:
:dependencies [[org.clojure/clojure "1.8.0"]
                 [org.clojure/clojurescript "1.9.229"]
                 [org.clojure/core.async "0.2.391"
                  :exclusions [org.clojure/tools.reader]]
                 [org.onyxplatform/onyx-local-rt "0.9.15.0"]]

colinhicks18:12:33

I think your latest error is a conflict between clojure 1.9.0-alpha14 and the clojure-future-spec dep in onyx-local-rt

michaeldrogalis18:12:42

@yonatanel You should be able to exclude clojure-future-spec and be okay

michaeldrogalis18:12:55

Talk about a dependency cage match

colinhicks19:12:29

Yeah. And you might also have to stub-in the clojure.future ns as shown in my link above.

yonatanel19:12:45

Yes, because now I get

Failed to compile "resources/public/js/compiled/onyx_cqrs_example.js" in 3.409 seconds.
----  Could not Analyze  resources/public/js/compiled/out/onyx/spec.cljc  ----

  No such namespace: clojure.future, could not locate clojure/future.cljs, clojure/future.cljc, or Closure namespace "clojure.future"

colinhicks19:12:57

try adding that in src/clojure/future.cljs

michaeldrogalis19:12:10

Bit of a crappy circular dependency problem that will go away when 1.9 is caught up on both sides of the clj and cljs house.

yonatanel19:12:49

Ok. Some progress

colinhicks19:12:19

If this works, I think I can PR a change to local-rt that will avoid this future monkey dance

yonatanel19:12:44

It made that big error go away

yonatanel19:12:06

OK. Really smooth this time

yonatanel19:12:15

Now I can debug the nil event :P

yonatanel19:12:43

Feels like a 90's quest game

yonatanel19:12:12

I love this channel by the way

michaeldrogalis19:12:18

We seem to have corralled an adventurous bunch of devs over the years. ❤️

michaeldrogalis19:12:57

Ohhhh yeah. That’s what I’m talkin’ about. 😄