Fork me on GitHub
#onyx
<
2016-11-17
>
asolovyov09:11:46

I wonder if I want to change number of peers in a single task, what's the best way to achieve it? Kill the job and start a new one?

lucasbradstreet09:11:37

@asolovyov that’s really the only way if you didn’t make it flexible in the first place

asolovyov09:11:11

@lucasbradstreet what do you mean under 'make it flexible'?

asolovyov09:11:29

I guess I would like to make this particular instance flexible

asolovyov09:11:41

but... will my instances re-read anything about it?

lucasbradstreet09:11:48

If you set it up with :onyx/max-peers and min-peers, and the right task scheduler, if you were to add more peers it would continue to scale as you add more peers

lucasbradstreet09:11:57

Then you all you need to do is start more peers / more nodes

asolovyov09:11:35

yeah I have vastly more peers than assigned right now, and I just set max-peers and they take all of them

asolovyov09:11:48

so there is no way to send a job update for an instance, right? 🙂

lucasbradstreet09:11:21

Nope, new job is the way to go

lucasbradstreet09:11:40

The idea is that the job is immutable

lucasbradstreet09:11:13

I could see more tooling around transitionining jobs though

asolovyov09:11:28

I just need to think of workflow how to do this without doing full-blown release right now, because on releases I just go with different instance-id so that nothing is carried on

stephenmhopper13:11:22

Can someone walk me through what the standard Onyx development cycle looks like? I'm accustomed to doing Clojure development where you open a REPL and start building functions until you're done. Can I still do all of my initial Onyx development within the REPL or do I need to drop out of the REPL to start my app on Onyx peers and run it that way?

yonatanel13:11:19

@stephenmhopper You can start locally and in-memory with https://github.com/onyx-platform/onyx-local-rt, and also learn from learn-onyx repository how to things locally but with the real runtime, before distributing: https://github.com/onyx-platform/learn-onyx.

lucasbradstreet13:11:42

@stephenmhopper our general recommendation is usually to use the with-test-env test method in onyx-template, and TDD it https://github.com/onyx-platform/onyx-template/blob/0.9.x/src/leiningen/new/onyx_app/test/onyx_app/jobs/basic_test.clj

lucasbradstreet13:11:49

however onyx-local-rt is a good way to go now

stephenmhopper13:11:13

Okay, I'll take a look at onyx-local-rt

stephenmhopper13:11:20

@lucasbradstreet I found the with-test-env macro and it looks like that will work great for cases where I know in advance what I want to do with my job. However, if I want to start up my job, bind to the input and output core.async channels and read and write to those arbitrarily from the REPL, is there a way to do that without bringing in onyx-local-rt?

lucasbradstreet13:11:14

@stephenmhopper there are some user.clj files floating around that people have used to get a repl reloaded workflow going. I’m not sure where they ended up though 😕

stephenmhopper13:11:21

So it seems like before onyx-local-rt was a thing, folks would just hack their functions together in the REPL and test the whole thing using with-test-env? But now, onyx-local-rt is the way to go?

lucasbradstreet13:11:35

@stephenmhopper that’s basically correct

stephenmhopper13:11:06

okay, cool. Thank you!

stephenmhopper14:11:31

I have onyx-local-rt up and running, but I'm hitting some errors. This works:

(-> (api/init job)
    (api/new-segment :in {:n 41})
    (api/new-segment :in {:n 84})
    (api/drain))
but adding (api/stop) to the end of that chain causes a null pointer exception

stephenmhopper14:11:40

Any ideas on what's going on there?

lucasbradstreet14:11:25

@stephenmhopper: could you paste the stack trace?

stephenmhopper15:11:16

I can do that, but the stacktrace isn't very helpful at all. It looks like it has something to do with my job map as the simple example job map is working, but the one that I got from the onyx-template is not working. Let me figure out what's different between the two and then I'll give you something more tangible to work with.

lucasbradstreet15:11:18

Sure, thanks. onyx-local-rt could probably use more validation

stephenmhopper15:11:06

Yeah, I plan on thoroughly kicking in the tires on all things "Onyx"

stephenmhopper15:11:51

I have some upcoming projects at work where I have freedom to not only pick the framework, but to pick the programming language too. I'm hoping to go with Clojure + Onyx

stephenmhopper15:11:30

I just have to convince myself that I can be successful with that combo and my boss will be cool with it 🙂

lucasbradstreet15:11:54

Great. Looking forward to the feedback

stephenmhopper15:11:49

okay, I think I found the issue. My job map works just fine until I add in the core-async lifecycle entries for the input and output channel. Is that because of this line from the onyx-local-rt readme?: Input and output plugins are ignored, use the API to put segments into/take segments out of the runtime.

lucasbradstreet15:11:16

Probably. I wonder if we can validate that easily cc/ @michaeldrogalis

stephenmhopper15:11:37

Do you want me to make an issue on onyx-local-rt?

michaeldrogalis16:11:39

@stephenmhopper @lucasbradstreet Yes, please make any issues for things you find. I didn’t add much in the way of validation because of the transition to Clojure spec. local-rt is meant to be able to run in the browser, and we currently handle validation in Onyx core with Schema. Since we’re not upgrading to the 1.9-alpha yet, I used a backport of Clojure spec, but that’s only compatible with Clj, not Cljs.

michaeldrogalis16:11:07

I would obviously like to have the same level of validation for both the distributed-rt & local-rt.

stephenmhopper16:11:56

okay, that's good to know. Thank you

michaeldrogalis16:11:09

Ill get a fix out for #2 in a few minutes.

michaeldrogalis16:11:32

Fixing issues in the local-rt is a dream. No dependencies, can just copy and paste your exact snippet and it works 🙂

michaeldrogalis16:11:38

For what it’s worth, I’m not getting the same error as you did in the Gist. Instead of an NPE, I get:

michaeldrogalis16:11:53

Could not resolve symbol on the classpath, did you require the file
that contains the symbol :onyx.plugin.core-async/out-calls?
{:kw :onyx.plugin.core-async/out-calls}

michaeldrogalis16:11:35

I presume you actually have the core.async plugin on the classpath and required?

stephenmhopper16:11:18

Well, I wasn't requiring it apparently. I was using the onyx-template for setting up this project and it doesn't appear to require it despite referencing the core.async onyx plugin. Is that bad?

stephenmhopper16:11:38

However, adding it to my dependencies doesn't change the error I'm getting

michaeldrogalis16:11:47

I’m just confused why we’re seeing two different errors.

stephenmhopper16:11:48

oh, requiring onyx pulls in core.async

stephenmhopper16:11:53

yeah, I'm confused too

stephenmhopper16:11:16

:dependencies [[aero "1.0.0-beta2"]
                 [org.clojure/clojure "1.8.0"]
                 [org.clojure/tools.cli "0.3.5"]
                 [org.onyxplatform/onyx "0.9.12"]
                 [org.onyxplatform/lib-onyx "0.9.10.0"]
                 [org.onyxplatform/onyx-local-rt "0.9.11.1"]]

stephenmhopper16:11:40

^Is something there out dated?

michaeldrogalis16:11:02

Can you drop your dependency on Onyx core? local-rt is tracking a particular release on Onyx that isn’t in master yet.

michaeldrogalis16:11:16

Sorry, that’s not obvious at all, and we do need to get that work back into master too.

stephenmhopper16:11:35

Sure, one second

stephenmhopper16:11:23

Should I keep lib-onyx in there?

stephenmhopper16:11:35

Well, I'm still getting the same error. My project.clj looks like this:

(defproject twitter-much "0.1.0-SNAPSHOT"
  :description ""
  :url ""
  :license {:name ""
            :url ""}
  :dependencies [[aero "1.0.0-beta2"]
                 [org.clojure/clojure "1.8.0"]
                 [org.clojure/tools.cli "0.3.5"]
                 [org.onyxplatform/onyx-local-rt "0.9.11.1"]]
  :source-paths ["src"]

  :profiles {:dev {:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]
                   :global-vars {*assert* true}
                   :dependencies [[org.clojure/tools.namespace "0.2.11"]
                                  [lein-project-version "0.1.0"]]}

             :uberjar {:aot [lib-onyx.media-driver
                             twitter-much.core]
                       :uberjar-name "peer.jar"
                       :global-vars {*assert* false}}})

michaeldrogalis16:11:59

Is it on GitHub somewhere that I can clone it?

stephenmhopper16:11:00

It's not as it was just a project that I was going to play around with locally, but I can push it

stephenmhopper16:11:22

I haven't really made any noteworthy changes to it; I generated it with the onyx-template

michaeldrogalis16:11:27

Okay. Only reason I ask is that I can’t figure out what’s throwing that error.

stephenmhopper16:11:09

okay, I've pushed it. For now, I've just slapped the code from the aforementioned GitHub issue at the bottom of the test file located here: https://github.com/enragedginger/twitter-much-clj/blob/master/test/twitter_much/jobs/basic_test.clj

stephenmhopper16:11:56

That test class and the project.clj are the only files I've altered since generating from the template

michaeldrogalis16:11:02

Shouldnt be hard to run it down now, thanks 🙂

stephenmhopper16:11:47

cool, let me know what you find!

michaeldrogalis16:11:00

So here’s what’s happening.

michaeldrogalis16:11:52

Plugin resolution intentionally never takes place. Since the local-rt doesn’t support I/O plugins, we don’t go in and look at the plugin to set up connections/state/sockets or whatever. Since you used the core.async plugin, Onyx tried to use it anyway without plugin resolution having happened, and it’s trying to rederef and atom that’s normally there.

michaeldrogalis16:11:46

I need to think on what the best place to would be to capture and stop plugins from being executed. We don’t want to make it a rule that you can’t submit them because the spirit is that you can take your same job and run it on both runtimes.

michaeldrogalis16:11:41

We could not execute lifecycles that start with onyx.plugin.*, but that’s a little too magic for my design taste.

michaeldrogalis16:11:19

Ill copy all that into the issue.

stephenmhopper17:11:44

That makes sense to me

stephenmhopper17:11:06

Do you have any ideas why you and I were seeing different error messages?

michaeldrogalis17:11:02

@stephenmhopper Yes, I didn’t require the core.async plugin onto my classpath

stephenmhopper17:11:54

So, it sounds like the issue with one map to rule them all is with the lifecycle calls to plugins and not with the catalog references to plugins, no?

stephenmhopper17:11:53

Because with the catalog entries, I assume onyx-local-rt knows to exclude them just based on the :onyx/type entries of either :output or :input no?

michaeldrogalis17:11:16

We could do that.

michaeldrogalis17:11:34

It wouldn’t be appropriate to exclude all lifecycles for an I/O task though.

stephenmhopper17:11:22

Right, I wasn't proposing that as a solution yet, I was guessing as to why things seem to work fine when I reference the plugins in the catalog, but as soon as I add the lifecycle calls, it breaks.

stephenmhopper17:11:40

The simplest thing would be to add a property like :onyx/rt-exclude true for lifecycle maps that would allow folks to manually specify whether or not local-rt should even bother with that entry. Is that an option?

stephenmhopper17:11:13

or rather :onyx/local-rt-exclude true

stephenmhopper17:11:00

That way, you don't have to add any extra magic with plugin namespace regexes and people can tell by looking at the lifecycle map whether or not they should expect it to fire when running locally

stephenmhopper17:11:11

Just a thought based on my minimal knowledge of Onyx thus far

michaeldrogalis17:11:14

It’s an option. I’ll think on it.

stephenmhopper18:11:39

I'm trying to use the Twitter plugin, but I'm getting this error:

Caused by: clojure.lang.ExceptionInfo: Could not resolve symbol on the classpath, did you require the file that contains the symbol :onyx.plugin.twitter/twitter-reader-calls? {:kw :onyx.plugin.twitter/twitter-reader-calls, :original-exception :clojure.lang.ExceptionInfo}
I have this in my namespace :require do I need something else?:
;; Load plugin classes on peer start
            [onyx.plugin [core-async] [twitter]]

stephenmhopper18:11:02

And I added this to my dependencies: [org.onyxplatform/onyx-twitter "0.9.11.1"]

stephenmhopper18:11:30

oh wait a second

gardnervickers18:11:35

How are you starting the peer?

gardnervickers18:11:45

Through the test macro or the jar entrypoint?

stephenmhopper18:11:06

I'm using the jar entrypoint. I think I figured it out

stephenmhopper18:11:19

I had started the peers, went off and did some development (such as adding the twitter dependency and building out the job for it) and then ran a separate lein command to submit my job. But it failed because the peers were started before I added the dependency to my project.

stephenmhopper18:11:21

It's working now

gardnervickers18:11:11

Nice, that can be a bit tricky.

stephenmhopper18:11:11

I was going to use local-rt for the whole thing, but I can't use the Twitter plugin with that until I have some sample data to work with

michaeldrogalis18:11:41

@stephenmhopper Beware the tweet stream. The internet is a dark place x_x

stephenmhopper18:11:08

I was looking at the plugin and it's looking like it's just firehosing everything, no?

gardnervickers18:11:22

Thats when I learned emacs can render emojis

gardnervickers18:11:39

Yup there’s no filter

gardnervickers18:11:54

It’s a toy plugin, designed to just get some data flowing

stephenmhopper18:11:21

Okay, I'll either do the filtering in my app or just make a PR to support filtering by hashtag

michaeldrogalis19:11:21

I think he meant PG-rated filter. 🙂

gardnervickers19:11:46

Oh haha thats for 0.10.x 😉

hugesandwich19:11:59

I'm late to the party, but regarding any kind of targeting of map entries like you might for excluding something from onyx local rt, rather than a flag, another thing to consider is a whitelist (or blacklist). I only bring this up as I've run into a lot of software where adding a specific bool flag for something became a problem later. Dealing with portability cross-platform in C/C++, this type of thing comes up in the form of pre-processor macros and such. Also, what happens if you add a 3rd "mode." It might be unlikely, but it can happen. The added advantage is you can be quite explicit if you want about targeting something when you have multiple runtimes. The downside with a whitelist I imagine is anything legacy, but in that case I guess you could assume if it's empty/not present, then fall-back to the default being the legacy (vanilla onyx for example).

michaeldrogalis22:11:31

Might be a good play to use reader conditionals here.

michaeldrogalis22:11:17

Ill keep thinking. Stripping out lifecycles earlier feels right, and a reader conditional will let you keep the entire job together.