This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-11-17
Channels
- # aws (3)
- # beginners (81)
- # boot (155)
- # capetown (2)
- # cider (32)
- # clara (14)
- # cljs-dev (40)
- # cljsrn (12)
- # clojure (158)
- # clojure-austin (5)
- # clojure-korea (6)
- # clojure-poland (1)
- # clojure-russia (63)
- # clojure-spec (45)
- # clojure-uk (75)
- # clojurescript (109)
- # code-reviews (1)
- # core-logic (12)
- # core-matrix (1)
- # cursive (36)
- # datomic (16)
- # defnpodcast (1)
- # devcards (2)
- # editors (3)
- # euroclojure (1)
- # events (3)
- # flambo (1)
- # hoplon (19)
- # javascript (4)
- # jobs (1)
- # lein-figwheel (4)
- # leiningen (1)
- # off-topic (1)
- # om (177)
- # onyx (121)
- # pedestal (14)
- # planck (19)
- # proton (3)
- # re-frame (36)
- # reagent (21)
- # remote-jobs (1)
- # ring (4)
- # ring-swagger (6)
- # spacemacs (1)
- # specter (2)
- # test-check (4)
- # untangled (9)
- # utah-clojurians (1)
- # yada (2)
I wonder if I want to change number of peers in a single task, what's the best way to achieve it? Kill the job and start a new one?
@asolovyov that’s really the only way if you didn’t make it flexible in the first place
@lucasbradstreet what do you mean under 'make it flexible'?
If you set it up with :onyx/max-peers and min-peers, and the right task scheduler, if you were to add more peers it would continue to scale as you add more peers
Then you all you need to do is start more peers / more nodes
yeah I have vastly more peers than assigned right now, and I just set max-peers and they take all of them
Nope, new job is the way to go
The idea is that the job is immutable
I could see more tooling around transitionining jobs though
I just need to think of workflow how to do this without doing full-blown release right now, because on releases I just go with different instance-id so that nothing is carried on
Can someone walk me through what the standard Onyx development cycle looks like? I'm accustomed to doing Clojure development where you open a REPL and start building functions until you're done. Can I still do all of my initial Onyx development within the REPL or do I need to drop out of the REPL to start my app on Onyx peers and run it that way?
@stephenmhopper You can start locally and in-memory with https://github.com/onyx-platform/onyx-local-rt, and also learn from learn-onyx repository how to things locally but with the real runtime, before distributing: https://github.com/onyx-platform/learn-onyx.
@stephenmhopper our general recommendation is usually to use the with-test-env test method in onyx-template, and TDD it https://github.com/onyx-platform/onyx-template/blob/0.9.x/src/leiningen/new/onyx_app/test/onyx_app/jobs/basic_test.clj
however onyx-local-rt is a good way to go now
Okay, I'll take a look at onyx-local-rt
@lucasbradstreet I found the with-test-env
macro and it looks like that will work great for cases where I know in advance what I want to do with my job. However, if I want to start up my job, bind to the input and output core.async channels and read and write to those arbitrarily from the REPL, is there a way to do that without bringing in onyx-local-rt?
@stephenmhopper there are some user.clj files floating around that people have used to get a repl reloaded workflow going. I’m not sure where they ended up though 😕
So it seems like before onyx-local-rt was a thing, folks would just hack their functions together in the REPL and test the whole thing using with-test-env
? But now, onyx-local-rt is the way to go?
This is the one https://gist.github.com/jeroenvandijk/95a7d09b3c8cb7904547c95c3b4a3360
@stephenmhopper that’s basically correct
okay, cool. Thank you!
any time
I have onyx-local-rt up and running, but I'm hitting some errors. This works:
(-> (api/init job)
(api/new-segment :in {:n 41})
(api/new-segment :in {:n 84})
(api/drain))
but adding (api/stop)
to the end of that chain causes a null pointer exceptionAny ideas on what's going on there?
@stephenmhopper: could you paste the stack trace?
I can do that, but the stacktrace isn't very helpful at all. It looks like it has something to do with my job map as the simple example job map is working, but the one that I got from the onyx-template is not working. Let me figure out what's different between the two and then I'll give you something more tangible to work with.
Sure, thanks. onyx-local-rt could probably use more validation
Yeah, I plan on thoroughly kicking in the tires on all things "Onyx"
I have some upcoming projects at work where I have freedom to not only pick the framework, but to pick the programming language too. I'm hoping to go with Clojure + Onyx
I just have to convince myself that I can be successful with that combo and my boss will be cool with it 🙂
Great. Looking forward to the feedback
okay, I think I found the issue. My job map works just fine until I add in the core-async lifecycle entries for the input and output channel. Is that because of this line from the onyx-local-rt readme?: Input and output plugins are ignored, use the API to put segments into/take segments out of the runtime.
Probably. I wonder if we can validate that easily cc/ @michaeldrogalis
Do you want me to make an issue on onyx-local-rt?
Sure, thanks
Thanks
@stephenmhopper @lucasbradstreet Yes, please make any issues for things you find. I didn’t add much in the way of validation because of the transition to Clojure spec. local-rt is meant to be able to run in the browser, and we currently handle validation in Onyx core with Schema. Since we’re not upgrading to the 1.9-alpha yet, I used a backport of Clojure spec, but that’s only compatible with Clj, not Cljs.
I would obviously like to have the same level of validation for both the distributed-rt & local-rt.
okay, that's good to know. Thank you
Ill get a fix out for #2 in a few minutes.
thanks!
Fixing issues in the local-rt is a dream. No dependencies, can just copy and paste your exact snippet and it works 🙂
For what it’s worth, I’m not getting the same error as you did in the Gist. Instead of an NPE, I get:
Could not resolve symbol on the classpath, did you require the file
that contains the symbol :onyx.plugin.core-async/out-calls?
{:kw :onyx.plugin.core-async/out-calls}
I presume you actually have the core.async plugin on the classpath and required?
Let me check
Well, I wasn't requiring it apparently. I was using the onyx-template for setting up this project and it doesn't appear to require it despite referencing the core.async onyx plugin. Is that bad?
However, adding it to my dependencies doesn't change the error I'm getting
I’m just confused why we’re seeing two different errors.
oh, requiring onyx pulls in core.async
yeah, I'm confused too
:dependencies [[aero "1.0.0-beta2"]
[org.clojure/clojure "1.8.0"]
[org.clojure/tools.cli "0.3.5"]
[org.onyxplatform/onyx "0.9.12"]
[org.onyxplatform/lib-onyx "0.9.10.0"]
[org.onyxplatform/onyx-local-rt "0.9.11.1"]]
^Is something there out dated?
Can you drop your dependency on Onyx core? local-rt is tracking a particular release on Onyx that isn’t in master yet.
Sorry, that’s not obvious at all, and we do need to get that work back into master too.
Sure, one second
Should I keep lib-onyx
in there?
Well, I'm still getting the same error. My project.clj looks like this:
(defproject twitter-much "0.1.0-SNAPSHOT"
:description ""
:url ""
:license {:name ""
:url ""}
:dependencies [[aero "1.0.0-beta2"]
[org.clojure/clojure "1.8.0"]
[org.clojure/tools.cli "0.3.5"]
[org.onyxplatform/onyx-local-rt "0.9.11.1"]]
:source-paths ["src"]
:profiles {:dev {:jvm-opts ["-XX:-OmitStackTraceInFastThrow"]
:global-vars {*assert* true}
:dependencies [[org.clojure/tools.namespace "0.2.11"]
[lein-project-version "0.1.0"]]}
:uberjar {:aot [lib-onyx.media-driver
twitter-much.core]
:uberjar-name "peer.jar"
:global-vars {*assert* false}}})
Is it on GitHub somewhere that I can clone it?
It's not as it was just a project that I was going to play around with locally, but I can push it
I haven't really made any noteworthy changes to it; I generated it with the onyx-template
Okay. Only reason I ask is that I can’t figure out what’s throwing that error.
okay, I've pushed it. For now, I've just slapped the code from the aforementioned GitHub issue at the bottom of the test file located here: https://github.com/enragedginger/twitter-much-clj/blob/master/test/twitter_much/jobs/basic_test.clj
Thanks!
That test class and the project.clj are the only files I've altered since generating from the template
Shouldnt be hard to run it down now, thanks 🙂
cool, let me know what you find!
So here’s what’s happening.
Plugin resolution intentionally never takes place. Since the local-rt doesn’t support I/O plugins, we don’t go in and look at the plugin to set up connections/state/sockets or whatever. Since you used the core.async plugin, Onyx tried to use it anyway without plugin resolution having happened, and it’s trying to rederef and atom that’s normally there.
I need to think on what the best place to would be to capture and stop plugins from being executed. We don’t want to make it a rule that you can’t submit them because the spirit is that you can take your same job and run it on both runtimes.
We could not execute lifecycles that start with onyx.plugin.*
, but that’s a little too magic for my design taste.
Ill copy all that into the issue.
That makes sense to me
Do you have any ideas why you and I were seeing different error messages?
@stephenmhopper Yes, I didn’t require the core.async plugin onto my classpath
ah, okay
So, it sounds like the issue with one map to rule them all is with the lifecycle calls to plugins and not with the catalog references to plugins, no?
Because with the catalog entries, I assume onyx-local-rt knows to exclude them just based on the :onyx/type
entries of either :output
or :input
no?
We could do that.
It wouldn’t be appropriate to exclude all lifecycles for an I/O task though.
Right, I wasn't proposing that as a solution yet, I was guessing as to why things seem to work fine when I reference the plugins in the catalog, but as soon as I add the lifecycle calls, it breaks.
Ah, right.
The simplest thing would be to add a property like :onyx/rt-exclude true
for lifecycle maps that would allow folks to manually specify whether or not local-rt should even bother with that entry. Is that an option?
or rather :onyx/local-rt-exclude true
That way, you don't have to add any extra magic with plugin namespace regexes and people can tell by looking at the lifecycle map whether or not they should expect it to fire when running locally
Just a thought based on my minimal knowledge of Onyx thus far
It’s an option. I’ll think on it.
I'm trying to use the Twitter plugin, but I'm getting this error:
Caused by: clojure.lang.ExceptionInfo: Could not resolve symbol on the classpath, did you require the file that contains the symbol :onyx.plugin.twitter/twitter-reader-calls? {:kw :onyx.plugin.twitter/twitter-reader-calls, :original-exception :clojure.lang.ExceptionInfo}
I have this in my namespace :require
do I need something else?:
;; Load plugin classes on peer start
[onyx.plugin [core-async] [twitter]]
And I added this to my dependencies: [org.onyxplatform/onyx-twitter "0.9.11.1"]
oh wait a second
How are you starting the peer?
Through the test macro or the jar entrypoint?
I'm using the jar entrypoint. I think I figured it out
I had started the peers, went off and did some development (such as adding the twitter dependency and building out the job for it) and then ran a separate lein command to submit my job. But it failed because the peers were started before I added the dependency to my project.
It's working now
thank you
Nice, that can be a bit tricky.
I was going to use local-rt for the whole thing, but I can't use the Twitter plugin with that until I have some sample data to work with
@stephenmhopper Beware the tweet stream. The internet is a dark place x_x
yeah....
I was looking at the plugin and it's looking like it's just firehosing everything, no?
Thats when I learned emacs can render emojis
Yup there’s no filter
It’s a toy plugin, designed to just get some data flowing
Okay, I'll either do the filtering in my app or just make a PR to support filtering by hashtag
I think he meant PG-rated filter. 🙂
Oh haha thats for 0.10.x 😉
I'm late to the party, but regarding any kind of targeting of map entries like you might for excluding something from onyx local rt, rather than a flag, another thing to consider is a whitelist (or blacklist). I only bring this up as I've run into a lot of software where adding a specific bool flag for something became a problem later. Dealing with portability cross-platform in C/C++, this type of thing comes up in the form of pre-processor macros and such. Also, what happens if you add a 3rd "mode." It might be unlikely, but it can happen. The added advantage is you can be quite explicit if you want about targeting something when you have multiple runtimes. The downside with a whitelist I imagine is anything legacy, but in that case I guess you could assume if it's empty/not present, then fall-back to the default being the legacy (vanilla onyx for example).
Might be a good play to use reader conditionals here.
Ill keep thinking. Stripping out lifecycles earlier feels right, and a reader conditional will let you keep the entire job together.