This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-07-06
Channels
- # admin-announcements (2)
- # boot (51)
- # bristol-clojurians (1)
- # capetown (14)
- # cider (4)
- # cljs-dev (3)
- # cljsrn (23)
- # clojure (76)
- # clojure-gamedev (2)
- # clojure-germany (2)
- # clojure-greece (2)
- # clojure-hk (5)
- # clojure-poland (1)
- # clojure-quebec (3)
- # clojure-russia (19)
- # clojure-spec (7)
- # clojure-sweden (10)
- # clojure-uk (77)
- # clojurescript (42)
- # clojurex (5)
- # core-async (40)
- # cursive (12)
- # datomic (58)
- # editors (1)
- # events (1)
- # heroku (1)
- # hoplon (4)
- # jobs (6)
- # jobs-discuss (1)
- # ldnclj (2)
- # lein-figwheel (1)
- # leiningen (5)
- # om (66)
- # onyx (51)
- # other-languages (80)
- # proton (20)
- # protorepl (12)
- # quil (3)
- # re-frame (3)
- # reagent (18)
- # spacemacs (14)
- # untangled (78)
- # yada (16)
just wanted to say thanks to everyone who worked on the learn-onyx workshop repo, it's a great way to get acquainted with onyx
Glad your enjoying it, Mike worked really hard on it and there have been a boatload of community contributions. Happy to hear it helped!
@jstokes: Thanks 🙂
I'll have to communicate with external API that has a rate limit (and ridiculously low, 1 call per minute per IP). I'm skimming through the Onyx docs and I can't find anything relevant
@v.solovyov: best approach is to batch your requests. So if you have a 1 call per minute limit, then your batch-timeout should be at least one minute (and the number of peers should be fixed accordingly)
We’ve dealt with something similar before, which is why we added https://github.com/onyx-platform/onyx-http#batch-http-output
The tough part will come if you want to spread your requests over all your IPs so you can do more than one request a minute
I don’t have any great suggestions for you there
I can't actually batch them. I mean, I can only do one call per minute, period (and I need to do up to 1000 per day). So maybe I could accumulate them in a state and do a trigger, that will do one call per minute, hmm
As per spreading my requests over all my IPs - I remember I suggested a solution to someone with a similar problem a few months ago on gitter
I probably will need to write a refinement mode for such a trigger
So one segment = one API call?
Ah that’s trickier.
If I consistently do more calls than one per minute, they will ban me
This does seem like a good use of state management
Ok, then to spread my requests over my IPs I'll have to have :onyx/max-peers
on the task equal to number of my peers, and somehow schedule just one task of such type per peer.
I think that will be very difficult to achieve without us giving you some way to describe your scheduler constraints (we want this in the future but it might be a while off)
It's like anti-`:onyx.peer/tags` 🙂
One way to do it is to expand out to n tasks, each with max-peers 1, and their own tag on each host
yeah, that's quite reasonable solution for now
For that to be fault tolerant you would probably only be able to do one peer per every two nodes, since you will need a fallback node for each
I see, haven't thought about that
That would be much better than having to do it one at a time anyway
Thank you very much! 🙂
Yeah, it's only necessary because you're restricting which peers they can schedule to.
You're welcome
Are you guys going to euroclojure ?
Probably not. We’ll all be in Seattle by then, and it’ll be quite far.
Will consider it though 😉
Then I will drink all the 🍻
Guys, silly question but in the examples in the learn-onyx workshop the logging messages are sent to an agent. Why? Is this something to do with accessing System.out outside the local thread of the onyx peer?
@agile_geek: System.out.println
isn't thread safe. Sending all print fn's through an agent will strictly serialize them, so stdout won't get clobbered.
We're eventually going to move away from checking out happened in stdout, that's obviously pretty hacky.
We really need to switch to operating on atoms now
The error output is going to be captured
@michaeldrogalis: thanks. I thought it was something like that.
@michaeldrogalis: can we change the error output to go to stderr?
That could at least avoid that issue for now
(the capturing part)
@lucasbradstreet: Sure, yeah. Not opposed to changing how that all works
I mean in onyx core
i.e. the helpful error message printing
Right, yeah. I'd also prefer to fix the ickiness in learn-onyx too while we're at it.
But yes, we need to fix that too
wrt capturing standard out in those learn-onyx exercises - sometimes it makes debugging what went wrong a little difficult, since the helpful error messages are captured in with-out-str
and if job submission fails aren’t reported with the test failure
Yep, it's definitely a huge pain after we introduced pretty error messages.
Here's the 0.9.7 blog post. Thanks again to everyone who contributed to the release! http://www.onyxplatform.org/jekyll/update/2016/07/06/Onyx-0.9.7-Refactoring.html