Fork me on GitHub
#onyx
<
2016-07-06
>
jstokes01:07:48

just wanted to say thanks to everyone who worked on the learn-onyx workshop repo, it's a great way to get acquainted with onyx

gardnervickers02:07:51

Glad your enjoying it, Mike worked really hard on it and there have been a boatload of community contributions. Happy to hear it helped!

v.solovyov07:07:58

I'll have to communicate with external API that has a rate limit (and ridiculously low, 1 call per minute per IP). I'm skimming through the Onyx docs and I can't find anything relevant

lucasbradstreet08:07:09

@v.solovyov: best approach is to batch your requests. So if you have a 1 call per minute limit, then your batch-timeout should be at least one minute (and the number of peers should be fixed accordingly)

lucasbradstreet08:07:20

We’ve dealt with something similar before, which is why we added https://github.com/onyx-platform/onyx-http#batch-http-output

lucasbradstreet08:07:46

The tough part will come if you want to spread your requests over all your IPs so you can do more than one request a minute

lucasbradstreet08:07:57

I don’t have any great suggestions for you there

v.solovyov08:07:20

I can't actually batch them. I mean, I can only do one call per minute, period (and I need to do up to 1000 per day). So maybe I could accumulate them in a state and do a trigger, that will do one call per minute, hmm

v.solovyov08:07:55

As per spreading my requests over all my IPs - I remember I suggested a solution to someone with a similar problem a few months ago on gitter

v.solovyov08:07:35

I probably will need to write a refinement mode for such a trigger

lucasbradstreet08:07:57

So one segment = one API call?

lucasbradstreet08:07:30

Ah that’s trickier.

v.solovyov08:07:33

If I consistently do more calls than one per minute, they will ban me

lucasbradstreet08:07:08

This does seem like a good use of state management

v.solovyov08:07:09

Ok, then to spread my requests over my IPs I'll have to have :onyx/max-peers on the task equal to number of my peers, and somehow schedule just one task of such type per peer.

lucasbradstreet08:07:04

I think that will be very difficult to achieve without us giving you some way to describe your scheduler constraints (we want this in the future but it might be a while off)

v.solovyov08:07:18

It's like anti-`:onyx.peer/tags` 🙂

lucasbradstreet08:07:32

One way to do it is to expand out to n tasks, each with max-peers 1, and their own tag on each host

v.solovyov08:07:58

yeah, that's quite reasonable solution for now

lucasbradstreet08:07:01

For that to be fault tolerant you would probably only be able to do one peer per every two nodes, since you will need a fallback node for each

v.solovyov08:07:30

I see, haven't thought about that

v.solovyov08:07:57

That would be much better than having to do it one at a time anyway

v.solovyov08:07:03

Thank you very much! 🙂

lucasbradstreet08:07:10

Yeah, it's only necessary because you're restricting which peers they can schedule to.

zamaterian10:07:17

Are you guys going to euroclojure ?

lucasbradstreet10:07:42

Probably not. We’ll all be in Seattle by then, and it’ll be quite far.

lucasbradstreet10:07:28

Will consider it though 😉

zamaterian10:07:51

Then I will drink all the 🍻

agile_geek12:07:06

Guys, silly question but in the examples in the learn-onyx workshop the logging messages are sent to an agent. Why? Is this something to do with accessing System.out outside the local thread of the onyx peer?

michaeldrogalis14:07:03

@agile_geek: System.out.println isn't thread safe. Sending all print fn's through an agent will strictly serialize them, so stdout won't get clobbered.

michaeldrogalis14:07:22

We're eventually going to move away from checking out happened in stdout, that's obviously pretty hacky.

lucasbradstreet14:07:57

We really need to switch to operating on atoms now

lucasbradstreet14:07:07

The error output is going to be captured

agile_geek14:07:53

@michaeldrogalis: thanks. I thought it was something like that.

lucasbradstreet14:07:01

@michaeldrogalis: can we change the error output to go to stderr?

lucasbradstreet14:07:12

That could at least avoid that issue for now

lucasbradstreet14:07:29

(the capturing part)

michaeldrogalis14:07:35

@lucasbradstreet: Sure, yeah. Not opposed to changing how that all works

lucasbradstreet14:07:50

I mean in onyx core

lucasbradstreet14:07:08

i.e. the helpful error message printing

michaeldrogalis14:07:10

Right, yeah. I'd also prefer to fix the ickiness in learn-onyx too while we're at it.

lucasbradstreet14:07:22

But yes, we need to fix that too

jstokes15:07:04

wrt capturing standard out in those learn-onyx exercises - sometimes it makes debugging what went wrong a little difficult, since the helpful error messages are captured in with-out-str and if job submission fails aren’t reported with the test failure

jstokes15:07:03

issue 614 looks like it would address though :thumbsup:

michaeldrogalis15:07:14

Yep, it's definitely a huge pain after we introduced pretty error messages.

michaeldrogalis15:07:30

Here's the 0.9.7 blog post. Thanks again to everyone who contributed to the release! http://www.onyxplatform.org/jekyll/update/2016/07/06/Onyx-0.9.7-Refactoring.html