Fork me on GitHub
#onyx
<
2015-12-09
>
robert-stuttaford08:12:40

just another quick update lucasbradstreet and michaeldrogalis, Highstorm is -awesome- now. it’s so good, i’m wondering just how on earth i could mess it up so badly before!

lucasbradstreet08:12:36

:thumbsup: at least there was light at the end of the tunnel :)

lucasbradstreet10:12:12

@tcrayford: Cognician's Onyx based back end

robert-stuttaford11:12:59

> The world of The Way of Kings is one periodically assaulted by highstorms, storms characterized by a very violent storm front traveling from East to West (beginning at the Origin), followed by weaker rains. Flora and fauna have evolved to cope with this condition.

robert-stuttaford11:12:12

@tcrayford: it watches our Datomic tx log and processes stats, updates short-cut cache collections (for faster web side queries), and sends emails, and then writes more stuff back to Datomic

robert-stuttaford11:12:47

basically a distributed, non-consistent, potentially side-effecting transactor function system simple_smile

robert-stuttaford13:12:03

hey lucasbradstreet, this external aeron thing. which do you recommend - building and running the media driver from aeron or making a clojure app as your docs display?

lucasbradstreet13:12:59

Either should work fine. We include one in our template, and I think that gives you a little more control but it really works either way

lucasbradstreet13:12:32

Scratch that, I think using one in your own code is best

lucasbradstreet13:12:39

because it ensures you’re always using the same version of aeron

lucasbradstreet13:12:45

as your Onyx code

robert-stuttaford13:12:07

i want to get it right with a minimum of fuss. sounds like we’d have to maintain a separate clojure app with its own build and deploy?

lucasbradstreet13:12:34

Naw, just stick something like this in your highstorm code base

lucasbradstreet13:12:05

then you can lein with-profile production run -m highstorm(after you’ve renamed the ns).aeron-media-driver

robert-stuttaford13:12:22

ah, so start the same jar twice

robert-stuttaford13:12:33

any sense of how much ram it’d need?

lucasbradstreet13:12:29

I think it mostly uses java’s Unsafe, so I’m not sure it’ll matter what you allocate it via -Xmx

lucasbradstreet13:12:44

i.e. I think it allocates off heap

robert-stuttaford13:12:21

ah. but we’d still need to set some aside for it

lucasbradstreet13:12:48

Yeah, it’ll use some RAM, certainly

lucasbradstreet13:12:18

The answer is I’m not really sure and it it depends on how much you’re pushing through it

lucasbradstreet13:12:00

It’s a pretty critical piece of the puzzle though. It doesn’t need a lot of resources (CPU/RAM) but if it gets starved bad things can happen

lucasbradstreet13:12:36

I think you may want to run it in shared mode

robert-stuttaford13:12:37

ok. being distracted by other things now, so… to-be-continued simple_smile

robert-stuttaford13:12:43

and i’m back. so we need to run an aeron for each onyx instance, right?

robert-stuttaford13:12:12

where should we start with ram - 500mb? 1gb?

lucasbradstreet14:12:25

maybe 1GB max, no min heap?

lucasbradstreet14:12:35

but don’t assume that’s all that it’ll use

robert-stuttaford14:12:24

also, i presume that aeron should be started before onyx and stopped after onyx is stopped

robert-stuttaford14:12:33

may i ask why a separate jvm is recommended?

lucasbradstreet14:12:31

Aeron works best when it has dedicated resources and doesn’t get starved

lucasbradstreet14:12:47

If you run it in the same JVM then it’s susceptible to GC pressure from the rest of your app

lucasbradstreet14:12:00

Another thing to consider, if you’re sending big messages around you need to play with MTU and other settings

lucasbradstreet14:12:34

I’ll create a doc from this discussion.

robert-stuttaford14:12:39

by message you mean segments?

lucasbradstreet14:12:51

Yes, but think in terms of batches of segments

robert-stuttaford14:12:42

ok. we’ll figure out the ops side of things and get a two-server test rig going

lucasbradstreet14:12:54

this is a good thing to test with (taken from the “going to production” section)

lucasbradstreet14:12:57

Test on a single node with without short circuiting: when :onyx.messaging/allow-short-circuit? is true, Aeron messaging will be short circuited completely. To test messaging on a local mode as if it's in production, set :onyx.messaging/allow-short-circuit? false.

lucasbradstreet14:12:20

only use that for testing locally because you want short circuiting in general

robert-stuttaford14:12:47

yes, that was going to be my first step - to compare the difference in perf with that off and on

robert-stuttaford14:12:23

our staging server is fully telemetered (new word?) so we’ll quickly see what kind of difference it’ll make

lucasbradstreet14:12:00

Cool, yep, that’s a good idea

robert-stuttaford14:12:47

cool. news as i have it simple_smile thanks man!

robert-stuttaford14:12:01

you’re in Singapore right?

lucasbradstreet14:12:52

In practice your peers will perform somewhat worse and somewhat better in practice, than single node with that setting. Worse because they actually have to communicate over the network rather than via loopback. Better because sometimes they get to use short circuiting

lucasbradstreet14:12:17

For the last 5 and a half years

robert-stuttaford14:12:36

awesome simple_smile nearly bed time there then? or are you a night owl?

lucasbradstreet14:12:07

10:15, but yeah, I’m quite a night owl, and @michaeldrogalis is a morning person, so the time difference isn’t so bad 😄

lucasbradstreet14:12:28

I’m sick at the moment so I’m going to get an early sleep, heh

robert-stuttaford14:12:56

vitamins and rest!

robert-stuttaford14:12:03

don’t let me keep you. feel better man!

lucasbradstreet14:12:20

Yup, I’m firing off Jepsen tests every now and again, but they don’t take much effort. Mostly just typing a few letters and hitting enter 😄

michaeldrogalis15:12:20

@robert-stuttaford: Glad to hear it's going well. simple_smile