This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2015-12-09
Channels
- # beginners (140)
- # boot (163)
- # cider (43)
- # cljs-dev (50)
- # cljsjs (5)
- # cljsrn (38)
- # clojure (152)
- # clojure-austria (10)
- # clojure-berlin (9)
- # clojure-dev (11)
- # clojure-japan (1)
- # clojure-miami (2)
- # clojure-russia (147)
- # clojure-sg (12)
- # clojurescript (244)
- # code-reviews (3)
- # cursive (104)
- # data-science (5)
- # datavis (15)
- # datomic (35)
- # editors (4)
- # hoplon (1)
- # ldnclj (11)
- # lein-figwheel (14)
- # leiningen (22)
- # off-topic (53)
- # om (373)
- # omnext (2)
- # onyx (67)
- # parinfer (193)
- # re-frame (23)
- # reagent (89)
- # yada (7)
just another quick update lucasbradstreet and michaeldrogalis, Highstorm is -awesome- now. it’s so good, i’m wondering just how on earth i could mess it up so badly before!
:thumbsup: at least there was light at the end of the tunnel :)
@robert-stuttaford: what is highstorm?
@tcrayford: Cognician's Onyx based back end
sorry, i’m a bit of a Brandon Sanderson nerd https://en.wikipedia.org/wiki/The_Stormlight_Archive#World
> The world of The Way of Kings is one periodically assaulted by highstorms, storms characterized by a very violent storm front traveling from East to West (beginning at the Origin), followed by weaker rains. Flora and fauna have evolved to cope with this condition.
@tcrayford: it watches our Datomic tx log and processes stats, updates short-cut cache collections (for faster web side queries), and sends emails, and then writes more stuff back to Datomic
basically a distributed, non-consistent, potentially side-effecting transactor function system
hey lucasbradstreet, this external aeron thing. which do you recommend - building and running the media driver from aeron or making a clojure app as your docs display?
Either should work fine. We include one in our template, and I think that gives you a little more control but it really works either way
Scratch that, I think using one in your own code is best
because it ensures you’re always using the same version of aeron
as your Onyx code
i want to get it right with a minimum of fuss. sounds like we’d have to maintain a separate clojure app with its own build and deploy?
Naw, just stick something like this in your highstorm code base
then you can lein with-profile production run -m highstorm(after you’ve renamed the ns).aeron-media-driver
ah, so start the same jar twice
in two ways
any sense of how much ram it’d need?
I think it mostly uses java’s Unsafe, so I’m not sure it’ll matter what you allocate it via -Xmx
i.e. I think it allocates off heap
ah. but we’d still need to set some aside for it
Yeah, it’ll use some RAM, certainly
The answer is I’m not really sure and it it depends on how much you’re pushing through it
It’s a pretty critical piece of the puzzle though. It doesn’t need a lot of resources (CPU/RAM) but if it gets starved bad things can happen
I think you may want to run it in shared mode
ok. being distracted by other things now, so… to-be-continued
no worries
and i’m back. so we need to run an aeron for each onyx instance, right?
where should we start with ram - 500mb? 1gb?
maybe 1GB max, no min heap?
but don’t assume that’s all that it’ll use
also, i presume that aeron should be started before onyx and stopped after onyx is stopped
may i ask why a separate jvm is recommended?
Aeron works best when it has dedicated resources and doesn’t get starved
If you run it in the same JVM then it’s susceptible to GC pressure from the rest of your app
Another thing to consider, if you’re sending big messages around you need to play with MTU and other settings
I’ll create a doc from this discussion.
by message you mean segments?
Yes, but think in terms of batches of segments
You should probably use shared mode for the driver, see the arg in here: https://raw.githubusercontent.com/onyx-platform/onyx-benchmark/master/src/onyx_benchmark/aeron_media_driver.clj
shared, got it
ok. we’ll figure out the ops side of things and get a two-server test rig going
this is a good thing to test with (taken from the “going to production” section)
Test on a single node with without short circuiting: when :onyx.messaging/allow-short-circuit? is true, Aeron messaging will be short circuited completely. To test messaging on a local mode as if it's in production, set :onyx.messaging/allow-short-circuit? false.
only use that for testing locally because you want short circuiting in general
yes, that was going to be my first step - to compare the difference in perf with that off and on
our staging server is fully telemetered (new word?) so we’ll quickly see what kind of difference it’ll make
Cool, yep, that’s a good idea
cool. news as i have it thanks man!
you’re in Singapore right?
In practice your peers will perform somewhat worse and somewhat better in practice, than single node with that setting. Worse because they actually have to communicate over the network rather than via loopback. Better because sometimes they get to use short circuiting
For the last 5 and a half years
awesome nearly bed time there then? or are you a night owl?
10:15, but yeah, I’m quite a night owl, and @michaeldrogalis is a morning person, so the time difference isn’t so bad 😄
I’m sick at the moment so I’m going to get an early sleep, heh
vitamins and rest!
don’t let me keep you. feel better man!
Yup, I’m firing off Jepsen tests every now and again, but they don’t take much effort. Mostly just typing a few letters and hitting enter 😄
Thanks mate
@robert-stuttaford: Glad to hear it's going well.