Fork me on GitHub
#onyx
<
2016-03-02
>
a.espolov02:03:03

I do not understand the startup section of the dashboard. What's behind the core.clj file that you want to run? and in which project add (def env (onyx.api/start-env env-config)) in onyx-starter or onyx-dashboard?

gardnervickers02:03:47

You’re supposed to go sexp by sexp through core.clj in onyx-starter evaluating forms until you get here (def env (onyx.api/start-env env-config)) That will start the “enviornment”, which the dashboard with connect to

a.espolov02:03:48

i’m not found core.clj file in onyx-starter project

gardnervickers02:03:37

Oops you’re right, looks like thats out of date

gardnervickers02:03:53

We’ve been changing how we launch test environments quite a bit lately

gardnervickers02:03:42

aka @(promise) or something. That will keep the environment up and running for you you to connect the dashboard to

a.espolov02:03:00

@gardnervickers: i must declare a var in the test?

gardnervickers02:03:22

That’s just to halt the test so it doesent teardown the environment

gardnervickers02:03:58

What happens is during the test, Onyx has some convenience functions that stand up the infrastructure needed for an Onyx cluster in-memory. At the end of the test, in the finally block, it’s shut down. I just suggested blocking with @(promise) if you wanted to stop the test from tearing down the in-memory infrastructure so that you could connect to it with the dashboard.

gardnervickers02:03:03

@a.espolov: Using the dashboard is going to be a little tricky for that particular example

gardnervickers02:03:29

We should either fix that or remove the suggestion to use it.

a.espolov03:03:06

As far as i understood on a single physical machine will work one peer?

gardnervickers03:03:41

You can run multiple “peers” inside a single JVM

gardnervickers03:03:49

each peer is just a couple threads

gardnervickers03:03:01

You will need 1 peer per task you would like to run

gardnervickers03:03:12

Each part of your workflow is a task

gardnervickers03:03:19

So a workflow that looks like this

[[:in :filter]
 [:filter :out]]

gardnervickers03:03:43

would need at least 3 peers

gardnervickers03:03:40

If you wanted to run 5 instances of :filter, say, in addition to 1 instance of an input and 1 instance of an output, you would need at least 7 “peers” running either locally or across your cluster.

a.espolov03:03:57

It is hard for me to evaluate whether onyx to my task. Suppose that there are several types of tasks: a. perform get request + disassemble + add reply to database b. run phantomjs run query + disassemble + add reply to database c. send email

a.espolov03:03:04

and it is not clear how to declare a data schema and work with data. https://github.com/onyx-platform/beginners-guide/blob/master/chapters/chapter-3.md here says that there is an automatic database schema generation

michaeldrogalis03:03:36

@a.espolov: Those headers are suggestions with things that you can do with Onyx data, and more generally with Clojure data.

michaeldrogalis03:03:05

Onyx doesn't ship something like that out of the box. It's a design suggestion for user applications.

lucasbradstreet14:03:50

@michaeldrogalis: onyx ClojureSG talk has been given. Time to write my Jepsen post!

robert-stuttaford18:03:41

looking forward to seeing your ClojureRemote talk, @michaeldrogalis simple_smile

robert-stuttaford18:03:51

happy news, our AWS testing cluster is up and running!

robert-stuttaford18:03:05

my goodness this stuff takes a lot of work simple_smile

michaeldrogalis18:03:53

@robert-stuttaford: Hah. All the dev-ops stuff for standing the machines up?

michaeldrogalis18:03:22

What did you end up using to orchestrate deployment?

gardnervickers18:03:23

I can get onboard with that sentiment

robert-stuttaford18:03:16

our usual tools, CircleCI + AWS CodeDeploy, and then upstart + some bash (there’s always some bash) to actually start jars

robert-stuttaford18:03:38

we have a nifty script which ensures that only one instance in an ASG actually submits jobs

michaeldrogalis18:03:29

That sounds neat. I'm glad you're up and running, that deserves a big congrats. simple_smile

michaeldrogalis18:03:35

Marching steadily to the 0.9.0 release. Stateful operations will get expressiveness. Streaming engine upgrade on its way too, will be right on par with the fastest known design.

robert-stuttaford19:03:02

hopefully we get the current 0.8.x live before you release 0.9.x 😝

michaeldrogalis19:03:41

I'm sure you will now that you have good experience on the operations piece.

michaeldrogalis19:03:46

Gonna scram for lunch. Have a good one!