I wish the user guide on the docs, is structured in logical way than just alphabetic order

I can’t help to think that this fake-s3 thing would be so much easier to write in Clojure (LOL), but looks useful!

(I was a Ruby fanboy once, for the record)

maybe it's possible to do on top of pithos (it might be "just" implementing a couple of protocols to do stuff in memory/disk)

@vijaykiran: sorry about that. We've been meaning to rearrange the docs in Jekyll

@mpenet: Sounds like a possibility, although the number of dependencies would scare me https://github.com/exoscale/pithos/blob/master/project.clj#L11

yeah, it's not exaclty lightweight

but should be solid given what they use it for

true

If anything, you could probably copy the test suite and build something light weight and know it is of a similar quality

(Ok back to work :))

@vijaykiran: Working on it, I dislike our current User Guide too. I've been working on compiling it into a one-page document with side-bar anchor links.

The more we grow, the more things that were once simple, like publishing docs, get more complex. 😛

How useful is Onyx on a single peer? It seems like a better way to handle running multiple "handlers" for core.async channels. Or is there something else more suited to this?

Onyx is a great way to model your computation through data!

@dominicm: some people use it for this, yes. It's a trade-off, since Onyx brings the full power and complexity of a tool that orchestrates a cluster. The flip side is that you get a very good computational model.

I guess it makes it easier to horizontally scale later too?

We are still running on only one peer. We chose onyx exactly because of these reasons

Correct. Also, if your input data source(s) are remote, you're already dealing with a distributed system, and so you can get fault handling for free.

You mean input to the channels?

This is very reassuring. I had dismissed Onyx as something for "later when scale comes" but I want an event based model, so I'll create serious bottlenecks if there's only one processor for it

Have a play around with learn-onyx and see if it fits your model. Might be a good fit.

Will do, thanks!

Np!

learn-onyx is awesome

Does a lifecycle before-task-start run before every segment that runs through that particular task? or does it run once as the job spins up?

No it only runs once

it will run any time a peer starts operating on a task

so this could happen any time during a job e.g. if a new node starts up.

right, i wasn’t sure if it meant before a segment ran through it. I didn’t want my db connections to get restarted everytime a segment came through. lol

I figured, I just wanted to make sure you got the whole picture 🙂

yep, its how i thought it worked but began questioning myself. lol

In the case of a trigger what would I use for the lifecycle/task ?

any hints on this?

It depends what you want to do

Triggers attach to tasks that have windows

@gardnervickers: I am setting a lifecycle before-task to setup some DB connections used by the trigger

so for the lifecycles/task would i use the sync function as well?

No, your lifecycle runs before-task and returns a map that looks like {:db-conn <db-conn>}. That map is merged into the event map. For any triggers that are attached to the same task your lifecycle is attached to, they will receive the event map as an arg, which you can grab the :db-conn out of.

so I am still confused what task would I attach the lifecycle to so the event ( with db conns ) is available in the trigger?

It would be whatever task you’re windowing on.

If you don’t need to do any processing, you can just make a dummy task

ok, i think i see

For the state you created in your lifecycle to be available to your trigger, both the lifecycle and trigger need to be attached to the same task.

Your trigger sync function will be supplied the event map

as an argument

@drewverlee: I meant to chime in earlier and say thanks for the PR on the dashboard. Were you able to connect to it?

i haven’t tried it yet, i’m currently struggling to understand why 3 separate datadog processes are running on this server. like who let the dogs out ahhaha 😭

haha