Fork me on GitHub
#onyx
<
2015-10-30
>
clojuregeek20:10:54

has anyone off loaded any of the processing of things in onyx to aws lambda? yet? or is that a crazy idea

michaeldrogalis21:10:07

@clojuregeek: They're more competing than complementary.

clojuregeek21:10:42

see i am still looking for excuse to use onyx 😉

michaeldrogalis21:10:54

You're a champ 😉

clojuregeek21:10:52

so the problem now i have withs 4 aws lambda functions, is some way to make sure things follow in order, how to see where i am in the process...

clojuregeek21:10:12

i thought, maybe I need a statemachine?

michaeldrogalis21:10:21

@clojuregeek: So, like, each piece of data, you want to track it as it moves through your processing pipeline - because presumably each stage takes a long time?

michaeldrogalis21:10:57

I haven't used Lambda, but I don't think it's a good fit for that - neither is Onyx. I mean, are we talking seconds or minutes+ for completion?

clojuregeek21:10:14

maybe useing kafka (kensises on aws) ... 4 queues

michaeldrogalis21:10:42

Oh - seconds is fine for both of these, then.

michaeldrogalis21:10:15

Since you're using AWS, maybe use Dynamo to track state as things move across the pipeline, keyed by some ID?

clojuregeek21:10:36

yes, i think surely i need a database ... the data can come from cloudtrail or maybe i have a collection of coludtrail yaml files that I want to use (as proof of concept for example)

michaeldrogalis21:10:49

What're the stages of your processing doing?

clojuregeek21:10:24

I'm reading in a cloudtrail log ... and if the log is someone changing tags on an instance .... i want to convert that change to TOSCA ... Then take that instance id and get "what are the tags right now" and make a new tosca document with those tags ... then run through a validation process to make sure it has the tags required for the business

clojuregeek21:10:28

TOSCA is a format to describe cloud orchestration http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/v1.0/csd02/TOSCA-Simple-Profile-YAML-v1.0-csd02.html .... that my boss thinks is the future and wants to move towards

michaeldrogalis21:10:47

Hah. Yeah, Onyx would be excellent for doing that.

clojuregeek21:10:24

i didn't explain all that before?

michaeldrogalis21:10:47

I hear a lot of use cases every day. Sometimes I forget 😉

clojuregeek21:10:25

i may have been vague at some points

clojuregeek21:10:32

but now i'm not so much

michaeldrogalis21:10:47

Yeah, it sounds like you have it down at this point.

clojuregeek21:10:51

part of what converts cloudtrail to tosca we are going to open source soon

michaeldrogalis21:10:25

Cool. simple_smile So yes, I'd pick either Lambda or Onyx here - mixing them will hurt.

clojuregeek21:10:38

so with oynx i'd have the pipeline that will do this ... with Lambda, i have to write something that will be the pipeline

michaeldrogalis21:10:09

I'm not really sure what Lambda offers you tbh.

clojuregeek21:10:42

its making it easy to deploy without figuring out how to do it 😉

clojuregeek21:10:48

just upload a jar

michaeldrogalis21:10:37

I see. You'll have to do a little more foot-work with Onyx. Same idea, though.

clojuregeek21:10:58

we've been trying to put output of this thing in sqs ... but then lambda can't pull from sqs.. only sns ...

clojuregeek21:10:06

its just alot of plumbing

michaeldrogalis21:10:26

Yeah, there's always a lotta that. 😧