Fork me on GitHub
#onyx
<
2015-12-30
>
raymcdermott10:12:39

It seems like the JVM is a pretty heavy dependency. I understand that but we would like to support composition in a stackless way. Maybe this is just too much of a dream at the moment and seems like a non-goal for Onyx (which is totally reasonable!)

lucasbradstreet10:12:49

Yeah, it'd be a "nice to have" for us, but is a non-goal/not a priority

michaeldrogalis14:12:35

@raymcdermott: For me right now, that feels like implementing Clojure for real on the CLR after having implemented it on the JVM. Tiring. I have an idea if you're still willing to entertain the JVM as a thing that needs to be there, but not interacted with directly.

raymcdermott15:12:01

@michaeldrogalis: go for it

michaeldrogalis15:12:57

We could set up a small wrapper in JavaScript that talks to Onyx's JVM peer via a local socket. The local socket would shuttle segments back and forth between the JVM and the JS runtime. Also, to facilitate the full power of Onyx, we'd maintain local state inside the JS runtime to mirror the event map. The socket would also send notifications of when to invoke lifecycle hooks so it could transition its local state accordingly.

michaeldrogalis15:12:30

It's a half-baked idea I had falling asleep last night, so I'm not positive that can work as stated - but I don't think it's too far off.

raymcdermott15:12:24

I think it’s an interesting possibility … we were thinking of Docker as the unit of functionality and Kafka as a pipe. The problem we are having is that we cannot expect all clients to all be able to consume data from Kafka

raymcdermott15:12:50

our simple thought was to put data on to named pipes and have consumers read from them

raymcdermott15:12:06

but we lose all ability to ack success and failure

raymcdermott15:12:28

unless the client has another named pipe for this purpose

raymcdermott15:12:47

nice thing is that the solution is stackless

raymcdermott15:12:32

but its very ad-hoc compared to Onyx - which is more closely competing with Spark if I understand properly

michaeldrogalis15:12:45

Right, yeah. I'd love to work on that, but something that big would be a commercial project for me.

raymcdermott15:12:50

so I’m probably way off topic but defining flow as a set of functions could fit over a more adhoc clow

michaeldrogalis15:12:51

Seems really neat.

michaeldrogalis15:12:55

I see it a little differently. When you're using functions to define flow, you're locked into a specific language. Pushing that stuff out into data is what leads you into what you call stackless territory.

raymcdermott15:12:08

ok - thanks for entertaining the crazy ideas and I think the JS wrapper with a socket could be an interesting option anyway!

raymcdermott15:12:02

to your point though, the functions must ultimately be called

raymcdermott15:12:19

which is why you are not stackless, no?

michaeldrogalis15:12:42

Can you define what exactly you mean by stackless?

raymcdermott15:12:10

I mean the functions can be implemented in any language / stack

raymcdermott15:12:42

it’s the dream of REST but REST is not good enough

michaeldrogalis15:12:17

Ah, okay. I grew up hearing the term "polyglot stack" for that. Alrighty, I'm on the same page. That's the ultimate goal of Onyx.

raymcdermott15:12:20

we want to think the Unix way …. but we don’t have a generalised pipe

michaeldrogalis15:12:24

One of them, anyway.

raymcdermott15:12:45

which is what we though Kafka could represent

michaeldrogalis15:12:59

Ideally we're looking at being able to say inside each catalog entry :onyx/language :ruby, and the scheduler can do appropriate placement.

raymcdermott15:12:44

same page in that case …

michaeldrogalis15:12:18

I'm less enthusiastic about isolating it to the extent that you are. That would require implementing all the threading, error handling, retry logic etc in every language. Definitely not feasible, for even a large engineering team.

michaeldrogalis15:12:28

The cost of adding another language would be astronomical.

raymcdermott15:12:40

that’s why we thought named pipes or something equally language neutral

raymcdermott15:12:34

and then you just send data up the pipe and listen on an acking pipe

michaeldrogalis15:12:42

Backing up once more - named pipes?

raymcdermott15:12:07

you can keep your Onyx stuff on ‘this’ side of the pipe

raymcdermott15:12:27

named pipes are OS constructs on *nix

raymcdermott15:12:07

like cat | sed | awk but named

raymcdermott15:12:44

so more like proceess1 > pipe1 & ; process2 < pipe1

michaeldrogalis15:12:43

Okay, I follow that.

raymcdermott15:12:11

so stdin is one named pipe and stdout is another

raymcdermott15:12:49

and one if fed from Kafka and the other feeds out to Kakfa (with topics also named or anonymous)

michaeldrogalis15:12:30

Gotcha. That transportation model most closely matches what Samza does. There are a number of Real Life TM things that need to be dealt with, though. Things like serialization, implementing acking in an efficient way, and maybe biggest in my mind - the performance profile

michaeldrogalis15:12:57

I think you could do it for sure, but that takes you into the territory of a new platform architecture.

raymcdermott15:12:45

er, yes and we don’t want that!

raymcdermott15:12:56

which is why I came here simple_smile

michaeldrogalis15:12:31

@raymcdermott: Heh. simple_smile Well, I understand your end goal now. I'll think on it over the weekend and see how we can experiment with this.

raymcdermott15:12:52

@michaeldrogalis: thanks - that’s very generous of you

michaeldrogalis15:12:38

No prob. Out of curiosity, what's the commercial usage context?

raymcdermott15:12:08

we work at a global car manufacturer and this is to link data between customers / product definitions / vehicles and retail

raymcdermott15:12:50

there is a lot of data in silos that we need to get on to streams and then start shipping around

raymcdermott15:12:13

we are mostly Java and JS but want to be open to other technology too

raymcdermott15:12:45

to be honest the chance that I would be able to sell Onyx as the backbone here is a little dreamy … but its a New Year next week so we are allowed to dream a little simple_smile

raymcdermott15:12:37

I have to go now but thanks for the generous time and hopefully speak next week

michaeldrogalis15:12:37

That's understandable. simple_smile