This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2015-12-30
Channels
- # admin-announcements (42)
- # alda (8)
- # aws (10)
- # beginners (22)
- # boot (165)
- # bristol-clojurians (1)
- # cider (6)
- # clara (21)
- # cljs-dev (23)
- # clojure (74)
- # clojure-dev (8)
- # clojure-russia (41)
- # clojurescript (180)
- # core-async (11)
- # cursive (26)
- # datascript (2)
- # datavis (7)
- # datomic (29)
- # editors (1)
- # hoplon (7)
- # jobs (3)
- # ldnclj (4)
- # lein-figwheel (47)
- # leiningen (2)
- # mount (26)
- # off-topic (3)
- # om (163)
- # onyx (56)
- # proton (4)
- # reagent (6)
- # remote-jobs (1)
- # ring-swagger (4)
- # spacemacs (9)
It seems like the JVM is a pretty heavy dependency. I understand that but we would like to support composition in a stackless way. Maybe this is just too much of a dream at the moment and seems like a non-goal for Onyx (which is totally reasonable!)
Yeah, it'd be a "nice to have" for us, but is a non-goal/not a priority
@raymcdermott: For me right now, that feels like implementing Clojure for real on the CLR after having implemented it on the JVM. Tiring. I have an idea if you're still willing to entertain the JVM as a thing that needs to be there, but not interacted with directly.
@michaeldrogalis: go for it
We could set up a small wrapper in JavaScript that talks to Onyx's JVM peer via a local socket. The local socket would shuttle segments back and forth between the JVM and the JS runtime. Also, to facilitate the full power of Onyx, we'd maintain local state inside the JS runtime to mirror the event map. The socket would also send notifications of when to invoke lifecycle hooks so it could transition its local state accordingly.
It's a half-baked idea I had falling asleep last night, so I'm not positive that can work as stated - but I don't think it's too far off.
I think it’s an interesting possibility … we were thinking of Docker as the unit of functionality and Kafka as a pipe. The problem we are having is that we cannot expect all clients to all be able to consume data from Kafka
our simple thought was to put data on to named pipes and have consumers read from them
but its very ad-hoc compared to Onyx - which is more closely competing with Spark if I understand properly
Right, yeah. I'd love to work on that, but something that big would be a commercial project for me.
so I’m probably way off topic but defining flow as a set of functions could fit over a more adhoc clow
Seems really neat.
I see it a little differently. When you're using functions to define flow, you're locked into a specific language. Pushing that stuff out into data is what leads you into what you call stackless territory.
ok - thanks for entertaining the crazy ideas and I think the JS wrapper with a socket could be an interesting option anyway!
Can you define what exactly you mean by stackless?
Ah, okay. I grew up hearing the term "polyglot stack" for that. Alrighty, I'm on the same page. That's the ultimate goal of Onyx.
One of them, anyway.
Ideally we're looking at being able to say inside each catalog entry :onyx/language :ruby
, and the scheduler can do appropriate placement.
I'm less enthusiastic about isolating it to the extent that you are. That would require implementing all the threading, error handling, retry logic etc in every language. Definitely not feasible, for even a large engineering team.
The cost of adding another language would be astronomical.
Backing up once more - named pipes?
Okay, I follow that.
and one if fed from Kafka and the other feeds out to Kakfa (with topics also named or anonymous)
Gotcha. That transportation model most closely matches what Samza does. There are a number of Real Life TM things that need to be dealt with, though. Things like serialization, implementing acking in an efficient way, and maybe biggest in my mind - the performance profile
I think you could do it for sure, but that takes you into the territory of a new platform architecture.
@raymcdermott: Heh. Well, I understand your end goal now. I'll think on it over the weekend and see how we can experiment with this.
@michaeldrogalis: thanks - that’s very generous of you
No prob. Out of curiosity, what's the commercial usage context?
we work at a global car manufacturer and this is to link data between customers / product definitions / vehicles and retail
there is a lot of data in silos that we need to get on to streams and then start shipping around
to be honest the chance that I would be able to sell Onyx as the backbone here is a little dreamy … but its a New Year next week so we are allowed to dream a little
That's understandable.
Yep, take care!