This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
- # admin-announcements (42)
- # alda (8)
- # aws (10)
- # beginners (22)
- # boot (165)
- # bristol-clojurians (1)
- # cider (6)
- # clara (21)
- # cljs-dev (23)
- # clojure (74)
- # clojure-dev (8)
- # clojure-russia (41)
- # clojurescript (180)
- # core-async (11)
- # cursive (26)
- # datascript (2)
- # datavis (7)
- # datomic (29)
- # editors (1)
- # hoplon (7)
- # jobs (3)
- # ldnclj (4)
- # lein-figwheel (47)
- # leiningen (2)
- # mount (26)
- # off-topic (3)
- # om (163)
- # onyx (56)
- # proton (4)
- # reagent (6)
- # remote-jobs (1)
- # ring-swagger (4)
- # spacemacs (9)
It seems like the JVM is a pretty heavy dependency. I understand that but we would like to support composition in a stackless way. Maybe this is just too much of a dream at the moment and seems like a non-goal for Onyx (which is totally reasonable!)
@raymcdermott: For me right now, that feels like implementing Clojure for real on the CLR after having implemented it on the JVM. Tiring. I have an idea if you're still willing to entertain the JVM as a thing that needs to be there, but not interacted with directly.
It's a half-baked idea I had falling asleep last night, so I'm not positive that can work as stated - but I don't think it's too far off.
I think it’s an interesting possibility … we were thinking of Docker as the unit of functionality and Kafka as a pipe. The problem we are having is that we cannot expect all clients to all be able to consume data from Kafka
our simple thought was to put data on to named pipes and have consumers read from them
but its very ad-hoc compared to Onyx - which is more closely competing with Spark if I understand properly
Right, yeah. I'd love to work on that, but something that big would be a commercial project for me.
so I’m probably way off topic but defining flow as a set of functions could fit over a more adhoc clow
I see it a little differently. When you're using functions to define flow, you're locked into a specific language. Pushing that stuff out into data is what leads you into what you call stackless territory.
ok - thanks for entertaining the crazy ideas and I think the JS wrapper with a socket could be an interesting option anyway!
Ah, okay. I grew up hearing the term "polyglot stack" for that. Alrighty, I'm on the same page. That's the ultimate goal of Onyx.
Ideally we're looking at being able to say inside each catalog entry
:onyx/language :ruby, and the scheduler can do appropriate placement.
I'm less enthusiastic about isolating it to the extent that you are. That would require implementing all the threading, error handling, retry logic etc in every language. Definitely not feasible, for even a large engineering team.
and one if fed from Kafka and the other feeds out to Kakfa (with topics also named or anonymous)
Gotcha. That transportation model most closely matches what Samza does. There are a number of Real Life TM things that need to be dealt with, though. Things like serialization, implementing acking in an efficient way, and maybe biggest in my mind - the performance profile
I think you could do it for sure, but that takes you into the territory of a new platform architecture.
@raymcdermott: Heh. Well, I understand your end goal now. I'll think on it over the weekend and see how we can experiment with this.
we work at a global car manufacturer and this is to link data between customers / product definitions / vehicles and retail
there is a lot of data in silos that we need to get on to streams and then start shipping around
to be honest the chance that I would be able to sell Onyx as the backbone here is a little dreamy … but its a New Year next week so we are allowed to dream a little