This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2016-06-29
Channels
- # admin-announcements (4)
- # arachne (19)
- # aws-lambda (3)
- # beginners (10)
- # boot (166)
- # capetown (32)
- # carry (160)
- # cider (5)
- # cljs-dev (5)
- # cljs-edn (19)
- # cljsrn (1)
- # clojure (100)
- # clojure-belgium (2)
- # clojure-dev (8)
- # clojure-greece (13)
- # clojure-new-zealand (12)
- # clojure-poland (1)
- # clojure-russia (93)
- # clojure-sanfrancisco (2)
- # clojure-spec (133)
- # clojure-uk (52)
- # clojurescript (129)
- # cursive (32)
- # datomic (13)
- # defnpodcast (5)
- # devcards (6)
- # dirac (4)
- # emacs (12)
- # euroclojure (5)
- # events (2)
- # hoplon (19)
- # immutant (45)
- # keechma (17)
- # lein-figwheel (27)
- # off-topic (9)
- # om (30)
- # onyx (17)
- # other-languages (3)
- # planck (2)
- # proton (11)
- # re-frame (7)
- # reagent (4)
- # ring (8)
- # sim-testing (2)
- # spacemacs (4)
- # testing (2)
- # untangled (162)
- # utah-clojurians (1)
- # yada (80)
Hello. I'm starting a new project, and I plan to make Onyx a heart of all data processing here. However, historically a lot of things were done in Python here (a lot of natural language processing, specific to a domain), so I can't just throw it out, and if we're going to phase it out someday and reimplement in Clojure, it will be done gradually. As I understand, you have some plans about making Onyx available for other languages, I've taken a look at onyx-ruby PoC, but it's not here yet, and it's not really an urgent priority
Basically, what I need is a foreign function calls, and right now I think I can get away with interacting from Onyx task with my Python code via http api, or something like that
I've taken a look at an approach that pyspark uses, and I think I'll be better of using a simpler thing
That sounds reasonable. I think to do so you will want something like :onyx/batch-fn
, which is something I’ve been considering adding. That would let you send a whole batch of segments to your end point
Or allow you to asynchronously make batch-size calls, and return the whole batch when you’re done
Well, it depends on the time that's needed to process one segment, if it takes a couple of seconds, I can just do it one-by-one
That would improve performance a lot. Ideally you’d use something like urania to handle the requests over the batch http://funcool.github.io/urania/latest/
Yes, that is very true
It totally depends on what you’re doing
cool, I'll take a look at urania
Is there an example of using the new task bundles approach using Kafka? I can't figure out how you would add a kafka task using task bundles?
No example yet per-say but the task bundles are available https://github.com/onyx-platform/onyx-kafka/blob/0.9.x/src/onyx/tasks/kafka.clj
gardnervickers: is that code published as a jar in Clojars/Maven Central or will I need to build from that branch?
All our plugins are tested and published against both official releases and our snapshot builds https://github.com/onyx-platform/onyx-kafka
[org.onyxplatform/onyx-kafka "0.9.6.0”]
is the current lein coordinate
Thanks
@agile_geek: For reference, our build matrix links out to every dependent project, and its coordinates are on the top of every README if they exist: https://github.com/onyx-platform/onyx#build-status