This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2018-01-09
Channels
- # adventofcode (1)
- # aleph (2)
- # beginners (28)
- # boot (26)
- # boot-dev (8)
- # cider (10)
- # clara (10)
- # cljs-dev (130)
- # cljs-experience (1)
- # cljsrn (12)
- # clojure (118)
- # clojure-austin (40)
- # clojure-boston (1)
- # clojure-chicago (1)
- # clojure-dusseldorf (1)
- # clojure-estonia (11)
- # clojure-france (1)
- # clojure-greece (3)
- # clojure-italy (19)
- # clojure-nl (1)
- # clojure-russia (1)
- # clojure-spec (19)
- # clojure-uk (34)
- # clojurescript (62)
- # core-logic (7)
- # cursive (11)
- # datomic (35)
- # emacs (15)
- # fulcro (264)
- # jobs (4)
- # leiningen (5)
- # midje (4)
- # off-topic (74)
- # onyx (27)
- # planck (14)
- # protorepl (4)
- # re-frame (37)
- # reagent (62)
- # rum (2)
- # shadow-cljs (171)
- # slack-help (5)
- # spacemacs (6)
- # specter (9)
or use queue?(This option will have to send the same data to the queue continuously)
@a.espolov we currently do this in two ways (there may be others) - either put a message on kafka for each request and let onyx control the concurrency / flow, or use either onyx batches ( http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/batch-size ) or implicit batches (each segment contains data for multiple requests) and then dispatch the request batch with async i/o (we use aleph as our http client)
we commonly have 10k simultaneous api requests which get executed in onyx batches of 1k iirc
we are currently running 3 onyx instances
what do you mean ? how many cores across all onyx processes ?
i'm not sure how useful a metric that is - we aren't cpu bound and there are other processes running on the same instances
the best answer i have off the cuff is that each onyx process is running on an aws m4.xlarge with 4 vcpus, and has 1 cpu share assigned in dc/os...
but i know from monitoring that there are between 100-200 threads (mostly in pools to service async ops) in each onyx process, so we don't have single threads monopolizing hardware threads
@a.espolov Im not sure that that's necessarily a fit for a streaming problem.
@michaeldrogalis onyx is not suitable for solving the problem periodic polling REST API?
I mean, it could. I'm just saying I'm not sure streaming is the best approach given your initial message.
@michaeldrogalis I found my issue. I needed to add [mysql/mysql-connector-java "5.1.25"] [org.postgresql/postgresql "42.1.1"]
to my dependencies. Are only MySQL and PostgreSQL supported for the SQL plugin or would I be able to use something else like SQL Server?
@dbernal Theoretically anything that supports the JDBC interface ought to work
@michaeldrogalis ok perfect. Thank you!
Anytime!
@a.espolov @mccraigmccraig http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/batch-fn-QMARK is also handy if you’re not relying on an output plugin for async batching.
oh, yeah, i should have referenced that too 😊
@dbernal The tests in onyx-sql are your best bet.
@michaeldrogalis ok cool. Thanks!