Fork me on GitHub
#onyx
<
2018-01-09
>
a.espolov11:01:26

Guys how can I use Onyx to solve the problem of executing 10 000 requests to APIs?

a.espolov11:01:52

Do I have to implement my plugin?

a.espolov11:01:54

or use queue?(This option will have to send the same data to the queue continuously)

mccraigmccraig11:01:23

@a.espolov we currently do this in two ways (there may be others) - either put a message on kafka for each request and let onyx control the concurrency / flow, or use either onyx batches ( http://www.onyxplatform.org/docs/cheat-sheet/latest/#catalog-entry/:onyx/batch-size ) or implicit batches (each segment contains data for multiple requests) and then dispatch the request batch with async i/o (we use aleph as our http client)

mccraigmccraig11:01:43

we commonly have 10k simultaneous api requests which get executed in onyx batches of 1k iirc

a.espolov11:01:19

one or more server on cluster?

mccraigmccraig11:01:59

we are currently running 3 onyx instances

a.espolov11:01:40

summary count cores?)

mccraigmccraig12:01:21

what do you mean ? how many cores across all onyx processes ?

mccraigmccraig12:01:02

i'm not sure how useful a metric that is - we aren't cpu bound and there are other processes running on the same instances

mccraigmccraig12:01:31

the best answer i have off the cuff is that each onyx process is running on an aws m4.xlarge with 4 vcpus, and has 1 cpu share assigned in dc/os...

mccraigmccraig12:01:37

but i know from monitoring that there are between 100-200 threads (mostly in pools to service async ops) in each onyx process, so we don't have single threads monopolizing hardware threads

michaeldrogalis15:01:46

@a.espolov Im not sure that that's necessarily a fit for a streaming problem.

a.espolov15:01:15

@michaeldrogalis onyx is not suitable for solving the problem periodic polling REST API?

michaeldrogalis15:01:56

I mean, it could. I'm just saying I'm not sure streaming is the best approach given your initial message.

dbernal15:01:55

@michaeldrogalis I found my issue. I needed to add [mysql/mysql-connector-java "5.1.25"] [org.postgresql/postgresql "42.1.1"] to my dependencies. Are only MySQL and PostgreSQL supported for the SQL plugin or would I be able to use something else like SQL Server?

michaeldrogalis16:01:34

@dbernal Theoretically anything that supports the JDBC interface ought to work

dbernal16:01:48

@michaeldrogalis ok perfect. Thank you!

mccraigmccraig17:01:05

oh, yeah, i should have referenced that too 😊

dbernal22:01:37

are there any examples available of a job that reads from a SQL table?

michaeldrogalis22:01:16

@dbernal The tests in onyx-sql are your best bet.