Fork me on GitHub
#onyx
<
2017-02-07
>
lucasbradstreet00:02:12

Is the take-segments! being called at all?

lucasbradstreet00:02:28

If the take-segments! is being called, then your problem is that this job will never end because there is no end-tx

lucasbradstreet00:02:51

and it’s a log / streaming input, so you will never get to a point where a :done is written to the output channel, and thus take-segments! will never yield

lucasbradstreet00:02:15

try taking from the channel with a regular core.async take function (`<!!`, etc)

rc114005:02:19

so im fairly sure this is in the docs (but im just missing it right now) , if i have two jobs that have 10 tasks each (assume no minimum peers are set) and I only have 10 peers , will the tasks be assigned to peers as soon as they are submitted (by calling submit job) or will they only be assigned when the job actually runs (i.e it receives input from its input tasks)

rc114005:02:29

sorry if I confused some of the terms

michaeldrogalis05:02:29

@rc1140 The job starts immediately on submit-job if there are enough peers to satisfy the job, it won’t wait for input before starting the tasks.

michaeldrogalis05:02:44

As for your specific scenario, it depends on which job scheduler you choose. http://www.onyxplatform.org/docs/user-guide/latest/#scheduling

michaeldrogalis05:02:01

Two jobs with 10 tasks each = 20 tasks. You have 10 peers, so only one job can start.

rc114005:02:16

bah , i thought it may be like that

rc114005:02:46

need to re look at the schedulers

rc114005:02:51

to see if there is one that can work

rc114005:02:02

basically need to run a buch of scheduled jobs in parrallel

rc114005:02:04

but the jobs read their input from a kafka chan which is what actually triggers the start of the jobs workflow

jeremy05:02:54

thanks michaeldrogalis and lucasbradstreet! the finally was never being called, so it was blocking in taking-segments!, but you're right that since the job never completes with a :done (because i didn't set an :datomic/log-end-tx, i presume), it never returned. so i added a loop with <!! as suggested and it seems to be working. thanks again!

rc114005:02:57

@michaeldrogalis the onyx docs mention that you should try keep the number of vpeers close to the number of cores on the machine , has any adverse effects been observed when you dont do that

rc114005:02:25

i.e. if i choose to allocate 1000 peers per server on a 12 core server for exampel

michaeldrogalis05:02:44

@jeremy Hooray! 🙂 Let us know if you get stuck again.

michaeldrogalis05:02:03

@rc1140 Yeah you’re going to get serious thread contention.

michaeldrogalis05:02:32

Onyx will end up context switching so frequently that almost no progress will get made.

rc114005:02:51

is there a better way to approach this , i.e. the starting and stopping of the scheduled jobs

rc114005:02:14

should i just call submit job as needed and then manually kill it off when its done

michaeldrogalis05:02:18

@rc1140 If it’s a job with only batch inputs, the job will stop by itself when it’s processed all the data. If it’s a streaming job, you must bring it down with onyx.api/kill-job, otherwise it presumes there’s more input coming.

rc114006:02:30

ok so effectively all i need to do is append a :done to my input data and it will kill itself after processing the job

michaeldrogalis16:02:54

@rc1140 Correct, if you’re using core.async or kafka. Things like SQL, S3 and some versions of the Datomic input plugin, will terminate on their own.