Fork me on GitHub
#onyx
<
2015-08-17
>
erichmond12:08:51

If you want the properties of kafka without managing it, you could use amazon kinesis

erichmond12:08:20

you get replayability for 24 hours, which beats any RAM-backed queue

erichmond12:08:36

and if the messages are infrequent, you won’t break the 1mb/sec message barrier

noisesmith16:08:37

what's the effect of :onyx/batch-size in a catalog entry when you don't have :onyx/bulk set to true? I am suspicious that my setup stops working after the batch-size is hit, and I'm probably misconfiguring something here...

noisesmith16:08:11

setting :onyx/batch-size to Long/MAX_VALUE eliminates the symptom, but I think it might not be fixing the actual problem here

michaeldrogalis20:08:20

@noisesmith: Batch size is a performance knob that scoops up messages together before performing actions. It's to increase the payload being sent over the wire in the case that a single message leaves you network bound. :onyx/bulk? is simply a functional application to all the segments collected by the batch. Bulk and batch-size have no functional correctness properties that tie them together.

noisesmith20:08:18

for some reason my workflow just stopped accepting tasks after a certain number were handled, and upping batch-size eliminated the symptom

noisesmith20:08:28

clearly the error is somewhere else

michaeldrogalis20:08:11

Tasks, or segments?

michaeldrogalis20:08:38

Are there exceptions in onyx.log? It sounds like your batch was partial processed and blew up.

noisesmith20:08:42

no errors, but I do see the suspicious looking message "Lifecycle chose not to start the task yet. Backing off a nd retrying..."

michaeldrogalis20:08:07

Is that once? Or repeated many times?

noisesmith20:08:00

sorry, I was looking at the wrong end of the log, so that definitely isn't it

michaeldrogalis20:08:03

FYI, we're going to be putting on private training sessions about creating, managing, and monitoring Onyx infrastructure if your place is interested.

noisesmith20:08:44

oh cool, where's a good place to get signed up for that email list?

noisesmith20:08:11

because I'll definitely try to talk my folks into paying for a training

michaeldrogalis20:08:16

Email <mailto:[email protected]|[email protected]> if/when you want to try it. Still crafting the material for it, but we'd like to do one-on-one sessions with companies and tailor the content to their problem domain.

noisesmith20:08:50

@michaeldrogalis: thanks for the tip about onyx.log - I had a separate issue in my job code and hadn't been watching that log file for stack traces. Is it possible to turn off ansi coloring in that file? I assume this is being done by some stack trace prettifier that onyx is pulling in

michaeldrogalis20:08:50

@noisesmith: I haven't tried it, but we're using Timbre for logging, which uses Pretty(https://github.com/AvisoNovate/pretty) for coloring. Perhaps try to exclude Pretty from the project?

noisesmith20:08:16

cool, thanks

noisesmith20:08:29

I'm not reading logs in a format where the ansi colors are helpful atm...

michaeldrogalis20:08:31

Yeah I've been there, it's hella annoying when that's turned off. If you want to file an issue, I can make an Onyx switch for it when I get time