Fork me on GitHub
#onyx
<
2016-08-15
>
devth15:08:51

ok, great! I'll test it out in the next few days.

michaeldrogalis15:08:04

Artifact deployed @ onyx-elasticsearch-0.9.9.1-20160815.153513-2.

Travis19:08:07

Is there anything special that needs to be done in order to send a sentinel value in for kafka? I set the kafka/partition value and published :done to the topic/partition but the job continued to run

michaeldrogalis19:08:56

@camechis: Sentinel values aren't supported with multi-partition Kafka topics.

Travis19:08:16

so in our case we made one partition

michaeldrogalis19:08:21

Reliably detecting end-of-stream for that scenario is error-prone, so we punted there.

michaeldrogalis19:08:26

Ah, should work alright then.

Travis19:08:52

How does the data need to be formatted for the sentinel? From the kafka client

michaeldrogalis19:08:02

Just :done. Single keyword

Travis19:08:13

ok, i sent in as a string assuming it would get serialized and deserialized. Using the same Kafka api as the kafka plugin

Travis19:08:44

ah, maybe i should have used the edn Serializer?

michaeldrogalis19:08:53

Strings wont be coerced to keywords

Travis19:08:11

thought that might be the issue so looks like i need to use the edn serializer

Travis19:08:24

so it the kafka client will accept the keyword

michaeldrogalis19:08:33

Yup, that should do it.

michaeldrogalis19:08:53

Probably in the future we'll support an offset as a marker for completion, and avoid using a special value.

michaeldrogalis19:08:55

That'd be far more robust.

Travis19:08:30

awesome, we are using this more for testing out stuff on our cluster with windowing/triggers

Travis19:08:23

one thing we are still experiencing is a lot of retries according to our metrics. Need to definitely figure out whats going on there