Fork me on GitHub
#onyx
<
2018-02-14
>
misha14:02:40

not exactly onyx question, but "can I do paginated consumption from kafka topic with gregor?"

misha14:02:02

say, 10 or less messages at a time

gklijs14:02:44

you can set ConsumerConfig/MAX_POLL_RECORDS_CONFIG to allow a maximum per poll, but you will get back less, if there are less ready

misha14:02:39

sweet! thanks @gklijs

misha14:02:48

for gregor config will look like {"max.poll.records" "10"}

gklijs14:02:54

It’s nice to have, and something which was missing before. I don’t know if a heartbeat is now send when not polling, bu we had problems before when consumers where taking to long, and thought dead, because they took to long after a pol

gklijs14:02:37

Could be, I only used java interop so far when working with clojure and kafka

misha14:02:05

yeah, writing UI for onyx ETL job reports, and if you are lucky to consume closed topic for huge input file – chrome hangs for minutes opieop

misha14:02:07

"max.poll.records" is kafka's setting name. gregor seems to just send it as string w/o any changes. will try it in a few minutes

gklijs14:02:33

That’s one of the reason I like the java interop, by using static strings there is less risk of making types + when they might rename/delete it in a newer version your covered.

misha14:02:00

#clojure-spec :)