Fork me on GitHub
#jackdaw
<
2019-09-09
>
bartuka03:09:13

I have a question about how to deploy several topologies in clojure projects. Usually you have a system component which fires up all the topologies defined in the code?

creese00:09:54

While its possible to start multiple topologies for a test, in production, we typically build these as separate uberjars.

bartuka16:09:32

uhmmm.. interesting... but that option will not be hard to manage after some time? I have never used Kafka in production systems, however the examples I see on the internet, the topology is a single "stream-processing-units", what I mean is each topology accomplishes a single "business objective" which could be proecssing a stream and publishing to another topic. But, the idea is to have many of these, right?

bartuka16:09:48

Maybe the definition of topology is what I might be missing at

creese15:09:27

A processor topology is just a collection of processor nodes you attach to a StreamsBuilder when defining your app. With the Kafka Streams DSL, these can be quite complex.

creese15:09:44

Our core accounting ledger has 40 processor nodes defined as one “topology”. It’s technically more than one topology because we use through but that happens under the hood.

creese15:09:36

We have separate topologies to handle different aspects of accounting. These could all be one “thing” but we wanted to able to deploy them to production separately.

creese16:09:26

You can develop all your “apps” in the same code repository if you want. We have about five “apps” in one ours.

dcj04:09:13

Apologies if this has been posted before, but I found this article well worth reading... https://clojure-conundrums.co.uk/posts/kafka-streams-the-clojure-way/