Fork me on GitHub
#off-topic
<
2020-02-29
>
Ahmed Hassan11:02:26

How do we analyze and decide whether PostgreSQL or Kafka should be used for specific problem or service? I think fundamental question to ask is whether we need a relational database for storing facts or an event driven queue and k-v store. But there is a lot of overlapping area. views? experiences?

kulminaator11:02:30

the question is confusing to say the least 🙂

kulminaator11:02:18

if you need relational queries, acid transactions - then you need a relational database....

kulminaator11:02:41

and if you need a messaging system with persistent storage then kafka is a good candidate ... but since these are so different things (at least in my ears), your question sounds like should one pick a boat or a greenhouse.

kulminaator11:02:43

event sourcing or event driven mechanisms can be built on both, but you should really go back to the white board to figure out what you are doing in the first place :thinking_face:

✔️ 4
Ahmed Hassan11:02:01

@U6MHHF36J thanks. I wanted to ask from the perspective of different kind of applications. In the sense of appropriate tool for the job. Like if I'm building a ticketing application for buses, which one should I choose. I think question is very broad, that's why it's confusing.

kulminaator12:02:52

if you get to a point where you need to do a cross join between tickets, buc companies, start and end station tows .... you will likely start to want a relational db ....

👍 4
ataggart17:02:44

@UCMNZLJ93 For those interested in this area, I strongly recommend reading Designing Data-Intensive Applications.

💯 20
Timur Latypoff10:03:58

I’d say, always go PostgreSQL. If you can’t, then try really hard to find a way. If that doesn’t work, and DB experts say you should use Kafka in that specific case, you can try using that cautiously :)

❤️ 4
fmjrey12:03:32

Kafka is great if 1/ you can afford the added complexity and ops 2/ you have or will have multiple producers, consumers, or data sinks that need to share data 3/ have sufficient amount of data to deal with, unless 1 is a non issue (e.g. ops is someone else's problem)

4
fmjrey12:03:24

If you do not have complex querying logic but have data that keeps accumulating you could imagine using Kafka. It's been used successfully for a redirect service in a past job, whereby the service would load all redirects from kafka upon starting, and you are able to see all operations made on a redirect so you can manage them properly, including detect unused ones so you can delete them after checking with original owners

fmjrey12:03:44

The volume of redirect was in the order of a few tens of thousands, and the logging of their usage is done on a separate topic from the config one

fmjrey12:03:53

The team however was already managing a kafka cluster

bartuka14:02:14

that feeling when a great conference is happening and you are in the other side of the world looking at tweets from attendees about all the awesome presentations

Crispin14:02:42

haha. Im following the tweet stream too

Crispin14:02:58

people at clojureD... tweet more!