Clojurians Log v2

@thom704 It is a cool idea to process postgresql job queues from clojure workers, but clojure backend can toss scheduled jobs to postgresql as well.

there is one problem we faced with this ofc, which is that scaling postgres may not be that optimised per task, like if you just wanted to scale up queues etc. that being said, scaling postgres isnt that expensive so

37signals uses https://github.com/rails/solid_queue for job queues.

https://dev.37signals.com/introducing-solid-queue/

Also, depending on your comfort in managing it or not. I'm a pretty big fan of Aurora serverless V2 using the data API. It's just SQL over rest, no DB drivers, no connection pooling, no connection management, I love it.

the advice isn't wrong, but the article definitely seems like a AI-generated SEO clickfarming regurgitation of similar arguments that have been made elsewhere without attribution

So, what's your advice?

use "before:2023" in your google searches

I certainly don't have a specific use case yet, so starting with postgresql sounds good. I don't know whether I should manually manage postgresql or rent managed postgresql instances.

There is something to be said for reducing the number of moving parts in your system -- and that can include using a single DB tech for a wide variety of processes. At work, we mostly use MySQL. We have tried various diversifications over the years but each time we've ended up returning to MySQL, to reduce complexity and overall infrastructure costs. One thing that's nice about settling into PostgreSQL is that there's an embedded version you can use for testing -- see what next.jdbc does for testing -- so you can have a throwaway instance as needed.

Instead of using either Clojure or Postgres as a cron daemon, you also have the option of using Postgres as a job queue and using FOR UPDATE SKIP LOCKED from some Clojure workers. On the whole you should have a specific reason for requiring an architecture more complicated that one Linux process and a relational database, but many such reasons exist.

I support using Postgres over a plethora of Postgres-like tools, like the one the article mention. However, I'm too fond of Datomic to ignore it. It all depends on context. If you know a bit of Clojure and a lot of SQL, Postgres is great! If you want to leverage Rich's full vision for decomplected software, Datomic is worth putting some time into. Regardless — I strongly support having fewer moving pieces in your system. If you already have Postgres in your stack, I'd hesitate before adding lots of other tools.

Or use XTDB instead of Datomic -- bitemporal, open source, and the primary API is still SQL and it's compatible with PG-wire so all PostgreSQL tools/drivers work with it out of the box. In particular, next.jdbc is tested against XTDB (every CI run). HoneySQL also has support for most of the XTDB SQL extensions. So you get immutable persistence, temporal queries, and "standard" tooling.

MySQL is fine. When you need to scale MySQL, I think there is vitess although vitess is quite complex. Someone started multigres which is vitess for PostgreSQL. Multigres is still being developed.

We use the Percona fork of MySQL and have a primary replicated to a secondary, and run reporting style queries off the secondary. Our database is very large -- we have several tables with over 250M rows, and quite a lot over 100M. But it still very performant. And if you need to scale beyond that, there's always AWS (RDS or Aurora)...

Do you manually manage mysql instances or rent managed mysql instances?

We have managed servers in an East Coast data center. They manage our MySQL instances completely for us.

They're actually virtualized servers, so it's sort of a private cloud 🙂

I guess I will start with manually managing my postgresql database.

I just discovered https://news.ycombinator.com/item?id=47123631 for postgresql sharding without extensions. multigres is also emerging. If I ever want multiple postgresql instances for performance or any other reason, these things are available. My app doesn't need to change. Even after multigres or pgdog becomes production-ready, I probably won't need anything more than vertical scaling for a long time.