Clojurians Log v2

Over the past several days I've been poking around with XTDB v2 and trying out different log/storage backends. The operational story for xtdb2 seems pretty nice: • in dev you can use the local filesystem for both log and storage. • in prod, if you're only running on a single machine as most biff users probably are, you can start out with local filesystem for the log and S3 (digitalocean spaces) for storage. • if/when you want to start running multiple machines, you can run a redpanda instance (drop-in kafka replacement) on each machine. some notes: • It's easy to transition from a filesystem log to a kafka/redpanda log since XT v2 mainly uses object storage for persistence. The log is only needed as a temporary place to put transactions that haven't been indexed yet. So the migration process is: (1) include e.g. :healthz {:port 8123} in your config; (2) run curl -X POST to trigger a "storage flush" (write everything in the log to s3); (3) restart your node with the kafka config instead of filesystem log config, and set :epoch 1 . • By default XT will do a storage flush at least once every 4 hours. So if you're doing filesystem-log-s3-storage in prod, and assuming having backups that are up to 4 hours out-of-date is ok, then you don't need to do anything else for backups. This is a nifty improvement over XT v1 where using a managed postgres instance on digitalocean is another $15... or more once you start running out of disk space. whereas digitalocean spaces is only $5/month for 250 GB, and you'll often want it for other stuff too. • I looked into a handful of kafka replacements (this seems to be a pretty active space). At first I was looking for/at ones that could write to object storage. However once I realized that XT already relies mostly on object storage for persistence as discussed above, that seemed like a moot point. So Redpanda seems like the way to go. By default it took up a couple gigs of ram on my local machine, but I was able to run it on as little as 128MB (`sudo rpk redpanda start --smp 1 --memory 128M --overprovisioned --unsafe-bypass-fsync=false`). Seems like the pathway for scaling here should be pretty smooth: e.g. if you just need to start using a worker machine but aren't trying to scale out a bunch, you can start with running redpanda on the same machines as your app processes in dev mode with reduced memory (i.e. the rpk command previous). If anyone using Biff actually manages to build something that needs to scale further, you can add more memory/start running redpanda on dedicated machines/switch to other kafka implementations (like digitalocean's managed kafka, which starts at $45/month, or managed redpanda, or whatever). • also it's nifty that since the log doesn't need to have infinite retention, running redpanda (which stores all its data on the local filesystem) shouldn't ever take up much disk space (though I guess with v1 the vast majority of disk usage IME came from docs storage and index anyway, not the tx log, so perhaps it's a moot point). With the operational stuff figured out, next step is I'll update the starter app to use XT v2 and figure out what the migration process will look like for existing apps, while still having Biff maintain support for XT v1 for anyone who would rather not migrate. I'm also planning to take all of Yakread's data (currently ~20 GB, including tx log + docs + index) and see how fast various queries are in XT v2 vs. v1. (Yakread does a lot of big queries...)