Clojurians Log v2

http://hetzner.de has cheap and decent VPS and dedicated servers. I would look into provisioning a VPS there and use https://kamal-deploy.org/ to deploy. Checking out this alley is on my list at least 🙂

Very interesting. I was not aware of kamal. It’s definetly something I’ll explore myself! Cheers

datomic-cloud has a setup cost, for sure. But then, the operational cost is very, very low. Deployments are just running (on CI) clojure -M:ion "{:op :deploy :group \"our-compute-group\" :rev \"$CURRENT_REV\"}" And it scales by turning a dial. Also just found this: https://github.com/vouch-opensource/datomic-ions-deploy

Awesome, I’ll look into datomic cloud + ions too

cloud looses out on the locality of peer, one its best feaures IMO

Locality of data + execution is still there in cloud. You just bind your fn as an ion over-there-where-the-data-is. Which is stratified: https://docs.datomic.com/operation/architecture.html#stratified-durability

Ah right. I thought ions and cloud where different things. Maybe it's changed since I last looked at it.

For http://fly.io, https://github.com/carrete/datopro-mic

I have been very pleasantly surprised by the amount of things I don’t have to think about hosting my application on Datomic Ions.

> I have been very pleasantly surprised by the amount of things I don’t have to think about hosting my application on Datomic Ions. Can you go into more details? Or do you have any good resources detailing the advantages? I’m curious about ions but haven’t dug too deep 😄

Datomic Ions is pretty nice It handles blue/green deploy, autoscaling, logging, metrics, etc etc etc But It requires datomic.client.api (and not datomic.api), which requires some code changes despite using the client api, it perform like datomic.api (in-memory aceess to the DB - no roundtrips) when it is running.

Here's a list off the top of my head of all the AWS infrastructure that Datomic Ions manages for me: AutoScaling, LoadBalancing, Lambdas (soup-to-nuts, 100%), API Gateways, VPCs, EC2 Instances, CodeDeploys, CloudWatch metrics, CloudWatch logs. And for Datomic Cloud itself, of course, there is S3, Dynamo, memcache, ELB, etc. That's only a partial list. It's not all perfect (I've opted to roll my own logging and metrics instead of using the built-in stuff) but otherwise, the Ion implementations have met my needs nicely in two completely different startups over the past ~four years.

For a startup, I think Datomic Ions is "The greatest force multiplier you've never heard of."

How do you work on a Datomic Ions system locally?

yes, but it requires some extra steps not easy as datomic:mem you need to "redirect" your connecton to local/file or work connected to "live system" (which is powerful) And some situations, like "testing a new transactor function" are pretty tricky to test

That's probably the biggest selling point: in general, you don't. Instead you know your plain Clojure functions will be called as handlers for all the Ion entry points (HTTP and lambdas) with pure data (documented in the Ion docs). So in terms of web servers and "reactions" to any AWS event that can be routed to a lambda (pretty much everything of interest), running and testing locally is no different that running and testing any function of a data structure (e.g. Ring handler).

If you want to test with a db, then @souenzzo’s comments are almost accurate (db local isn't really the same as datomic:mem).

as a programmer who has used both for several years datomic "peer" is way delightful to develop

Redirect is a clunky way to test and I have not done it in years.

I actually forgot that was a feature.

Instead, db-local or connect to cloud dbs.

@cch1 you can run it locally pedestal support both "jetty" and "ions" backend Not a issue here BUT • datomic ions supports ONLY HTTP (no websocket) • If you don't use JSON, this could be a problem • if you need very low response times, or partial-streaming responses, it may require some extra config in ions

We don't use pedestal (we use papillon) but similarly, it supports ions and aleph (for local development with a web server).

As for your BUT list...., I'm not sure why the "If you don't use JSON, this could be a problem". I've also never had an issue with response times that weren't directly attributable to expensive queries (IOW, a self-inflicted DB problem, not an Ion infrastructure problem). Our Ions support GraphQL, JSON:API and HTMX on the HTTP-direct entry point. On the lambda side, the list of sources includes SQS messages, EventBridge Scheduler events (which is effectively a distributed, fault-tolerant cron server) and a couple of other AWS services. I think just about every AWS service could ultimately trigger a lambda so Ions are very easy to integrate into the larger AWS ecosystem without having to code or deploy any lambdas yourself (it's built into the ion infrastructure).

@souenzzo, it's sounds like you are an experienced ion user. I'm curious, do you use the query groups feature at all?

AFIK, non-json payloads were encoded in base64 during APIg->lambda process this ended up implying a limit on the body size. Not sure it if was fixed (most of my knowledge about ions is from 2019~2020) integrate external AWS services into a ions is awesome: just add the security group and go And response times: APIg+lamda adds some unwanted millis to your requests. At scale it is pretty irrelevant, but for rendering hiccup on server, it may be noticeably slow (not slow in seconds, but "why does it takes a while to load an empty page") I didn't use query groups 😞 but there were plans to create one for each client - sharing the same storage.

I would probably host the transactor on digital ocean with jdbc binding to managed postgres instance, leave that running, and use http://fly.io to ship your peer backend. If you can get a larger disk on your http://fly.io node then use valcache

Hmmm.... I think the new HTTP direct feature might have been added after 2020 -it removed a lot of constraints on the HTTP side and improved performance (no more lambda in the middle IIUC)

> it may require some extra config in ions 🙂

wrt peer, one transactor on decent hardware, assuming you are starting with a good partitioning model, can do a LOT of throughput

and it's worth mentioning the aws scam: "just enable this service", yeah but to enable X you need Y and Y requires a source from Z also it is recommended enable W so you have visibility ..... and in a flash you have a whole alphabet of new services

> and in a flash you have a whole alphabet of new services I confirm this happened to me

The whole networking and security aspect of AWS has devolved into a morass that takes a PhD to unravel. And the consequences of getting it wrong are kinda scary. I am more than happy to trust Cognitect to manage that for me.

that’s a very real aws issue, but I would also say almost nobody else can offer something like Dynamodb without owning an entire data center

and its a nice capability of Datomic

easy to switch to if your app needs to grow though

About alternatives: one nohup ./transactor using dynamodb backend is quite enough to scale a startup until you can hire the infrastructure team.

Datomic Pro with SQLite is very operationally simple. Your deployment consists of two processes (your app and the transactor) and a SQLite file. I dare say this is the simplest possible production ready datomic deployment. IMO this simple deployment architecture + vertical scaling will take you very very far.

I'm going to disagree: even ignoring all the extras, a Datomic Ion app gets you the transactor and the hosting platform for your app deployed from one CloudFormation template. No real decisions need to be made and scaling is both trivially vertical (EC2 instance size -literally a drop-down select box) and horizontal (how many instances, with or without a query group). It does, of course, bind you to AWS. If that is unacceptable then Ions are not for you.

That is datomic ions. My proposed architecture is a single Linux box + nginx + two JVM processes. It will cost 5$/mo starting out. And it is straightforward to scale vertically as needed. This is more suitable for a SAAS using a bootstrap/starting template project like Ovi's.

Yes, fair. But the CloudFormation template Does All That. So I see it thusly: Some Experts have figured out the AWS pain. What I do: • Download the template, run it on AWS • Take some time understanding How To Ion • Write some fns, they become lambdas (and one HTTP direct endpoint) When a new release is available, update with that new template

That's been my experience as well... almost all the complexity of that diagram is not something I have to worry about -it's been taken care of by Cognitect in the CF template. I'm not sure what the minimum cost for running an ion-based app is, but my guess is less than $10/month.

Important point here is to understand what you get for the price. Running a SQLite/PG/writes-to-a-single-drive backend on a VPS doesn't have any of the same safety/redundancy/resiliency characteristics for your data-at-rest as using cloud storage services. If your trying to keep your startup lean by trading off storage risk, your whole startup might die overnight when the drive on the Hetzner instance dies. Is that worth risking your livelihood over?

So @joe.lane do you also recommend the way of datomic ions + datomic cloud?

Not necessarily, there are tradeoffs between each product. What I am saying is don't pick a less resilient backing storage just to save a few bucks. Make an informed decision based on your (or your prospective customer's) tolerance for risk.

For reference and ease in this thread I asked AI to summarize the discussion so far: Three Main Deployment Paths Emerged 1. Datomic Ions + Cloud (Recommended by several experienced users) Pros: • “Greatest force multiplier you’ve never heard of” - handles AWS infrastructure automatically • Auto-scaling, load balancing, deployments, monitoring all managed • Very low operational overhead once set up • Deploy with just: clojure -M:ion "{:op :deploy}" • Proven in production by multiple startups Cons: • Requires migrating from datomic.api to datomic.client.api • AWS lock-in • Local development is more complex (no datomic:mem) • HTTP only (no WebSockets - this could be a blocker for your Sente setup) 2. Hetzner VPS + Kamal (Modern, cost-effective) Pros: • ~€12/month starting cost • Full control over your stack • Supports WebSockets for your Sente integration • Modern deployment tooling • No vendor lock-in Cons: • You manage the infrastructure • Need to handle backups, monitoring, scaling yourself 3. Datomic Pro + SQLite (Simplest) Pros: • Just two processes: your app + transactor • Single SQLite file for storage • $5/month VPS cost • Extremely simple to understand and debug Cons: • Limited scalability • Single point of failure (as Joe Lane pointed out) For reference, currently the websocket setup can be changed with SSE, so that’s not a blocker

You can run a client-based application in cloud that runs your WebSockets / SSE just fine. I think there is a misconception that in Cloud the only way to interact with Datomic is through deploying your app as an ion. If I wanted websockets + ions I would consider using https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-websocket-api.html w/ ion-lambdas. Local development with cloud is not "more complex" it's just different. You can use Datomic Local or connect to your cloud system directly from your laptop via the client API. I would never purchase a product in 2025 that was storing my production data on a single machine when I have Cloud storage services available to me. The risks just aren't worth it. And backups don't count 🙂 If you wanna work through this more I suggest throwing it in a google-sheet and making a DM to compare the tradeoffs.

I'll back Joe's point about local development: db-local is fast, API-compatible with cloud/ion and trivial to setup. It is particularly awesome for running our test suite in a CI pipeline.

https://docs.datomic.com/datomic-local.html - I am using it (as we speak!) for a local cache in a manual-migration. :datoms 51904987!

> There are tradeoffs between each product. [...] Make an informed decision based on your (or your prospective customer's) tolerance for risk. 💯 agree. I think this is a key point that anyone interested should take away, there is no answer that will suit everyone, and asking an open-ended question along the lines of "what's the best/right deployment model?" without additional data is probably a fool's errand. (and leads to discussions like this I suspect hah) > your whole startup might die overnight when the drive on the Hetzner instance dies. > [...] > I would never purchase a product in 2025 that was storing my production data on a single machine I agree with those statements also, but in many situations I would still use a Hetzner vm/bare metal box. I have to push back against what I feel is a strawman. Because the data isn't only on one machine, you would have backups. In Hetzner cloud this is a one click for daily backups. And if you want backups more often, you can purchase their storage box and setup a cron to perform a db backup on a schedule. In my opinion, with the small/medium businesses I work with the cloud is often extremely oversold and there needs to be more awareness brought to the fact that you can go very far without complexity. However, I can also relate to the fact that there is a generation of engineers now who are more comfortable clicks-opsing through a bunch of AWS marketplace and CloudFormation deployments than they are doing relatively simple Linux sysops. So that's also something to keep in mind.

Streaming sqlite backups are trivial with litestream. The irony is I've seen more early stage startups die from runaway AWS bills than I have from hard disk failure. 😅

Anecdotes about AWS bills aside (we all have them 🙂 ), what are the consistency models of these "streaming sqlite backups"? How do you know the backups are semantically correct w.r.t what Datomic needs in order to restore a database? I suspect it isn't and would thus be unsound. We certainly don't test against them. This is the same reason we don't support using DDB PITR, it is not and cannot be sound due to log tearing. RE: "You have the data in a backup". This is an apples to oranges comparison. Backups here are not equivalent to storing the transaction log on redundant storage on every transaction. If your transaction log is only written to one machine on each transaction, it is at risk, not a strawman. Again, the important thing here is communicating risk and tradeoffs. That hetzner instance running sqlite is probably gonna be fine for most folks, but that should be their decision (and risk to take on), not one made for them. I also agree with the point about cloud services being oversold in general but that is making a different argument than the one about safety of cloud storages.

Is there any concern with that default option? just in case also, there is an option for you to use that as unreserved concurrency if that is what you are worried about > :concurrency-limit sets https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html. Set it :none if you want the AWS lambda to use unreserved concurrency

kind of, I discovered some throttling due to that limit, I was doing a little bit more than 20 concurrent calls

yup, I discovered that yesterday, that I can configure the limit, just asking to know if there is any suggestion or thing I should consider when moving that limit

I think that default parameters, in general, tend to be conservative, so we can tune them appropriately to the workload. this case is an extra control you have, the next boundary would be the aws quota per account

kk, also I think I would need to monitor this also just to see what is the right number according to my context, looking for memory usage, performance, etc thanks for the advice Filipe 🙂

Is it possible to get version 0.9.229 of dev-local?

You're looking for https://docs.datomic.com/datomic-local.html

I know, would be nice to still be able to pull in the older artifacts though