Clojurians Log v2
sci 2026-07-22
A coupled of years ago I wrote https://github.com/SimpleValue/peer-sci which was my attempt to keep using Datomic's Entity API, while other factors forced us to migrate from Datomic's peer API to the Datomic's peer-server (which don't have the entity API). The idea of peer-sci is that the Clojure/SCI code travels to the database, where it can do things that requires super low latency (like the Datomic entity API, no n+1 problem). A bit like Datomic Cloud Ions but without forcing you adapting all its deployment requirements and the need to run on top of AWS. We also ran peer-sci successfully in production, but I always considered it a workaround. Since two years I'm researching different ways to move our SaaS to a https://community.fly.io/t/shared-nothing-architecture-still-valid-on-fly/28336 where each customer (tenant) receives its own logical database and Docker container (or MicroVM). We had a version running in production that really provided each tenant with an own Docker container. We run on very cheap dedicated servers from Hetzner, but even there for cost reasons we needed to switch back to a multi-tenant JVM app, where each tenant at least still have its own logical Datomic database. But moving the tenant database from one server to another is painful (even when automated) but it is required for doing these low latency things mentioned above. Enough context, here my question: would it be a good idea to use SCI for this purpose and turn the workaround into a real solution for this difficult problem? My guess is that even when everything already resides locally on a very fast NVMe disk the IO costs would still dominate everything else, meaning the overhead of the Small-Clojure-Interpret vs. AOT compiled Clojure code can be ignored in this scenario?
Let me see if I get your question correctly, are you asking if it makes sense to use Sci in a multi tenant setup?
I think in a JVM scenario I can make SCI much faster using the same approach used for JIT on JS. Would be a couple of days work. Then you would pay almost no overhead. But still, SCI is already pretty fast in interpreted mode, so I think it's a matter of do the research and you will know
@jeroenvandijk the idea is to send SCI code to your server that also runs your database, much like you send SQL to a relational database. Instead of trying to have a local database in your app server (Sqlite, Datomic peer, Datahike, Datalevin, etc.), since then you need to solve all kind of cold-start problems, for edge cases where you database is not yet on the local disk of a new server (we have some customers which dbs are north of 2,5 GB which takes 30 or more seconds to download from an object store in the same region).
@borkdude that would be awesome 🤩 Yeah I should do some benchmarks for this use case. But my guess would be that SCI is already fast enough for this use case (even without any JIT). I just wanted to ask this question here to not overlook any blockers that would make this unreasonable due to other reasons.
@maxweber I see. Yeah like Michiel said, SCI is already pretty fast. And if you would come across cases where SCI is adding signifcant overhead compared to Clojure you could decide to compile (parts of) the query with Clojure's eval (you can check if it is safe to do so with SCI). Or another option is to add expensive/frequently used custom Clojure functions that you invoke from SCI (so less hot loops in SCI). Regarding the multitenant thing. SCI has had some recent additions that might be interesting. Such as the https://github.com/babashka/sci/#bounding-execution-with-interrupt-fn which allow you to limit execution time e.g. in case someone wrote bad code. This combined with being able to override existing class methods with safe implementations for instance. BTW maybe you already aware, but in the peer-sci code you set up the SCI context https://github.com/SimpleValue/peer-sci/blob/master/src/sv/peer_sci/core.clj#L40-L52. So things like this can also be optimized
If hot loops end up being a problem, you can hire me to fix that problem ;)
@jeroenvandijk thanks a lot, I wasn't aware of the interrupt-fn and the SCI context caching, both will help a ton for my use case
@borkdude I would be happy about this opportunity 🎉 Checked your site http://michielborkent.nl are you offering this as freelancer days, Github sponsors or something else?
@maxweber we could talk about both
my site is hopelessly outdated
I can invoice you as a freelancer for sure, Github sponsors is also open
@borkdude very good to know 👍