Clojurians Log v2

Performance: not good. I am actively planning work to transition the multi-tenancy piece of our app out of Datomic into a bespoke product like OpenFGA. All depends on the size of your database, your specific data model, the actual technique by which you establish ownership of an entity, and the number of users you've got, but I would not recommend it. Our model and implementation is very close to the one demonstrated in this video: https://www.youtube.com/watch?v=7lm3K8zVOdY. I'd really love a followup video some day where somebody from Nubank talks about how this system evolved over time, whether they kept it, how they had to adapt it as they scaled, etc. It definitely leads to simpler code and queries, which is a huge plus. Having one central, canonical place in your application where your authorization and multi-tenancy stuff is handled means that your application queries get to be as dumb as possible, and that is a major plus for code simplicity and peace of mind.

If you search "multitenancy" and authorization in this channel there are a bunch of helpful threads on the topic 🙂

Thanks! I do see some interesting prior discussions. Out of curiosity, what made you decide to transition away from Datomic rather than toward a different tenancy model within Datomic (handling tenancy in queries or using multiple DBs)? Can you say any more about the performance problems you are seeing? In the past I worked on a good-sized Datomic system where tenancy was all handled at the app layer, you had to be sure to always restrict your queries etc. by tenancy. We did have the tenancy information denormalized so it was repeated on essentially every entity so you usually didn't have to do extra joins. That performed well enough, but definitely added complexity. Now I'm in a different situation where tenancy isn't really a concern but I do need to enforce very-fine-grained access control, and this access control information is not denormalized so often you have to do several joins just to get the required authorization for a given entity. The simplicity of a filter-based system makes this appealing. But I do worry that if performance turns out to be insufficient, it doesn't seem like there are a lot of knobs you can turn.

I have actually made a couple of those transitions you mentioned already 😛 We started out using the filtered DB method, which was based around a set of Datomic rules that filters the entire DB by looking at the requestor's identity, and assessing every entity in the DB by following a chain of references from that entity back to a "permissions" entity, and determining ownership on that basis. Does this entity eventually connect back to a permission entity that says the requestor owns it? Great, here's your new database value with only the entities that satisfy that constraint. Query away, there's nothing in here you aren't supposed to see. That quickly became way too slow. Not only does this filtering take a long time to actually perform on even a modestly sized database (this amounts to performing a depth-first search on the whole DB for every entity within it), but your app has to hold that database value in memory, and every time the database is changed, you have to rebuild the filtered DB. Our next step was switching to running the rules as a part of the query itself instead of using the rules to produce a filtered database value. Insanely cool that this refactor was even possible to do in the first place. Our legacy filtered DB and the new authorization query use the exact same set of rules. This resulted in much faster queries, but in a lot of parts of the app eventually became unacceptably slow as well. You're running the rules on a smaller dataset (the result of any given query, not the entire database), but if your search results in thousands or tens of thousands of row expansions, then that means thousands or tens of thousands of individual depth-first permission checks. Acceptable in some situations, not a silver bullet for the whole app, especially for "hot path" queries, which either search through a large batch of entities, or are frequently executed, or both. Then we got some help refactoring THAT approach (what's up @potetm!) to this hybrid thing where we bail on Datalog entirely when we're performing a query where we already know the fastest path back to the permissions entity. This is basically where we are today. Works, but it's a little weird and feels brittle, and also takes us further away from the whole point of separating the authorization and business logic queries in the first place, which is a separation of concerns and consistency. It gets pretty exhausting after a while inventing your own tools, and there's limited literature out there or standard practices for anything Datomic-related, so you really feel like you're out on a limb after a while. Datomic is a really excellent database when it comes to modeling data and flexibly adapting to new business needs, but at this point a specialized authz solution like OpenFGA is looking like the best option for this specific need. Long-ish development history, plenty of specific, concrete examples and documentation for common use-cases and needs, and it's specifically oriented around performance and fast lookups. We'll pay a complexity price by splitting our data between two different data sources, but I think the long-term flexibility and performance gains are going to outweigh a little bit of increased complexity. When it comes to implementing multi-tenancy by means of separating out data into multiple DBs, it's pretty much universally discouraged. A database is a pretty expensive abstraction. One of our team member's first questions when he joined up was "how are you implementing multi-tenancy?", and when I told him we only had one DB, his only reply was "good" 😅

Thank you, this is very interesting and helpful

wassap @andyfry01! yeah my recommendation is to just do your authorization queries directly (pretty much what any regular db would have you do).