Fork me on GitHub
#datomic
<
2021-04-19
>
jamesmintram09:04:30

Hey all. I asked a little while back about doing multi-tenant apps with datomic and a couple of ideas came up. One of those was using a single DB per customer. So the connection string might look like : "datomic:" This looks interesting - do this mean that all databases backed by the same Postgres instance (for example) share the same DB? In which case, the only different between: "datomic:" and "datomic:" Is some sort of partioning key that Datomic uses to keep data separate within it's backing store?

jaret13:04:34

Hi, For on-prem we generally recommend that you run a single logical database per transactor (pair). Some customers use additional DBs for operational tasks, but generally a datomic on-prem system (and license) includes a transactor pair, single DB, associated peer applications. Perhaps we can discuss your needs for separate DBs. Have you evaluated https://docs.datomic.com/cloud/whatis/architecture.html? Cloud is more suitable for per db mutli tenancy. How many DBs do you envision long term? What are the sizes of each DB that you expect? If you'd like to share more details you can shoot a ticket to <mailto:[email protected]|[email protected]> and/or we can arrange a call to discuss.

jamesmintram09:04:06

If that is the case, then is it feasable to "open" a new DB connection per http request? or keep a pool of conns in some sort of LRU cache? From the docs

Datomic connections do not adhere to an acquire/use/release
pattern. They are thread-safe and long lived. Connections are
cached such that calling datomic.api/connect multiple times with
the same database value will return the same connection object.

prnc10:04:04

Hi 👋 I am looking into application level access restriction/permissions to entities. I couldn’t find any specific guidelines for cloud, that would allow for a separation of concerns between query logic and permissions. In on-prem filter seems to be the useful for that. What are some good ways to go about this for datomic cloud? [It depends ofc, it’s early stage so I don’t have specific requirements, just looking for some general guidelines/mechanisms in datomic that could be leveraged]. Thanks!

tatut12:04:20

afaict, there is no similar functionality in cloud

prnc12:04:53

Even if there is no filter fn as such I would be interested how you should go about restriction user access (say only to “their own” data) in e.g. queries? I’m just new and not sure what the right approach is 😉 Is it just “do it in every query explicitly” kind of thing?

thumbnail12:04:34

In our system we only check the authorization (i.e. access) at the borders of the system. When you're passed that interceptor/layer we don't bother. That way the queries are simpler and faster (at the cost of always running one (or more) queries beforehand.

Joe Lane13:04:03

Queries are data and compose, programmatically building queries is a normal thing to do :)

shields13:04:49

Good example at the 12 minute mark of how Nubank controls ownership and authorization w/ their schema using rules. https://youtu.be/7lm3K8zVOdY

prnc14:04:51

I’ve started drafting an impl of what I need with rules—will definitely check out the nubank talk as is seems to go that direction. Thanks everyone!

tatut04:04:17

we too have authorization rules at the boundary, all commands and queries from clients are authorized… I think it’s common that authorization is based on a few core entities that are part of the query args / command payload

4
tatut06:04:15

that nubank presentation with filter was interesting, it would be to see actual performance numbers… it would seem to me that the filter approach can’t use indexes efficiently, but don’t know any details

prnc15:04:11

Thanks @U11SJ6Q0K! Being new to datomic this is quite an interesting problem for me. The filter solution (on-prem only!) seems to be very clean, and it’s what nubank are using. Query logic is separated from access restrictions. Leveraging “database as a value” and thus playing to datomic strengths. The other nice thing about datomic is ofc “queries are data” as mentioned by @U0CJ19XAM I’ve just used this approach to write a boundary query-as-a-user fn, which rewrites the queries to refer to access restriction rule. Somehow it doesn’t really feel right, less separate, less explicit. Maybe it’s just my shoddy impl 😜 We will see how it fares in practice! If anyone has examples of those kinds of things in real world (w/ users, permissions, auth etc) open source code I would love to see them! Cheers!

Joe Lane15:04:23

How complex is your permission model?

prnc15:04:35

ATM I’m only concerned with the ~trivial case of: “Alice owns some resources and only Alice can see them” where resources are entity level, so not very complicated and not very granular, but that will probably evolve. It’s a knowledge base type of application so where this will potentially get more interesting is in the cross-sections of public and private information--private knowledge graphs embedded in public ones. So I’m just trying to model this in a nice way with datomic without prior experience with datomic ;) Presently I was mainly concerned with the simple mechanics i.e. where those restrictions should sit (in terms of best practices), so it’s not error prone. So today I’ve just added a rule and a centralised query fn for restricted resources that just adds those access constraints captured by the rule to the :where clause... so far so good I think ;)

cjsauer18:04:27

Has anyone experimented with defining “virtual attributes” in the db schema? Meaning, attributes that are not meant to be involved in the transaction of data, but that only serve as documentation, or to make the system even more self-describing. For example, I might include :product/href in my schema as a card-one string type attribute, but only ever derive its value on-demand. I’m finding that describing the system fully in the schema gives you really great “meta-query-ability”, but I’m wondering if there is any downside to doing this.

cjsauer18:04:41

I’ve seen quite a few other projects that do this out-of-band in code. But it struck me that this type of metadata really belongs in the schema proper. It could even allow datomic to help you with making backwards compatible changes by warning you if you break schema (?)

Joe Lane18:04:11

There are limits to how many schema elements you can have, but that https://docs.datomic.com/on-prem/schema/schema-limits.html

Joe Lane18:04:37

I've seen migration tools built using this capability, diagramming tools, etc.

cjsauer18:04:11

Aha, cool! For a moment it felt odd to be capturing schema about data that I don’t intend to ever store on disk, but after pondering a bit, why should the durable storage have a monopoly on the schema definition??

Joe Lane18:04:02

... I mean, it is durable. These attributes are the same data as would be in transactions and are stored in durable storage.

Joe Lane18:04:12

Datomic is built out of itself

cjsauer18:04:02

True true. I mean to say that I never intend to actually transact a value of that attribute onto domain data. Of course the schema itself is stored durably.

cjsauer18:04:04

Another way to put this: essential state shouldn’t have a monopoly on the schema. So it’s not really about its durability. More its use in the system.

Joe Lane18:04:58

Makes a nice compile target too 😉

cjsauer18:04:46

Yea definitely

ghadi21:04:00

txacting data about attributes such a useful thing to do ^^^^^^

💯 4