hey @U0BKWMG5B, long time no speak 👋 how're things? @U050CTFRT is working on XT authorization at the moment within the https://github.com/juxt/site project, he'll likely have some insights. he's going for your 'wrap the query and translation syntax' approach though, fwiw.

as to your question about wrapping the storage layer: • the main query engine uses the content/temporal indices on the nodes - you could try to wrap the index-store with your authz logic, although this is quite low-level - I'm guessing you'd need to convert the serialised values back into Clojure values to check authz. happy to help out if you do decide to go this route though 🙂 a starting point would be xtdb.kv.index-store/KvIndexSnapshot, and there'd likely be some further wiring up required. • pull and friends then use the document store - once the main query engine has an entity in hand, it just hands the content hash over to pull for further fetches

Hey there! Things are good, how are things with you? I hadn't heard of the Site project before, and it looks similar-ish to what I've been thinking about.

Hi @U0BKWMG5B, good to hear from you. Yes, as James mentions, the ACL work I've been sunk into all this year is on the 'actions' branch. Basically, the Datalog is compiled from GraphQL (via EQL) with rules which depend on the action being performed, either read or write.

One of the difficulties with protecting certain data is including the context, such as the performing subject and purpose of the data access. So Datalog is a good tool to encode such rules and access policies, close to the data itself. I'd be happy to share more or feel free to suggest a good time for a call/catch-up

That's interesting, and I think a good direction to go in. There's a lot of replication around web apps. I was considering an approach that exposes more, so rather than GraphQL, instead to expose sandboxed transactions and queries with some max run time limit.

There are many ways of achieving ACL through Datalog, so it depends on your overall system design.

Yes, I agree. I've been using SCI to sandbox code within transaction functions, it has some nice features for sandboxing, allow/deny var lists and so on.

Yeah. My original thought was to rewrite the datalog query to add on additional conditions; essentially simulating a SQL view, albeit a dynamic one. Though the problem with that is that pull syntax complicates things a little - I'd have to be very careful not to let anything slip through.

Yes that's right. You have to protect each join really.

The approach I'm thinking about now is to have a fast predicate function to check for authentication; given a user, entity ID and attribute, return true or false.

Exactly. The signature of my own predicate function happens to be just that. (allowed? subject resource permission) returning a truthy permission

I have a design where permissions are ephemeral documents that act rather like switches. But that's just a nuance.

That's fed through a GraphQL front end? Does GraphQL need to be set up manually, or does it automatically configure itself from the database?

I wrote a GraphQL compiler lib called grab which compiles GraphQL to an AST. I convert that to EQL, which embeds the rules (actions in my design) as EQL params. Then I compile the EQL AST into Datalog.

The Datalog contains the entire GraphQL query. I'll also target XTDB Core2 SQL in due course.

Note, this is all on a separate Site branch called 'actions'

That's interesting, and very similar to what I was thinking about. I'll definitely take a closer look around the project.

I'll send some links to the relevant github code sections when I get home 🙂

Thanks!

I took some time to clean up this namespace, so it's in a better state now. The first 600-odd lines are for preparing the database for various test scenarios, but it's worth a skim read. The test proceeds through EQL-defined queries and onto compiling GraphQL -> EQL at the end. https://github.com/juxt/site/blob/actions/test/juxt/site/eql_datalog_compiler_test.clj#L617

@U0BKWMG5B ^ I hope this is useful if only for seeing how Datalog can be used.

I should mention a few design constraints here. I don't want to build a 'computer says no' access control system here, with simple 403s denying access. What I want is for the queries to return just the data the user has access to, with the rest nil'd out. So, for example, in a user interface, one user might see some extra icons or data in tables depending on their access rights.

Also, I don't want to define access in terms of 'just' subjects and rows, as per row-based security models. I'm trying to include the 'context' in which a subject is access a set of records. For example, the business process of extracting email addresses from a database for the purposes of spam emailing might have a different access policy (perhaps requiring user consent) from another business process (say, extracting the email for the purposes of initiating a lost password flow).

I'm also attempting to build access control as close to the data as possible. Since external 'authz' services require policies to be written in terms of up-to-date data. It's not easy a) to know in advance which subset of the database the authz service might need and b) to keep that data fresh. In the case of ACLs, stale data can lead to problems around consistency which in some domains might be highly problematic.

These design constraints have led me to this action/permission model. But I'm sure there are plenty of other designs that would work.

Also, note, I'm not too concerned about performance right now. The sorts of systems I'm trying to build don't have aggressive performance requirements (but do have strict consistent access-control requirements). As you pointed out yesterday, there's a lot of duplicated work in back-ends around access-control, it tends to be done ad-hoc and thus requires extensive testing. But I believe it's also very hard to solve this at the library level.

Thanks for the example. I also agree that performance shouldn't be the first concern, and that this is a hard problem to solve. Did you mean it's hard to solve as a library, but easier as a framework/application?

I think I shouldn't have added 'at the library level', but what I meant was is that it's hard to abstract this problem because the decisions depend on logic at the level of the domain itself, and it's tempting to just 'code it' concretely.

I agree. I think my approach might be to write a syntax layer over the query and transaction syntaxes, to allow arbitrary restrictions to be added to queries and transactions, and extra data to be added to transactions as well (i.e. like the current user).

I'm guessing you've considered the style of datomic's filter-db? I used that previously to implement an attribute based access control "policy decision point (pdp)" - I found filter-db easy to reason about and simple to implement. the pdp was a function which took context (like the user, headers, etc) and the applied a filter-fn to the db, and then the user's (read) query was executed on the filtered db. writes were two step process whereby the TX was executed in a forked db (ala https://github.com/vvvvalvalval/datomock) on an unfiltered db, and then compared to executing the same TX on a filtered db - a difference between the two resulting in a policy denial....

filter-db was something I was worried about losing when I migrated a project from datomic to xtdb, but we reasoned that until we had enterprise clients who needed real ABAC like that, it wouldn't matter! and thanks to xt being opensource, we could probably add it if needed...

I was aware if filtered db. But your approach sounds very smart, particularly the 2 phase way of restricting writes. I feel ultimately a filtered db is really only a view, and views are queries, so it's all the same idea underneath: restrictions on queries by adding additional where clauses.

interesting. so one thing that I liked about filter-db was that it was a Clojure function and you had the full power of being able to query the entire (filtered or unfiltered) db to determine whether the datom satisfied your filter. I don't know enough to know whether they're actually equivalent? but at face value (or to a non-expert) it felt like filter-db was a more expressive approach than somehow pre-computing an appropriate where clause in datalog? or are they truly equivalent?

Note that can actually provide any Clojure function to XTDB, called a predicate function, but it's not quite the same. Of course, Datalog runs against the whole db too, but requires you write your logic as clauses.

I like filter-db, it's very powerful, but of course you do have to be careful about the performance of your filtering predicate.

One of the challenges in building access control is surfacing up the logic into the form of policies that business folks can understand and make changes to.

Datalog clauses map reasonably well to additive rules that can map onto business entities if you squint. I would argue this is at least better than encoding access control as Clojure code.

I didn't know about filter-db. It looks like it filters on the client, so performance is likely an issue. On the other hand, restricting for security is often distinct from restricting for performance.

Thanks for the tip, xlfe

All good - I was using Datomic OnPrem (although, not OnPrem - deployed to Google Kubernetes Engine) so the filter-db was happening on the server and performance was pretty good for a "real" ABAC system. Interested in seeing what you guys end up doing in this space though