Fork me on GitHub
#datomic
<
2016-09-27
>
robert-stuttaford04:09:08

it'll be the full database, @misha, as the limit is about how big the database roots are in peer memory before it becomes unwieldy. that'll include history.

robert-stuttaford04:09:31

and you're right, each database uses its own ram, which means two 10bn databases uses twice the ram

robert-stuttaford04:09:44

it's not a hard limit of the database system

misha09:09:19

thanks, @robert-stuttaford and "sharding" user-generated data into many db-s is not particularly useful, right? (data for users A-M goes here, for N-Z goes there)

yonatanel12:09:41

Does anyone know how to catch and handle transact exception specific to when a :db.unique/value conflict occurs? I can look inside the exception but I don't know what data is part of the public api and can be trusted (as mentioned in http://docs.datomic.com/exceptions.html#clojure-iexceptioninfo and a little in http://docs.datomic.com/clojure/#datomic.api/transact) I need this for cqrs idempotency (handling at-least-once behavior of my events log), so I'm transacting the event ID as unique so duplicates just abort the transaction, so if there's a better way that doesn't rely on exceptions it could be helpful as well. This works but uses the undocumented ex-data value:

(try
  @(d/transact conn [{:db/id             (d/tempid :db.part/user)
                      :cqrs.command/uuid #uuid "a1ad6690-8498-11e6-8495-c4fae74533c7"}]))
  (catch java.util.concurrent.ExecutionException e
    (= (:db/error (ex-data (.getCause e))) :db.error/unique-conflict))

stuartsierra12:09:34

@yonatanel Although the contents of ex-data is not promised as a stable public API, I think it is not likely to change. One way to be safer is to write a unit test for the behavior you expect, and run it every time you upgrade Datomic. As an alternative, you could write your own transaction function to enforce the constraint, throwing its own ex-info with data you can catch and use in your application code.

yonatanel13:09:14

@stuartsierra Good idea, though I still don't want to touch transaction functions. Also I changed my code just now to be even dirtier, checking the exception message string because it's the only thing that contains my specific uuid attribute ident, as it's the only one I want to silently ignore. Would be nice to have a silent "conditional put" or more info outside of a message string.

stuartsierra13:09:03

"Conditional put" is a clear use case for transaction functions.

yonatanel13:09:16

It might have been simpler to just do it instead of the whole exception thing :)

annataberski13:09:58

I changed an ident name in an in memory database, which seems to change the name as expected, except when I try to query by the new name I get :db.error/not-an-entity. Any insight as to why this is happening? Maybe related to renaming an ident in an in memory db?

marshall13:09:54

@misha sharding by either time or domain (ie users) can be a good approach if necessary. However, it's generally recommended that each shard be served by its own transactor

jaret14:09:20

@annataberski Can you try the same ident change on local storage and not in-memory?

annataberski14:09:10

@jaret sweet. i’ll give that a try

mlimotte19:09:30

Does anyone have any suggestion, or better yet, examples of how to limit data access for incoming queries when they are coming from a internet facing client? For context, the relevant portion of the proposed architecture looks something like this: We have a client application (Web UI). It authenticates and connects to a backend service. Then it can hit a generic /query/ endpoint in the service. The generic end-point allows the user to pass in an arbitrary datalog query. The authentication means that not just anyone can execute a query, but once authenticated, that user should also be limited. For example, if this were an e-commerce site, and a customer logs in. They should be able to query for things in their shopping cart, but not another customers's info. I'm thinking some sort of Access Control Lists associated with roles that indicates what entity types can be retrieved and what constraints are required.

stuartsierra19:09:16

@mlimotte: This is difficult. Datomic datalog supports arbitrary code execution, in the form of predicate functions.

stuartsierra19:09:18

If you're facing the public Internet, I would recommend developing your own query format — perhaps a subset of Datomic datalog — which supports only the features the client will need.

stuartsierra19:09:41

Also consider alternative query formats, such as GraphQL or Om.next, which allow a lot of flexibility in the client but are still less open-ended than Datomic datalog.

mlimotte19:09:29

Have been considering Om.next. Don't fully understand the query format there-- kind of like datomic pull syntax I think. Even in the Om.next case, though, I think I would still need some additional ACL limits.

mlimotte19:09:26

fyi, the idea for this open /query interface comes from this Bobby Calderwood video on ES+CQRS: https://www.youtube.com/watch?v=qDNPQo9UmJA&amp;noredirect=1

kenny21:09:34

@mlimotte I have been thinking about that for a while now and I think it is an incredibly powerful idea (I also got the idea from the video you linked). As @stuartsierra said, arbitrary code execution is allowed so your first step would be to sanitize the query by having a whitelist of allowed functions. Next, run the query against the database and filter the results down based on the user's role (I recommend using Clojure's isa? and derive to create your use roles). I have defined my user's roles as a map with the key being the role (e.g. :role/admin) and the value being a map of various whitelists, blacklists, and authorization functions which are used to create the aforementioned query result filter.

misha22:09:21

@kenny you did it in code, not in datomic, right?

kenny22:09:40

Yes, in code