Fork me on GitHub
#datomic
<
2017-07-27
>
hmaurer14:07:26

Datomic’s default behaviour in a transaction is to “upsert” (e.g. if an ID specified as :db/id does not represent an existing entity then it’s considered a temporary ID). This seems prone to errors; is there a way to throw an error on transact if :db/id is not referring to an existing entity? and vice-versa?

hmaurer14:07:52

Ah nevermind, I am wrong on this. It’s not that :db/id gets considered as a temporary ID if it does not represent an existing entity ID. It’s that it uses the value of :db/id as the new entity’s ID instead of auto-generating it.

hmaurer14:07:42

My question still stands though: can I disallow upserts

hmaurer14:07:54

(without doing a read before hand, or using a custom transaction function)

hmaurer14:07:01

Basically, I want to make sure that when I am updating an entity I am actually updating an existing entity, and not creating a new one

hmaurer15:07:32

@favila would you have an insight on this?

favila16:07:30

entity ids are just numbers, so there is no solid notion of one "existing" or not

favila16:07:58

the closest you can get is to say there are no datoms which contain that number as a reference

favila16:07:35

(precisely: no datoms in :eavt index with that number in :e, and no datoms in :vaet index with that number in :v)

favila16:07:45

you can arbitrarily assert datoms against any entity id you want whose t value is <= the internal t counter in the db (and whose partition id is valid? not sure if this is checked)

favila16:07:30

What I suggest you do is define some notion of entity "existence" for your application (e.g., has a certain attribute asserted) and use a transaction function to assert that the attribute either exists or has a certain value.

favila16:07:18

:db.fn/cas with same old and new value might work? not sure

favila16:07:24

otherwise you need custom fn

favila16:07:41

transaction fn should throw if the precondition it is testing fails

favila16:07:10

another idea, you can use a lookup ref as the :db/id

favila16:07:30

if the lookup ref and the notion of an entity existing are the same

favila16:07:01

{:db/id [:unique-attr "unique-value"] :more "assertions"} will fail if the lookup ref cannot resolve

hmaurer16:07:24

@favila thanks! That’s helpful. I was wondering the same thing about :db.fn/cas working with same old and new value; I’ve to try it

hmaurer16:07:46

would db.fn/cas work with old value nil ? To check whether an attribute has NOT been set?

hmaurer16:07:54

or would I need a custom transaction function for this too?

hmaurer16:07:04

@favila are you doing “existence” checks like this internally? It seems like something that would be quite common, e.g. some users might have the right to update an entity but not create one, etc

favila16:07:04

:db.fn/cas works with nil with meaning you said.

favila16:07:10

(says so on docs)

favila16:07:25

it doesn't support new value of nil though, which I have wanted

favila16:07:32

(i.e. a conditional retraction)

favila16:07:57

We don't typically do checks like this

favila16:07:15

If they happen to write to a db id that was deleted, it just gets orphaned

favila16:07:45

if they create a new thing, it's with a unique attribute and an entid

favila16:07:03

doesn't mean we couldn't be more careful about it though

hmaurer16:07:04

ok. And slightly unrelated question: do you use :db/id in your product directly? or do you tag every entity with a uuid?

favila17:07:59

:db/id if used, is always short-lived

favila17:07:07

we don't use it as external reference

favila17:07:19

we use it in pull, compare, update scenarios

favila17:07:50

It's really only necessary for isComponent entities

hmaurer17:07:46

@favila what do you mean by “short lived”?

favila17:07:33

on the order of minutes. db id is not persisted anywhere, it's just in a client's memory

favila17:07:50

usually a browser

hmaurer17:07:56

@favila oh I see, in your application.

devth15:07:31

can a map-form transaction contain a reverse lookup that associates multiple other entities with "this" entity? can't find any docs on this.

devth15:07:38

it appears i can associate a single entity with a reverse ref:

{:db/id "new-entity"
 :book/_subject 12312312312
 :person/name "foo"}

devth15:07:12

but not multiple:

{:db/id "new-entity"
 :book/_subject #{12312312312 456456456456}
 :person/name "bar"}

favila15:07:20

@devth that is correct. Anything seq-like will be interpreted as a lookup ref

devth15:07:53

ah. so it's impossible to assoc multiple refs

favila15:07:03

it's possible with additional maps

favila15:07:34

{:db/id foo :book/_subject 123}{:db/id foo :book/_subject 456}

favila15:07:44

(in same tx)

favila15:07:48

but not in a single map

devth15:07:51

ah, right

devth15:07:53

makes sense. thanks

favila16:07:34

I don't know if this is documented. I had to reverse engineer some map format edge cases

favila16:07:16

forward refs do sometimes accept many, but I don't remember how it decided between one lookup ref vs many items

devth16:07:17

yeah, finding the docs are a little light in this area

timgilbert17:07:27

Is there a better / preferable way to get a list of datomic entity-API items out of a query besides this?

(map #(d/entity db %) (d/q '[:find [?eid ...]] db))

hmaurer18:07:28

Is there a way to prevent peers from execution d/delete-database? (unless they are “priviledged” or something)

robert-stuttaford18:07:45

stop using the peer lib, and use the client lib instead

hmaurer18:07:34

@robert-stuttaford client lib is in alpha and lacks support for some features though, no?

robert-stuttaford18:07:10

i guess. i don’t use the client lib 🙂 but that’s basically what it boils down to. the peer is considered to be inside the database

hmaurer18:07:29

mmh. Would you use the client lib if you started a new project?

hmaurer18:07:34

the peer lib seems to have some nice perks

robert-stuttaford18:07:57

probably not. now that there’s no peer limit, i prefer its programming model

robert-stuttaford18:07:44

we’re all in with Datomic; its our only database, and we’re full-stack Clojure. so the peer is everywhere

hmaurer18:07:07

the idea of being able to delete your production database with a single line is scary

hmaurer18:07:28

although as you said in a thread, wth continuous/regular backups it’s not so bad

hmaurer18:07:55

how do you go about doing continuous backups by the way @robert-stuttaford ?

robert-stuttaford18:07:31

we have a t2.small that has one job; this script

hmaurer18:07:03

oh, so it just runs it on repeat

hmaurer18:07:23

I thought you were doing something clever observing the transaction log with Onyx and piping it to a backup location

hmaurer18:07:33

That works too though

robert-stuttaford19:07:45

we are looking at DDB streams for cross-region replication so that in Disaster Recovery we can be back up quicker

hmaurer19:07:54

Are there some guarantees that Datomic’s backup system won’t fuck up the incremental backup?

hmaurer19:07:01

or do you also continuously backup that to another location?

robert-stuttaford19:07:09

because when we dry run it right now, the longest part of downtime is copying the backup to another region

robert-stuttaford19:07:31

we have regularly scheduled backups to non-AWS yes

hmaurer19:07:01

@robert-stuttaford out of curiosity, do you test that your backups are actually working? this is a non-datomic question, but I was planning to do this on a project

hmaurer19:07:12

e.g. periodically restore a DB from backups and run some tests on it

hmaurer19:07:00

@marshall Hi! Maybe you could answer this question: what exactly happens when you call d/destroy-database? Does it send a message to the transactor? Does it destroy the storage directly? Also, would it be possible to disallow d/destroy-database calls on, say, a production database? (by configuring the transactor or the storage in a certain way)

marshall19:07:22

@hmaurer delete-database tells the transactor to remove the database from the catalog. it doesn’t destroy any of the storage directly (that happens when you later call gc-deleted-databases) There is currently no way to disable it or launch a peer that can’t call it

hmaurer19:07:02

at which point will it delete storage if you don’t call gc-deleted-database? and if it doesn’t delete storage, is there a way to “restore” a deleted database?

marshall19:07:24

not really; there is probably some way to recover it manually, but it wouldn’t be pretty

marshall19:07:25

basically, you probably shouldnt have any code paths in your system that include a call to delete-database Think of it a bit like having a DROP TABLE somewhere in your code

marshall19:07:38

useful as an administrative tool, not so great in your app 🙂

hmaurer19:07:06

Yep, of course. But on PG/MySQL I could configure the production db user to not be able to drop tables at all

hmaurer19:07:26

Which is reassuring, even if code reviews/linting tools can ensure that your production code does not call those methods

marshall19:07:30

true enough. I would suggest that is a reasonable request to put into our Suggest Features portal 🙂

hmaurer19:07:10

Will do. Thanks for your insights!

hmaurer19:07:48

@marshall another small thing I discussed earlier in a thread: is the “transact” function in the Datomic peer clojure library part of a protocol that I could implement?

hmaurer19:07:58

e.g. to wrap some behaviour around d/transact

hmaurer19:07:12

I couldn’t figure it out from the doc

marshall19:07:26

Not sure i understand the question. If you need to enforce constraints on the transacted data you can either use a transaction function, or build up the transaction data structure in your peer and do a compare-and-swap against some known value

hmaurer20:07:12

My bad, my question wasn’t very clear. I would like to add some attributes on every transaction in my application for auditing purposes (e.g. the ID of the current user). To this end, I could wrap d/transact with my own function, so as to add the necessary tx-data to every transaction. I was wondering if the d/transact function in the Clojure API is part of a protocol that I could reify to add my own behaviour. I am quite new to clojure so this might not make sense at all

hmaurer20:07:52

e.g. is datomic.Connection a protocol?

marshall20:07:58

gotcha - I dont believe it is extensible in that manner. I’d probably suggest you write a wrapper fn that adds the user info you’re wanting to include and use it exclusively throughout your application

timgilbert14:07:32

^ that's the approach we've taken at my shop, it works great

hmaurer14:07:29

@U08QZ7Y5S do you pass the “current user” context manually all the way down to this function? Or implicitly through something like a dynamic var?

timgilbert14:07:30

Yep. We keep a kind of metadata/session object around which gets initialized from a JWT token at the top of the stack and contains the user-id and roles and some related stuff, then we pass it all the way down to the datomic layer, and then our wrapper function adds a {:db/id (d/tempid :db.part/tx) :user/id blah :meta/data foo} to the transaction data that the calling code passed to it.

timgilbert14:07:45

We did experiment with some ways to avoid needing to explicitly pass the metadata around, but nothing seemed to be significantly better and most of our experiments introduced subtle context semantics that we didn't want

hmaurer17:07:12

@U08QZ7Y5S thanks for the explanation 🙂 out of curiosity did you wrap other datomic functions (e.g. d/entity) to enforce some security rules based on roles too?

timgilbert17:07:44

Nope, and having three separate datomic APIs (pull / query / entity) has been a source of some architectural friction for us that we haven't quite solved. What we tend to do these days is return entities from our data layer and then do filtering at the higher levels, but that spreads the filtering logic around our codebase a bit more than we like

timgilbert18:07:59

It's nothing unsolvable, but returning entities rather than data from the data-access layer has some implications for the complexity of the rest of the code

timgilbert18:07:27

On the other hand, it keeps the data layer simple and is very flexible in terms of what we can do at the controller level

robert-stuttaford20:07:00

@hmaurer we restore production to our staging environment daily

robert-stuttaford20:07:22

part of our business has a content creation component to it, so we’re constantly testing new content with new code

stijn21:07:31

i'm trying to generate a datalog clause that looks like this [(.before ^Date ?event-start ^Date ?start-before)] (with type hints)

stijn21:07:51

?event-start and ?start-before are generated symbols though

stijn21:07:08

when I don't quote the ^Date they get removed (which seems logical to me): [(list '.before ^Date (calculate-symbol param) ^Date (calculate-symbol other-param))] ==> [(.before ?event-start ?start-before)]

stijn21:07:48

but when quoting them: [(list '.before '^Date (calculate-symbol param) '^Date (calculate-symbol other-param))] ==> [(.before (calculate-symbol param) (calculate-symbol other-param))] they also disappear and the calculate-symbol calls get quoted

stijn21:07:41

so, how to do type hints with calculated symbols?

favila21:07:53

@stijn < should work with date

favila21:07:35

the type hint issue is because of quoting

favila21:07:51

the metadata is put on the LIST calculate-symbol, not the result

stijn22:07:36

and < on dates doesn't give any reflection warnings

favila22:07:54

you need something like (with-meta (calculate-symbol param))

favila22:07:20

the query comparison operators are magic in datalog

favila22:07:49

they follow datomic's comparator rules for the types that are indexed

favila22:07:04

and datalog query optimizer can often understand them to avoid full scans

stijn22:07:24

so it's even better to use < then .before?

favila22:07:29

much better

favila22:07:22

built-in means "not clojure.core"

favila22:07:13

note e.g. != is magic builtin, not= is clojure.core/not=

favila22:07:07

should prefer != < <= > >= on all types that are valid :db/valueType (except bytes)

souenzzo16:07:46

Double check (class your-date-in-clojure) I have trouble with java.util.Date(works) x java.sql.timestamp (dont work)