Fork me on GitHub
#datomic
<
2017-09-08
>
kommen09:09:57

I’m looking for some feedback on our datomic schema. anybody can give us some guidance? schema & questions here: https://gist.github.com/kommen/a902e4c5bfef1395e69a617d1cb427ac

hmaurer10:09:33

@kommen I would personally favour the first approach and use :db/isComponent on the profile refs. Disclaimer: I am new to Datomic, so take my word with a grain of salt.

hmaurer10:09:09

Lately I have been using the “document” analogy to help me decide if i should use :db/isComponent. I ask myself: “if I were to represent this entity as a document (e.g. in a document-oriented database), would I put this sub-entity in the document as well?”

hmaurer10:09:56

This is basically asking “is this other entity (profile) a part of / a component of my entity (person)?” (quite naturally)

hmaurer10:09:17

or “does a Profile have an existence of its own, or does it only exist as a part of some other entity?”

hmaurer10:09:49

I am not sure these are the semantics that were intended for :db/isComponent, but that’s how I have been using it so far

hmaurer10:09:52

@favila made a remark to me the other day which is related to what you are asking now

hmaurer10:09:52

I was asking if it made sense to add a :db/unique constraint on attributes which have :db/isComponent. It seemed to me that these should always be unique, and so that :db/isComponent entailed the :db/unique constraint

hmaurer10:09:32

he pointed out that it entailed an even stronger constraint: an entity that is referenced by some attribute with :db/isComponent should not be referenced by any other attribute in the database

hmaurer10:09:50

(if I recall correctly, @favila can correct me if I don’t :<)

kommen10:09:08

alright, thanks for your input!

hmaurer10:09:26

Np 🙂 By the way on :profile/owner, I tend to avoid polymorphic references like this because they’re harder to validate

hmaurer10:09:37

that’s purely a matter of taste though

favila13:09:07

@hmaurer @kommen that is correct, entities reachable by an isComponent attr should not be reachable (by forward references) any other way

favila13:09:57

(d/datoms db :vaet component-entity-id) should only have 1 or 0 results for all times

hmaurer13:09:15

@favila out of curiosity, what is your opinion on polymorphic refs? (however you define “polymorphic”, since there isn’t a clear notion of “entity type” in Datomic). Is it something you try to avoid? Or do you not pay particular attention at it?

favila13:09:57

type generally depends on use

favila13:09:53

there's a distinction between entity-level validity/expectation constraints for the entities in the db, and an explicit type marker is necessary for sanity here

favila13:09:54

and what a program does with the map view of this, which seems to be more interested in attrs and sets of attrs than types

favila13:09:00

at least in my experience

favila13:09:51

but never, ever make the meaning of an attr dependent on some other attribute

hmaurer13:09:36

@favila that’s a similar philosophy to clojure.spec I assume? The meaning of attributes should be independent of the aggregates in which they’re used

hmaurer13:09:04

or did you mean something else?

augustl13:09:29

"foo" meaning different things in different contexts I suppose?

favila13:09:34

Essentially. I just want to make sure no one misinterprets me to think that {:entity-type "earthquake" :magnitude 7} {:entity-type "decimal-number" :magnitute 2} is ok

hmaurer14:09:26

makes sense

hmaurer14:09:12

out of curiosity, have you written a small lib internally for pre-processing data being transacted to datomic? (validation, etc)

hmaurer14:09:21

or are you doing it in a ad-hoc manner?

favila15:09:09

ad hoc. We should do better

favila15:09:49

we have an entity-type attr on all entities which anchors our expectations as to what other attrs should/may be on the attr

favila15:09:05

but nearly every attr is namespaced by its entity type

favila15:09:35

so in practice, code doesn't look at the explicit entity type much

favila15:09:43

just the attrs

mgrbyte19:09:30

Will the ensure-cf datomic command work without creating new AWS resources (security groups, roles etc) if existing ones are specified in the input properties files?

cjhowe20:09:15

how do i use datomic's tempids? what are those -92233... numbers in {-9223301668109598138 17592186045422, -9223301668109598137 17592186045423} from http://docs.datomic.com/getting-started/transact-data.html ?

hmaurer20:09:28

@cjhowe those are the IDs assigned to the 3 entities that were created by this transaction

cjhowe20:09:44

yeah, but how do i know which is which?

hmaurer20:09:53

you could add a :db/id attribute to each entity in that transction as a string

hmaurer20:09:11

e.g.

{:db/id "movie-1"
 :movie/title "The Goonies"
 :movie/genre "action/adventure"
 :movie/release-year 1985}

hmaurer20:09:50

the returned tempids should then be something like

{"movie-1" 17592186045422, -9223301668109598137 17592186045423}

favila20:09:16

@cjhowe use the d/tempid function

favila20:09:38

sorry, d/resolve-tempid

hmaurer20:09:54

@favila taking this opportunity to ask: is there any advantage to using d/tempid and d/resolve-tempid over string tempids?

favila20:09:20

not really

favila20:09:35

you don't need an auto-increment string function?

favila20:09:50

(str (name (gensym)))

hmaurer21:09:58

Also I assume string tempids resolve to an ID living in the :db.part/user partition?

favila21:09:16

you can control partition with tempids, not with strings

hmaurer21:09:24

I see; thanks!

favila21:09:04

(let [tid1 (d/tempid :db.part/user)
      tid2 "string-tempid"
      {:keys [tempids db-after]} @(d/transact conn [{:db/id tid1 :db/doc "tid1"}
                                           {:db/id tid2 :db/doc "tid2"}])
      tid1-eid (d/resolve-tempid db-after tempids tid1)
      tid2-eid (d/resolve-tempid db-after tempids tid2)])
@cjhowe

hmaurer21:09:08

Are partitions something important to think about when developing a small to medium sized app with Datomic? As far as I understand they are “locality hints” and if used properly can increase performance; is that right?

favila21:09:31

they reduce index fragmentation and increase locality

favila21:09:44

they are not mere hints, they control sorting order

hmaurer21:09:10

@favila how so? (regarding sorting order)

favila21:09:53

an entity id is a long composed of partition id bits and an auto-increment-id bits

favila21:09:26

since the partition bits of the entity id are higher, they have a stronger effect on magnitude than the lower bits

favila21:09:42

so entities with the same partition will group together when you sort

favila21:09:42

(d/entid-at db :db.part/db 1000)
=> 1000
(d/entid-at db :db.part/tx 1000)
=> 13194139534312
(d/entid-at db :db.part/user 1000)
=> 17592186045416

hmaurer21:09:17

When would the grouping property when sorting by entity ID be desirable? (sorry if the question is silly)

favila21:09:09

all datoms are stored in sorted blocks

favila21:09:02

if all items are "close" to one another (same partition), reduces the likelyhood you need to fetch additional blocks or fragment blocks while indexing

favila21:09:33

e.g. suppose you have two companies using the same db for their data (not that I would recommend this)

favila21:09:57

read and write patterns are such that txes and queries only deal with one partition at a time if you have companies in separate partitions

hmaurer21:09:14

@favila so, for example, you might want to store entities and their components in the same partitions?

hmaurer21:09:47

or have a partition per entity type?

hmaurer21:09:00

depending on what is often fetched together

favila21:09:23

actually components automatically get an entid in the same partition as their parent

favila21:09:55

{:db/id (d/tempid :some-part) :some-component-attr {:foo "bar"}} (for eg) was legal even before implicit tempids

favila21:09:21

the inner entity would be given a :some-part partition id automatically