Fork me on GitHub
#datomic
<
2021-10-20
>
Tobias Sjögren04:10:14

Say I want to define the schema attribute “first_name” in a database - what would the needed datoms look like (in storage)?

Tobias Sjögren05:10:03

These are not actual Datoms in storage, right? That is what I’m looking for - to see the actual triples..

tatut06:10:53

the map format is a convenient way to express multiple facts about a single entity, so it's not exactly the datoms in storage

Tobias Sjögren06:10:56

What I tried to ask is how the datoms would look like in storage..

tatut06:10:33

for that particular case, it looks like the below... I have no idea how they are actually represented in storage

(d/q '[:find ?e ?a ?v :where [?e ?a ?v] :in $ ?e] (db) :user/given-name)
=> [[:user/given-name 40 23]
 [:user/given-name 10 :user/given-name]
 [:user/given-name 41 35]
 [:user/given-name 63 "User's given name"]]

Tobias Sjögren06:10:17

I wonder how accurate this is..

tatut06:10:17

afaict, entities are always identified by the 64bit number, but the number :db/id and keyword :db/ident is often used interchangeably when showing info

Tobias Sjögren07:10:06

That fits with the example database here: https://docs.datomic.com/cloud/time/filters.html I guess the actual stored entity id is the 64bit number and :db/id and :db/ident are just aliases..

favila12:10:42

You can see the raw datoms using d/datoms

favila12:10:55

It’s a tuple of [e a v tx op], where e, a, tx are entity ids (longs) op is a boolean (true=assert, false=retract) and V is an object whose type depends on the valuetype of A

favila12:10:19

note that :db/id is just syntax for the map form, it’s not an ident. There’s no attribute :db/id nor is there a datom corresponding to it. it’s just the E that everything in the map has in common

1
😯 1
Tobias Sjögren14:10:52

I actually don’t use Datomic (yet) - I’m trying to recreate the basic functions in FileMaker to understand everything before probably moving on to using Datomic.. So “a” is also a long and referencing the attribute somehow? If I could see a “entity id entity”, an “attribute entity” and a “transaction entity” as raw data Datoms it would be great.. Your last message I didn’t quite understand..

favila15:10:42

attributes are entities

favila15:10:49

“entity id entity” is nonsensical

favila15:10:20

the unit of truth in datomic is a datom, which I described. Everything you see that looks like a map and represents an “entity” is merely a projection into/out-of those datoms

favila15:10:20

entities don’t exist in the system as entities---they are merely “the datoms which share an E value”

Tobias Sjögren15:10:28

Right. What would an “attribute entity” look like as raw datom(s) ? The “:db/ident” is not really the raw data, right?

favila15:10:50

what do you mean by an “attribute entity”?

Tobias Sjögren15:10:36

A datom where you define an attribute.

Tobias Sjögren15:10:12

You said the “A” were entity ids (longs)..

favila15:10:45

An entity becomes a legal attribute by being asserted on the db entity’s :db.install/attribute attr, e.g. [:db.part/db :db.install/attribute <ATTR> <some-tx> true]

favila15:10:09

(In fact you used to have to do this explicitly in the transaction, it’s implicit now)

Tobias Sjögren15:10:59

Every “A” is an entity id ? So “:db/ident” is a representation of an underlying entity id? If I add the :person/name-first attribute e.g., this attribute will be represented by an entity id in a datom?

Tobias Sjögren15:10:03

(layman warning...)

favila15:10:05

every attribute is an entity. :db/ident is a way to “name” an entity so that a keyword where an entity id is expected will resolve to that entity. (this is called “entity lookup”, and there’s one more way to do it using a unique attribute and the syntax [:unique-attr unique-value])

favila15:10:52

:db/ident itself is an entity

Tobias Sjögren15:10:16

represented by some datom and an entity id?

favila15:10:26

If you really want to understand how datomic bootstraps itself, I recommend looking at the earliest transactions and seeing how they establish the foundation for the system. You can see them with this:

(d/create-database "datomic:")
 (-> (d/connect "datomic:")
     (d/db)
     (d/seek-datoms :eavt 0)
     (->> (group-by :tx)
          (sort)
          vals))
In particular, you’ll notice that entity 10 is the :db/ident attribute itself, and it names itself self-referentially with datom [10 10 :db/ident 13194139533312 true] . Entity 0 is the database, entity 13 is the :db.install/attribute attribute, and attribute installations have the pattern [0 13 <entity-id-of-attribute> TX true]

Tobias Sjögren15:10:34

Is there a place to read all those “foundation datoms” without having Datomic?

Tobias Sjögren15:10:20

But with the right Datomic command, you can see all of it?

favila15:10:32

It’s what I just pasted above

favila15:10:08

these are “normal” datoms, there’s no extra meta “DDL” layer here made of different stuff

favila15:10:21

the data are datoms and the schema are datoms

Tobias Sjögren15:10:49

but the entity ids are hard coded for these foundation stuff?

favila15:10:03

yes, but since they all have :db/idents, it doesn’t matter

favila15:10:14

everything refers to them via ident

favila15:10:34

that’s what gives them meaning to code that reads or manipulates these entities

favila15:10:56

To blow your mind a bit more, entity ids are composed of partition in the high bits and a T value in the low bits. The partitions are also entities, and there are three predefined ones (0=db, 3=tx 4=user). so entity ids have entity ids inside them too

favila15:10:33

db = 0, which is why these bootstrap entities are small numbers, because all the high bits are clear

favila15:10:58

but transaction ids are large numbers

favila15:10:18

d/entid-at lets you construct entity ids

Tobias Sjögren15:10:26

So, during installation, datoms are added to storage that represent the foundation for each subsequent added datom? In this way about everything consists of datoms?

favila15:10:15

they come with a newly-created database

Tobias Sjögren12:10:40

@U09R86PA4 When you say “V is an object whose type depends on the valuetype of A” I’m not really following what you mean by “V is an object” - in what sense is it an object?

favila12:10:10

The Java sense

favila12:10:14

It’s a Java reference type in the Datom type, not a primitive type

favila12:10:26

The E A Tx op fields are all known fixed primitive types

Tobias Sjögren12:10:57

I don’t quite get that.. If I put the string “Francis” as the V - it is not anything else than just a string?

favila12:10:33

The data type Datom must allow any legal type in the system in the V slot

favila12:10:04

So in the java implementation that field must be type Object

favila12:10:09

That’s all I mean. I’m not making any deep statements here about triple modeling

Tobias Sjögren13:11:39

@U09R86PA4 When you wrote [0 13 <entity-id-of-attribute> TX true], the datom order is A, E, V, TX, OP - right ?

favila13:11:40

Datom slots are E A V Tx Op

Tobias Sjögren13:11:03

That was what I excepted when I saw these “foundational datoms”:

([10 0 :db.part/db 13194139533312 true]
 [11 0 0 13194139533366 true]
 [11 0 3 13194139533312 true]
 [11 0 4 13194139533312 true]
 [12 0 20 13194139533366 true]
 [12 0 21 13194139533366 true]
 [12 0 22 13194139533366 true]
 [12 0 23 13194139533366 true]
 [12 0 24 13194139533366 true]
 [12 0 25 13194139533366 true])
But from what I can understand this is A E V Tx OP…

favila13:11:17

It looks like it might be. What code made this?

Tobias Sjögren13:11:58

I actually don’t know..

Tobias Sjögren13:11:17

Here’s the full batch..

Tobias Sjögren13:11:30

If that is not A E V Tx OP - I have to conclude that I don’t understand this at all…

favila13:11:31

These aren’t even datoms, it’s one long vector

favila13:11:14

Where did you get this that you don’t have the code that made it?

Tobias Sjögren13:11:03

From someone that, unlike me, have Datomic installed..

Tobias Sjögren13:11:39

The content of the vector are datoms in a specific slot order, right?

favila14:11:26

The txt file you provided has only one vector in it. I guess you could presume based on line breaks and order and other clues that this is a cycling pattern of slots, but these are not datoms. As you noticed, it doesn’t seem to be E A V T Op

Tobias Sjögren14:11:41

If so, I’m still looking to get the “foundational datoms” ...

favila14:11:25

so, install datomic, run the code above

favila14:11:57

Why reverse engineer datomic from slack conversations?

Tobias Sjögren14:11:04

I know I should install Datomic. I need help to do it though. The amount of time it would take me to do it alone would make me loose focus on what I’m trying to achieve..

favila14:11:29

what are you trying to achieve?

Tobias Sjögren14:11:40

Possibly just as an intermediate step: Represent the core principles and ideas behind Datomic in my current platform (FileMaker).

Tobias Sjögren14:11:12

Doing so will hopefully allow me to be sure if I should leave it (FileMaker) or not..

favila14:11:48

implementing an immutable graph database in a relational database seems like a much, much bigger ask than installing and evaluating datomic (or any other dbs you may be considering) on its own merits.

Tobias Sjögren14:11:34

For sure. It’s just - I want to do it.. I want to see how far I can reach. So far it looks promising. It will be far from as performant as Datomic, but possibly still usable..

favila14:11:10

ok, then maybe getting the “foundational datoms” of datomic is a distraction. Think about the minimum you would need to self-describe the system before you could implement the code that emits datoms from transaction data. You only need to know enough to perform ident lookups, and it needs some known idents that describe transaction and schema-related operations

Tobias Sjögren14:11:25

I would agree to that they are partly just a distraction, but it makes me think about that self-describe thing.. I’m currently trying to construct an interface for constructing the schema attributes..

Tobias Sjögren08:10:38

By looking at storage data, how do I know when an entity id (in the E position of the datom) represents a transaction id? Is it when the datom attribute is “:db/txInstant” ?

Linus Ericsson09:10:42

example - transaction: [[1 :cat/name "bob" 1001 true] [1001 :db/txInstant <date> 1001 true] [1001 :transaction/source "internet" 1001 true]] As you can see, there can be more datoms on the transaction eid. One way to know is to find out if there is a :db/txInstant datom with E=1001. Then E=1001 is a transaction eid. There is also the concept of different database partitions, which is mostly an implementation detail, but if you do (part E) you will get back yet another eid, where (ident db (part E)) results in :db.part/db if it is a db-internal thing (partition, schema and some more things) :db.part/user as the default for your ordinary data and :db.part/tx if the E a transaction.

👍 1
Tobias Sjögren10:10:51

Essentially you are saying yes to my second question?

Linus Ericsson10:10:05

The datom [1001 :transaction/source "internet" 1001 true] has a transaction E without having :db/txInstant.

Linus Ericsson10:10:17

the entity 1001, however, has an attribute :db/txInstantwhich (I think) is equal to being in a transaction entity at all times.

Ivar Refsdal12:10:16

Can't you simply check if E equals T in the (EAVTO) datom? Correct me if I'm wrong please.

Linus Ericsson12:10:22

That would not work in the general case - other transactions are allowed to update older transacton entites (except for the special attribute :db/txInstant)

Ivar Refsdal12:10:27

Hm, how about checking the existence of [?e :db/txInstant _ ?e true] in the history database?

favila12:10:16

at least in on-prem, transaction entity-ids are in the transaction partition (3), which you can find via (d/part entity-id)

👍 1
favila12:10:33

that doesn’t tell you that an entity id is an actual transaction, just that it could be. But if you found this entity id in the E of a datom you can be pretty confident that it is unless you have been asserting on entity ids without minting them

favila12:10:33

if you want to be doubly sure, look up (d/datoms db :aevt :db/txInstant entity-id) if you find an assertion, it’s a transaction

favila12:10:01

:db/txInstant is special--only transactions have it, and you can’t remove it.

Michael Stokley16:10:33

i don't want to unify when entity ?e has values not specified by a given set. is there a way to express that with datalog? "give me all the companies that don't have offices outside of Virginia and North Carolina"

Fredrik17:10:22

You can use ground to bind a variable to specific values within a query https://docs.datomic.com/on-prem/query/query.html#ground

Fredrik17:10:28

Simple example

(d/transact conn-mem [{:sensor/humidity 124.0}
                      {:sensor/humidity 125.0}
                      {:sensor/humidity 126.0}])

(d/q '[:find [?a ...]
       :where
       [?e :sensor/humidity ?a]
       [(ground #{124.0 125.0}) [?a ...]]]
     (d/db conn-mem))
=> [124.0 125.0]
Is this what you want?

Michael Stokley17:10:31

i want the complement of that, sort of

Michael Stokley17:10:50

"exclude any sensor that has any readings that /don't/ fall into this range"

Michael Stokley17:10:27

"only return sensors for which ALL readings fall within a given range"

Michael Stokley17:10:55

(assuming there are multiple sensors each with multiple readings)

Fredrik17:10:50

I see what you mean. You want to take the set difference with those entities having at least one value outside a prescribed set.

Fredrik17:10:51

Modyfing the above example slightly, this was the most concise thing I could get

(d/transact conn-mem [{:db/id "sensor 1" :sensor/humidity 124.0}
                      {:db/id "sensor 1" :sensor/humidity 125.0}
                      {:db/id "sensor 2" :sensor/humidity 124.0}
                      {:db/id "sensor 2" :sensor/humidity 100.0}
                      {:db/id "sensor 3" :sensor/humidity 125.0}])

(d/q '[:find [?e ...]
       :where
       [?e :sensor/humidity]
       (not-join [?e]
                 [?e :sensor/humidity ?a]
                 (not
                  [(ground #{124.0 125.0}) [?a ...]]))]
     (d/db conn-mem))

Fredrik17:10:48

The not-join clause removes from consideration any entity satisfying the body, and differs from not in that you can select which variables needs to be pre-bound in the body. It looks like a double negation of my first example, but instead of looking at each E-A pair, it considers only the entity.

👍 1
Michael Stokley18:10:54

thanks, i'll try this out

Michael Stokley18:10:34

i was trying a not join but it didn't occur to me to have a new humidity lvar unbound to surrounding scope - and it didn't occur to me to use ground