This page is not created by, affiliated with, or supported by Slack Technologies, Inc.
2021-10-20
Channels
- # announcements (33)
- # aws (1)
- # babashka (8)
- # beginners (100)
- # calva (59)
- # clara (4)
- # clj-kondo (33)
- # cljdoc (9)
- # cljs-dev (30)
- # cljsrn (1)
- # clojure (28)
- # clojure-australia (1)
- # clojure-boston (1)
- # clojure-dev (4)
- # clojure-europe (14)
- # clojure-france (5)
- # clojure-italy (7)
- # clojure-nl (1)
- # clojure-uk (36)
- # clojurescript (13)
- # clojureverse-ops (6)
- # conjure (2)
- # cursive (2)
- # datahike (11)
- # datalevin (1)
- # datomic (106)
- # graphql (3)
- # helix (10)
- # holy-lambda (24)
- # kaocha (2)
- # lambdaisland (3)
- # lsp (199)
- # malli (35)
- # off-topic (16)
- # pathom (7)
- # polylith (38)
- # portal (16)
- # quil (2)
- # re-frame (18)
- # reagent (57)
- # shadow-cljs (11)
- # testing (3)
- # xtdb (9)
Say I want to define the schema attribute “first_name” in a database - what would the needed datoms look like (in storage)?
something along the lines of https://github.com/solita/mnt-teet/blob/master/app/backend/resources/schema.edn#L55
These are not actual Datoms in storage, right? That is what I’m looking for - to see the actual triples..
the map format is a convenient way to express multiple facts about a single entity, so it's not exactly the datoms in storage
What I tried to ask is how the datoms would look like in storage..
for that particular case, it looks like the below... I have no idea how they are actually represented in storage
(d/q '[:find ?e ?a ?v :where [?e ?a ?v] :in $ ?e] (db) :user/given-name)
=> [[:user/given-name 40 23]
[:user/given-name 10 :user/given-name]
[:user/given-name 41 35]
[:user/given-name 63 "User's given name"]]
I wonder how accurate this is..
afaict, entities are always identified by the 64bit number, but the number :db/id and keyword :db/ident is often used interchangeably when showing info
That fits with the example database here: https://docs.datomic.com/cloud/time/filters.html I guess the actual stored entity id is the 64bit number and :db/id and :db/ident are just aliases..
It’s a tuple of [e a v tx op], where e, a, tx are entity ids (longs) op is a boolean (true=assert, false=retract) and V is an object whose type depends on the valuetype of A
note that :db/id is just syntax for the map form, it’s not an ident. There’s no attribute :db/id
nor is there a datom corresponding to it. it’s just the E that everything in the map has in common
I actually don’t use Datomic (yet) - I’m trying to recreate the basic functions in FileMaker to understand everything before probably moving on to using Datomic.. So “a” is also a long and referencing the attribute somehow? If I could see a “entity id entity”, an “attribute entity” and a “transaction entity” as raw data Datoms it would be great.. Your last message I didn’t quite understand..
the unit of truth in datomic is a datom, which I described. Everything you see that looks like a map and represents an “entity” is merely a projection into/out-of those datoms
entities don’t exist in the system as entities---they are merely “the datoms which share an E value”
Right. What would an “attribute entity” look like as raw datom(s) ? The “:db/ident” is not really the raw data, right?
A datom where you define an attribute.
You said the “A” were entity ids (longs)..
An entity becomes a legal attribute by being asserted on the db entity’s :db.install/attribute attr, e.g. [:db.part/db :db.install/attribute <ATTR> <some-tx> true]
(In fact you used to have to do this explicitly in the transaction, it’s implicit now)
Every “A” is an entity id ? So “:db/ident” is a representation of an underlying entity id? If I add the :person/name-first attribute e.g., this attribute will be represented by an entity id in a datom?
(layman warning...)
every attribute is an entity. :db/ident
is a way to “name” an entity so that a keyword where an entity id is expected will resolve to that entity. (this is called “entity lookup”, and there’s one more way to do it using a unique attribute and the syntax [:unique-attr unique-value]
)
represented by some datom and an entity id?
If you really want to understand how datomic bootstraps itself, I recommend looking at the earliest transactions and seeing how they establish the foundation for the system. You can see them with this:
(d/create-database "datomic:")
(-> (d/connect "datomic:")
(d/db)
(d/seek-datoms :eavt 0)
(->> (group-by :tx)
(sort)
vals))
In particular, you’ll notice that entity 10 is the :db/ident
attribute itself, and it names itself self-referentially with datom [10 10 :db/ident 13194139533312 true]
. Entity 0 is the database, entity 13 is the :db.install/attribute attribute, and attribute installations have the pattern [0 13 <entity-id-of-attribute> TX true]
Is there a place to read all those “foundation datoms” without having Datomic?
But with the right Datomic command, you can see all of it?
these are “normal” datoms, there’s no extra meta “DDL” layer here made of different stuff
but the entity ids are hard coded for these foundation stuff?
To blow your mind a bit more, entity ids are composed of partition in the high bits and a T value in the low bits. The partitions are also entities, and there are three predefined ones (0=db, 3=tx 4=user). so entity ids have entity ids inside them too
db = 0, which is why these bootstrap entities are small numbers, because all the high bits are clear
So, during installation, datoms are added to storage that represent the foundation for each subsequent added datom? In this way about everything consists of datoms?
@U09R86PA4 When you say “V is an object whose type depends on the valuetype of A” I’m not really following what you mean by “V is an object” - in what sense is it an object?
I don’t quite get that.. If I put the string “Francis” as the V - it is not anything else than just a string?
@U09R86PA4 When you wrote [0 13 <entity-id-of-attribute> TX true], the datom order is A, E, V, TX, OP - right ?
That was what I excepted when I saw these “foundational datoms”:
([10 0 :db.part/db 13194139533312 true]
[11 0 0 13194139533366 true]
[11 0 3 13194139533312 true]
[11 0 4 13194139533312 true]
[12 0 20 13194139533366 true]
[12 0 21 13194139533366 true]
[12 0 22 13194139533366 true]
[12 0 23 13194139533366 true]
[12 0 24 13194139533366 true]
[12 0 25 13194139533366 true])
But from what I can understand this is A E V Tx OP…I actually don’t know..
Here’s the full batch..
If that is not A E V Tx OP - I have to conclude that I don’t understand this at all…
From someone that, unlike me, have Datomic installed..
The content of the vector are datoms in a specific slot order, right?
The txt file you provided has only one vector in it. I guess you could presume based on line breaks and order and other clues that this is a cycling pattern of slots, but these are not datoms. As you noticed, it doesn’t seem to be E A V T Op
If so, I’m still looking to get the “foundational datoms” ...
I know I should install Datomic. I need help to do it though. The amount of time it would take me to do it alone would make me loose focus on what I’m trying to achieve..
Possibly just as an intermediate step: Represent the core principles and ideas behind Datomic in my current platform (FileMaker).
Doing so will hopefully allow me to be sure if I should leave it (FileMaker) or not..
implementing an immutable graph database in a relational database seems like a much, much bigger ask than installing and evaluating datomic (or any other dbs you may be considering) on its own merits.
For sure. It’s just - I want to do it.. I want to see how far I can reach. So far it looks promising. It will be far from as performant as Datomic, but possibly still usable..
ok, then maybe getting the “foundational datoms” of datomic is a distraction. Think about the minimum you would need to self-describe the system before you could implement the code that emits datoms from transaction data. You only need to know enough to perform ident lookups, and it needs some known idents that describe transaction and schema-related operations
I would agree to that they are partly just a distraction, but it makes me think about that self-describe thing.. I’m currently trying to construct an interface for constructing the schema attributes..
By looking at storage data, how do I know when an entity id (in the E position of the datom) represents a transaction id? Is it when the datom attribute is “:db/txInstant” ?
example - transaction:
[[1 :cat/name "bob" 1001 true]
[1001 :db/txInstant <date> 1001 true]
[1001 :transaction/source "internet" 1001 true]]
As you can see, there can be more datoms on the transaction eid.
One way to know is to find out if there is a :db/txInstant datom with E=1001.
Then E=1001 is a transaction eid.
There is also the concept of different database partitions, which is mostly an implementation detail, but if you do
(part E)
you will get back yet another eid, where (ident db (part E))
results in
:db.part/db
if it is a db-internal thing (partition, schema and some more things)
:db.part/user
as the default for your ordinary data and
:db.part/tx
if the E a transaction.
Essentially you are saying yes to my second question?
The datom [1001 :transaction/source "internet" 1001 true]
has a transaction E without having :db/txInstant
.
the entity 1001, however, has an attribute :db/txInstant
which (I think) is equal to being in a transaction entity at all times.
Can't you simply check if E equals T in the (EAVTO) datom? Correct me if I'm wrong please.
That would not work in the general case - other transactions are allowed to update older transacton entites (except for the special attribute :db/txInstant)
Hm, how about checking the existence of [?e :db/txInstant _ ?e true] in the history database?
at least in on-prem, transaction entity-ids are in the transaction partition (3), which you can find via (d/part entity-id)
that doesn’t tell you that an entity id is an actual transaction, just that it could be. But if you found this entity id in the E of a datom you can be pretty confident that it is unless you have been asserting on entity ids without minting them
if you want to be doubly sure, look up (d/datoms db :aevt :db/txInstant entity-id)
if you find an assertion, it’s a transaction
i don't want to unify when entity ?e has values not specified by a given set. is there a way to express that with datalog? "give me all the companies that don't have offices outside of Virginia and North Carolina"
You can use ground
to bind a variable to specific values within a query
https://docs.datomic.com/on-prem/query/query.html#ground
Simple example
(d/transact conn-mem [{:sensor/humidity 124.0}
{:sensor/humidity 125.0}
{:sensor/humidity 126.0}])
(d/q '[:find [?a ...]
:where
[?e :sensor/humidity ?a]
[(ground #{124.0 125.0}) [?a ...]]]
(d/db conn-mem))
=> [124.0 125.0]
Is this what you want?i want the complement of that, sort of
"exclude any sensor that has any readings that /don't/ fall into this range"
"only return sensors for which ALL readings fall within a given range"
(assuming there are multiple sensors each with multiple readings)
I see what you mean. You want to take the set difference with those entities having at least one value outside a prescribed set.
Modyfing the above example slightly, this was the most concise thing I could get
(d/transact conn-mem [{:db/id "sensor 1" :sensor/humidity 124.0}
{:db/id "sensor 1" :sensor/humidity 125.0}
{:db/id "sensor 2" :sensor/humidity 124.0}
{:db/id "sensor 2" :sensor/humidity 100.0}
{:db/id "sensor 3" :sensor/humidity 125.0}])
(d/q '[:find [?e ...]
:where
[?e :sensor/humidity]
(not-join [?e]
[?e :sensor/humidity ?a]
(not
[(ground #{124.0 125.0}) [?a ...]]))]
(d/db conn-mem))
The not-join
clause removes from consideration any entity satisfying the body, and differs from not
in that you can select which variables needs to be pre-bound in the body. It looks like a double negation of my first example, but instead of looking at each E-A pair, it considers only the entity.
thanks, i'll try this out
i was trying a not join but it didn't occur to me to have a new humidity lvar unbound to surrounding scope - and it didn't occur to me to use ground