Fork me on GitHub
#datomic
<
2023-05-30
>
icemanmelting17:05:54

Let's say we have a transaction query that is basically increasing a counter to create auto-increment ids for a specific entity. How could I use the resulting tx-data from a transaction of said entity, to retrieve the id used? Is there a straighforward way of doing it?

favila17:05:14

query the db-after db?

icemanmelting17:05:38

Can't I use the datoms collection in the tx-data result?

icemanmelting17:05:48

That way I would avoid querying the db once again

favila17:05:55

you asked for a “straightforward” way 🙂

favila17:05:18

I would not expect querying just-transacted data to be costly

favila17:05:24

it’s going to be in the mem log

icemanmelting17:05:41

true, although, I would need to know the id to get the event 😄

favila17:05:28

if this is a “Create” context, I assume that’s just resolve-tempids, or else you already know some other minted id?

favila17:05:39

but that’s about resolving to the entity, not retrieving its autoincrement id

icemanmelting17:05:53

let me show you this

icemanmelting17:05:06

{:db-before {:database-id "iacts-loan-amendments-afe56c56-82a4-4c09-ae56-91c48470b755",
             :db-name "iacts-loan-amendments",
             :t 2110,
             :next-t 2116,
             :type :datomic.client/db},
 :db-after {:database-id "iacts-loan-amendments-afe56c56-82a4-4c09-ae56-91c48470b755",
            :db-name "iacts-loan-amendments",
            :t 2116,
            :next-t 2122,
            :type :datomic.client/db},
 :tx-data [#datom[13194139535428 50 #inst"2023-05-30T17:23:34.050-00:00" 13194139535428 true]
           #datom[17592186046533 80 "asdasdasda" 13194139535428 true]
           #datom[17592186046533 82 1231341234 13194139535428 true]
           #datom[17592186046533 79 "this is a comment" 13194139535428 true]
           #datom[17592186046533 83 1214134123 13194139535428 true]
           #datom[17592186046533 91 17592186046534 13194139535428 true]
           #datom[17592186046534 119 17592186045475 13194139535428 true]
           #datom[17592186046534 120 17592186046535 13194139535428 true]
           #datom[17592186046535 123 0 13194139535428 true]
           #datom[17592186046535 124 "this is a a general narrative" 13194139535428 true]
           #datom[17592186046533 90 17592186046536 13194139535428 true]
           #datom[17592186046536 115 "001" 13194139535428 true]
           #datom[17592186046536 116 17592186045428 13194139535428 true]
           #datom[17592186046536 117 "this is option 1" 13194139535428 true]
           #datom[17592186046533 90 17592186046537 13194139535428 true]
           #datom[17592186046537 115 "002" 13194139535428 true]
           #datom[17592186046537 116 17592186045428 13194139535428 true]
           #datom[17592186046537 117 "this is option 1" 13194139535428 true]
           #datom[17592186046533 84 12341234134 13194139535428 true]
           #datom[17592186046533 96 "ffrancisco" 13194139535428 true]
           #datom[17592186046533 72 "sch" 13194139535428 true]
           #datom[17592186046533 78 123123.2 13194139535428 true]
           #datom[17592186046533 76 17592186046308 13194139535428 true]
           #datom[17592186046533 10 :e17bfada-1933-465e-b3b3-006483f74c34 13194139535428 true]
           #datom[17592186046533 81 17592186045438 13194139535428 true]
           #datom[17592186046533 95 17592186046221 13194139535428 true]
           #datom[17592186046533 73 "sch-ch" 13194139535428 true]
           #datom[17592186045418 98 #uuid"e61efbd6-b368-4970-9a5a-e1f38aa2275e" 13194139535428 true]
           #datom[17592186045418 98 #uuid"7586e7a4-7182-4476-adee-7f0ff0b47ede" 13194139535428 false]
           #datom[17592186045418 99 6 13194139535428 true]
           #datom[17592186045418 99 5 13194139535428 false]
           #datom[17592186046533 74 5 13194139535428 true]
           #datom[17592186046533 75 ["sch" "sch-ch" 5] 13194139535428 true]],
 :tempids {"e17bfada-1933-465e-b3b3-006483f74c34" 17592186046533,
           "datomic.temp-1000000026938" 17592186046534,
           "datomic.temp-1000000026939" 17592186046535,
           "datomic.temp-1000000026940" 17592186046536,
           "datomic.temp-1000000026941" 17592186046537}}

icemanmelting17:05:20

I am interested in #datom[17592186046533 75 ["sch" "sch-ch" 5] 13194139535428 true]]

icemanmelting17:05:29

the number 5, or the entire vector

favila17:05:39

So "e17bfada-1933-465e-b3b3-006483f74c34" is your tempid

icemanmelting17:05:03

how can you use the temp id to resolve the value?

icemanmelting17:05:14

i am sorry, but i am still a noob in these matters 😄

favila17:05:41

cloud or on-prem?

icemanmelting17:05:21

is there a way to do it with the client api?

favila17:05:51

in client api, :tempids is a string->eid map only, so you just look in there

favila17:05:15

(get (:tempids tx-response) "e17bfada-1933-465e-b3b3-006483f74c34")

icemanmelting17:05:20

yeah, true, I just looked at the code again, I generate the id, so I can just get the eid, and resolve it to a value

favila17:05:14

there’s no way to avoid the round-trip on client, unless you cache the entity id of the attribute somewhere

icemanmelting17:05:49

well, I can always filter the datom collection and get the value out of the result as well

icemanmelting17:05:56

my problem was matching the eid

favila17:05:16

to filter the datom collection safely and reliably, you need to know the entity id of the attribute

icemanmelting17:05:46

and with your input, I can now know it for sure, since the temp id, is generated by me in the code

icemanmelting17:05:03

so I just need to use it to get the actual eid, and then filter the datoms to retrieve the value

favila17:05:26

but if you filter by entity id, most of the datoms in there are assertions on that entity

favila17:05:42

you only want a specific one (I think attribute 75?)

favila17:05:54

but you lack the mapping from an attribute keyword to the number 75

icemanmelting17:05:07

well, then query it is I guess 😄

icemanmelting17:05:32

attr 75, but the problem is that if the db changes, the attr number will change as well, let's say on a new installation

favila17:05:11

yes, or if you migrate to a new attribute with the same name

favila17:05:35

(not a great practice, but it is another way)

icemanmelting17:05:56

yeah, I will just resolve the entity and get the correct attribute by name

icemanmelting17:05:04

that way it will actually be correct

favila17:05:56

(d/pull (:db-after tx-response) [:counter-attr] (get (:tempids tx-response) "e17bfada-1933-465e-b3b3-006483f74c34") after putting it all together

favila17:05:32

or generalizing into a query: [:find ?tempid ?realid ?incid :in $ [?tempid …] [[?tempid ?realid]] :where [?realid :counter-attr ?v][(untuple ?v) [_ _ ?incid]]

icemanmelting17:05:11

yeah, I got it, although I like the pull pattern one, looks way more clean 😄

favila17:05:22

(d/q the-query (:db-after tx-response) ["e17bfada-1933-465e-b3b3-006483f74c34"] (:tempids tx-response))

icemanmelting17:05:13

(defn retrieve-id [temp-id tx-data]
  (ffirst (let [eid (get-in tx-data [:tempids temp-id])
                db (:db-after tx-data)]
            (d/q {:query '[:find ?id
                           :in $ ?e
                           :where
                           [?e :some-entity/some-attr ?id]]
                  :args [db eid]}))))

icemanmelting17:05:22

I have that, it works properly as expected

👍 4
icemanmelting17:05:55

thanks for your help, if you're ever in Portugal, I will pay you a beer 😄

favila17:05:16

or ice cream 🙂

icemanmelting17:05:28

whatever you like 🙂

favila18:05:31

I think that joke didn’t land 🙂 The two sound similar to me in Portuguese so I was always getting them mixed up.

icemanmelting18:05:48

ahhhh well, it depends on the accent maybe

icemanmelting18:05:09

usually people tend to be more familiar with the brazilian accent

icemanmelting18:05:20

so, that might be the reason why it didn't land properly

icemanmelting18:05:34

Brazilians say sorvete, we say gelado

icemanmelting18:05:58

which is worlds different from cerveja 😄

favila18:05:05

yeah I was in brazil, and it was sorvete vs cerveja

icemanmelting18:05:26

yeah, here it is different, we understand it, but we speak way differently than them

Gonzalo Rafael Acosta19:05:43

Hello! I am using datomic in prod (first time), we have 4.6G datoms, 2.3M rows of the main entity, 2 transactor instances type i3.large, 2 query instances type t3.xlarge. Currently trying to do a query that pulls almost all the data of a reduced well defined group of entities, let's say the entities that match a certain address. Among the attributes of this entity there's one of cardinality many, and there's 6.7M of them in total. I been doing some performance testing and our api is having trouble handling a ramp of 50 users in 10 secs. I am a bit lost because I don't know if we don't have enough resources or I am doing something wrong myself. Any help or hint would be appreciated.

Daniel Jomphe09:05:42

https://www.youtube.com/watch?v=bvEsnJiCs7E might improve your intuition and give you observability tooling to learn about your performance profile: > Datomic plays a major role in enabling Nubank to scale to over 75 million customers. Across the company, there are more than 3,000 Datomic databases, with over 300 storing more than 10 billion datoms. > > This talk explains Nubank’s use of two features of Datomic to ensure that transactions and queries are as performative as they can be: io-stats, and query-stats. > > The Operating Datomic at Scale talk also aims to show two experiments currently running at Nubank: tuning the memory index to reindex less often, severely reducing expenses with DynamoDB; and the use of ZGC as the garbage collector for JVMs running transactors.

💚 4
Gonzalo Rafael Acosta10:05:55

👌 thanks. Will definitely watch it

Robert A. Randolph16:05:52

The speakers are both here in the channel if you have any questions as well. We'd love to offer assistance via <mailto:[email protected]|[email protected]> if you want to dig in further on your specific systems.

Gonzalo Rafael Acosta19:05:32

Thanks Robert, I am already in touch with support