Fork me on GitHub
#datomic
<
2024-05-15
>
cch101:05:06

The Datomic Cloud https://docs.datomic.com/client-api/datomic.client.api.html#var-datoms states that datoms are "associative and indexed". What does "indexed" mean? I expected it to mean that I can access the e/a/v/tx/added properties with their index (e.g. (my-datom 1) or (get my-datom 1)) but that does not work.

cch101:05:41

At the end of the day, I want to query the returned tx-data from a transaction. The documentation on this is scant and other examples I have seen no longer work reliably in Datomic Cloud and/or Datomic Local.

Joe Lane01:05:50

(nth my-datom 1) -> my-attr

(let [[e a v t op] my-datom]
 {:e e :a a :v v :t t :op op})

cch101:05:37

I did notice that sequential destructuring does work (but isn't it weird that seq does not?).

Joe Lane01:05:15

Nope. It is indexed, but it does not implement sequence

cch101:05:16

Anyways, in order to get the datoms back into a format for sending back to the server for a query, that can work but it's pretty convoluted.

cch101:05:45

(since I need to turn it into a relation)

Joe Lane01:05:45

I'm not quite sure what you're trying to do, so I can't give guidance on what is idiomatic

cch101:05:48

Send tx-data (from a transaction) back into a query. I have seen out-of-date examples treating the tx-data as a data source, but they no longer work consistently AFAICT with local and cloud. Next option is to convert the tx-data to a set of relations. That's where the indexed interface seems appropriate.

Joe Lane01:05:55

Got a link to the example you're referring to?

cch101:05:15

give me a moment...

Joe Lane01:05:17

This should work, and I want a repro (not necessarily from you) if it doesn't

cch101:05:37

This -> treating tx-data as a data source?

Joe Lane01:05:58

> I have seen out-of-date examples treating the tx-data as a data source, but they no longer work consistently AFAICT with local and cloud. Whatever you're referring to ^^

cch101:05:47

Yes, as a data source (`$...`), instead of as relations (`[[?x ?y]...]`).

cch101:05:07

Let me see if I can reproduce.

Joe Lane01:05:22

You can also link to what you think should work, again, not trying to task you, I know it is late. Just trying to get something queued up for tomorrow morning.

cch102:05:29

And, trying to replicate it in my context is this:

(defn- acquire-shard
  "Transactionally claim the current shard, incrementing the persisted value modulo `modulus`"
  [conn modulus]
  ;; Sometimes DC seems broken... why can't this be done in one db call?!
  (let [tx-data [(template (st.datomic/rotate-attr :st.data-source/shard :st.db/shard ~modulus))]
        {:keys [db-after tx-data]} (d/transact conn {:tx-data tx-data})]
    (ffirst (d/q {:query '{:find [?v]
                           :in [$ $tx-report]
                           :where [[$tx-report ?e ?a ?v ?tx]
                                   [?e :st.db/index ?v]]}
                  :args [db-after tx-data]}))))

cch102:05:15

The tx-data is returned correctly, but the query throws

Execution error at com.cognitect.transit.impl.AbstractEmitter/marshal (AbstractEmitter.java:195).
Not supported: class datomic.client.impl.shared.datom.Datom

Joe Lane02:05:30

Got it, I'll set a reminder for tomorrow to queue this up.

cch102:05:23

Thanks. In the meantime, I'll construct a workaround converting the tx report to a relation ...via associative and back to sequential.

cch102:05:33

(or something like that)

Joe Lane02:05:56

And you've confirmed turning it into explicit [e a v t op] tuples works?

(defn tx-data->transitable-datom-vecs
  [tx-data]
  (into [] (map (fn [[e a v t op]] [e a v t op])) tx-data))

yes 4
Joe Lane02:05:24

(might have missed a paren, slack seems to be only my 3rd best editor these days)

Sven09:05:27

If I have defined a composite tuple where one of its :db/tupleAttrs is a :db.type/ref then is there a way in an :xform option inside pull pattern to pull the :db/ident of that attribute (or any other value by key)? Here’s an example.

(def schema
  [#:db{#_#_:id 96757023244426
        :ident :task-status/DONE}

   #:db{:ident :task/due-date
        :valueType :db.type/string
        :cardinality :db.cardinality/one
        :doc "YYYY-MM-DD"}

   #:db{:ident :task/status
        :valueType :db.type/ref
        :cardinality :db.cardinality/one
        :doc "e.g. :task-status/DONE"}

   #:db{:ident :task/status+due-date
        :valueType :db.type/tuple
        :tupleAttrs [:task/status
                     :task/due-date]
        :cardinality :db.cardinality/one}])

;; Added to :xforms in ion-config.edn
(def enum :db/ident)
(defn progress [status+due-date] (update-in status+due-date [0] enum))

(d/transact CONN {:tx-data [#:task{:status :task-status/DONE :due-date "2020-01-01"}]})

(->> {:query '[:find (pull ?e [:task/due-date
                               [:task/status :xform enum]
                               [:task/status+due-date :as :task/progress :xform progress]])
               :where [?e :task/status+due-date]]
      :args [DB]}
     d/q
     ffirst)

;; I would expect
#:task{:due-date "2020-01-01" :status "DONE" :progress ["DONE" "2020-01-01"]}
;; ...but get
#:task{:due-date "2020-01-01" status: "DONE" :progress [96757023244426 "2020-01-01"]}
I’d like to be able to pull the :db/ident of a ref inside the xform as this ties business logic to pull patterns and then they can be really conveniently composed (e.g. for GraphQL). It works well for cases with primitive tupleAttrs but not with refs 😞
(defn display-name
  "Takes a tuple of `email+given-name+family-name` and returns:
  * full name as string if given and/or family name exist
  * email as a fallback"
  [email+given-name+family-name]
  (or (some->> email+given-name+family-name
               rest
               (remove nil?)
               (remove str/blank?)
               seq
               (map str/trim)
               (str/join " "))
      (first email+given-name+family-name)))

(def user
  [:user/given-name
   :user/family-name
   [:user/email+given-name+family-name :as :user/display-name :xform 'display-name]
   :user/email])

(def project
  [:project/id
   {:project/owner user}])

(def task
  [:task/id
   :task/project project
   {:task/approver user}
   {:task/assignee user}])

wevrem16:05:58

I don’t have a solution, just a “I wonder…“. Can you put more binding forms under :where instead of doing the pull? Maybe something like this? (not tested)

:where [?e :task/status+due-date ?tup]
       [(untuple ?tup) [?status-ref ?due-date]]
       [?status-ref :db/ident ?status]       

Sven17:05:26

Yes, that’s exactly what I am doing but would like to avoid if possible :)

favila11:05:28

You can’t follow refs in a tuple via a pull. If this were not the root of the query you could use xform to add the composite tuple entry manually, but not at the root like you have here

favila11:05:14

You should also explicitly include :db/ident in your pull on task status—I’m a little surprised this works without it, and it may be due to a bug in cloud vs on-prem

👀 1
Sven12:05:53

> You should also explicitly include :db/ident in your pull on task status—I’m a little surprised this works without it, and it may be due to a bug in cloud vs on-prem I am wondering if anyone from the Datomic team can confirm it. I have never included :db/ident in the pull as I have been able access it inside xform. As this has been working, I was expecting a ref inside a composite tuple to work the same way 😕

favila12:05:44

In on-prem, a bare [:some-ref] is the equivalent of [{:some-ref [:db/id]}], but cloud has a bug where [:db/id] acts like *

1
favila12:05:06

I’ll look for the conversation

Sven12:05:26

Anyway, thanks for the help! I now know how things should be working 🙂

favila12:05:39

{(:enum-ref :xform enum) [:db/ident]} should be future-proof against this getting fixed

👍 1
favila12:05:39

You can even wrap this up in a function (enum-ref :attr) that returns the above and use it in pull expression

Sven12:05:48

that’s neat! Thanks.

jaret12:05:10

📢 Hey Datomic folks! Big news! 🎉 We’ve merged and updated the Datomic Cloud and Pro docs into one spot. http://docs.datomic.com and let us know what you think. Your feedback is super important to us! This was a big project, so I imagine there will be some problems. Pardon our dust!

🚀 15
❤️ 9
catjam 5
datomic 8
cch113:05:16

+1 . I'm all for merging the docs -there is much commonality and anything to make the Datomic team more efficient is in all of our interests.

cch113:05:56

Observation: the javadoc is only for the peer. You should either include the javadocs for the other client libs or make it clear that the javadocs are only for the peer.

👍 1
jjttjj16:05:26

This looks awesome. Was there an intention to not break old URLs? Because all the ones in my browser history don't work: https://docs.datomic.com/cloud/query/query-data-reference.html#inputs https://docs.datomic.com/cloud/client/client-api.html#sync (This one also seems to be a deadlink in the new docs' sidebar) https://docs.datomic.com/cloud/transactions/transaction-data-reference.html

jjttjj16:05:02

I don't personally have extreme opinions on it, but datomic docs breaking old URLs seems pretty ironic 😄. At the least it's probably bad for SEO and would lead to a ton of broken links on the internet

jjttjj16:05:12

I see this was immediately fixed already, so thanks!

jaret16:05:17

We're trying to fix some redirects right now 🙂 ... our general goal is to keep iterating on the docs for the next few weeks so any feedback we see we will try to address ASAP.

miwal13:05:59

Is https://github.com/dazld/awesome-datomic still in use? Or is there now an alternative? If not, an idea: perhaps some of us could take a look, and see what there is to add? (last commit was 2 years ago).

jjttjj14:05:49

Not a direct answer to your question but I often find myself doing github code searches to figure out if anyone built a thing out of two other things e.g. https://github.com/search?q=language%3Aclojure+datomic+clojure.java.classpath&amp;type=code

👍 1
miwal13:05:29

I was ruminating that a project template with the sole purpose of defining classpath transaction functions and compiling them for inclusion on transactor classpath would make sense for my project for modularity reasons, yet this seems generic functionality, so before building it, need a way to check if it's already been built.

fabrao16:05:42

something went wrong into Datomic documentation

Darlei Soares16:05:02

we are actively looking into it, can you provide more details?

fabrao16:05:34

I think now it's ok. Previously it was including some extra / into url like -> https://docs.datomic.com//transactions/transaction-functions.html

fabrao16:05:51

in search field

Darlei Soares16:05:28

I see, thanks for the report, if you find anything please share with us

fabrao16:05:40

You bet ya

Darlei Soares17:05:54

thanks, we are moving this link, this should be fixed soon

Alex Miller (Clojure team)16:05:24

datomic docs were just overhauled to combine pro and cloud, if you found something wrong, I'm sure they would welcome a report

enn18:05:12

I like how clearly this describes some of the peculiarities of Datomic’s intra-transaction behavior. I have to confess that it took me quite a while using Datomic to really grok how different its transactions are from those I’d been familiar with from the SQL world, and what those differences mean for correctness.

👍 2
stuarthalloway17:05:14

Jepsen recently released their https://jepsen.io/analyses/datomic-pro-1.0.7075 (thanks @aphyr), and it is evident that our docs were insufficient. Rich and I have written what we hope is https://docs.datomic.com/transactions/model.html. We hope that this can preempt common misconceptions and we welcome your feedback!

8
cch118:05:49

I too appreciate this clarified documentation. The emphasis on what values a tx function sees is now crystal clear.

lwhorton21:05:27

I reference and share datomic's "grow, don't break" schema link often enough that I immediately noticed the new docs layout/links. Great job team!

hifumi12323:05:05

Has the Datomic team considered requesting Google to re-crawl the new documentation page? cf. https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl I think this would be beneficial to users at the moment, since Google is returning lots of obsolete links to documentation

jaret23:05:56

Thanks @U0479UCF48H I will look into this.

jaret00:05:59

Apparently @UEFE6JWG4 has already requested a re-crawl. So we will look into monitoring that or figuring out when its finished.

👍 1
jaret00:05:13

> Site is still in "priority crawl queue"

stuarthalloway17:05:14

Jepsen recently released their https://jepsen.io/analyses/datomic-pro-1.0.7075 (thanks @aphyr), and it is evident that our docs were insufficient. Rich and I have written what we hope is https://docs.datomic.com/transactions/model.html. We hope that this can preempt common misconceptions and we welcome your feedback!

8