Fork me on GitHub
#datomic
<
2016-10-07
>
djjolicoeur01:10:39

anyone ever have an issue with a datomic connection being null when trying to call transact after a long running query?

djjolicoeur02:10:03

the transactor appears to be healthy, other peers are transacting. It sometimes recovers and reconnects, other times it does not. I thought it may be due to transactor load, but I can reproduce it locally with no other peers hitting my transactor

pesterhazy11:10:24

@djjolicoeur what exceptions are you seeing?

djjolicoeur15:10:57

@pesterhazy it appears this is due to running several large, concurrent queries. Temporary fix was to bump up datomics TTL, but will likely be rewriting that section of code to be a bit smarter about how those queries are built and run.

bmays15:10:53

I have a pretty simple question about the query api on a history database — is there anyway to take the max txid for a single attribute lookup in the where clause?

(d/q '[:find ?e ?state ?tx
       :in $ ?eid
       :where
       [?e :someEntity/id ?eid]
       [?e :someEntity/state ?state ?tx] ;; want to only choose the latest tx value
       [?tx :db/txInstant ?inst]] hist-db "<valid id>")

=>

[[17592186484088 :someEntity.state/created 13194139995319] ;; want to choose this value in the query
[17592186484088 :someEntity.state/patientUnresponsive 13194139978521] 
[17592186484088 :someEntity.state/created 13194139972983]]

pesterhazy15:10:43

I'd try filtering in clojure code after the query completes @bmays

bmays15:10:35

Yup, that’s what I’ve been doing but wasn’t sure if there was an easier way

bmays15:10:56

I was curious if the single tuple syntax exposed a fn to select the tuple it returned

bkamphaus15:10:39

(max ?tx) in the :find clause produces a different grouping behavior than you want?

bmays15:10:24

I haven’t had success; the grouping isn’t what I expected:

user=> (d/q '[:find ?e ?state-name (max ?tx)
       :in $ ?eid
       :where
       [?e :someEntity/quartetId ?eid]
       [?e :someEntity/state ?state ?tx true] ;; want to only choose the latest tx value
       [?state :db/ident ?state-name]
       [?tx :db/txInstant ?inst]] hist-db #uuid "<totally valid uuid")

[[17592186484088 :someEntity.state/created 13194139995319] 
 [17592186484088 :someEntity.state/patientUnresponsive 13194139978521]]

djjolicoeur15:10:06

@bmays I have no idea how if this is a good idea or supported or not, but you can bind ?tx in a query evaluated within the query. I will whip an example real quick.

bmays15:10:01

okay thanks, that sounds promising

bmays15:10:28

@bkamphaus — intuitively I would expect this to return the max-tx for all the txs of this entity but it doesn’t seem to be grouping that way:

(d/q '[:find ?e (max ?tx) ?inst
              :in $ ?e
              :where
              [?e _ _ ?tx] ;; want to only choose the latest tx value
              [?tx :db/txInstant ?inst]] hist-db 17592186484088)

=>
[[17592186484088 13194139972983 #inst "2016-09-08T18:39:04.471-00:00”]
[17592186484088 13194139974019 #inst "2016-09-08T19:31:56.518-00:00"] 
[17592186484088 13194139974022 #inst "2016-09-08T19:32:00.594-00:00"] 
[17592186484088 13194139974058 #inst "2016-09-08T19:35:20.979-00:00"] 
[17592186484088 13194139974062 #inst "2016-09-08T19:35:48.438-00:00"] 
[17592186484088 13194139976273 #inst "2016-09-09T15:23:52.788-00:00"] 
[17592186484088 13194139976274 #inst "2016-09-09T15:24:00.969-00:00"] 
[17592186484088 13194139978521 #inst "2016-09-12T14:42:47.598-00:00"] 
[17592186484088 13194139978522 #inst "2016-09-12T14:43:49.859-00:00"] 
[17592186484088 13194139995319 #inst "2016-09-16T17:34:59.882-00:00"] 
[17592186484088 13194139995394 #inst "2016-09-16T17:46:15.081-00:00"] 
[17592186484088 13194139995605 #inst "2016-09-16T18:01:02.649-00:00"] 
[17592186484088 13194139995659 #inst "2016-09-16T18:05:33.927-00:00"] 
[17592186484088 13194139995660 #inst "2016-09-16T18:05:34.005-00:00"] 
[17592186484088 13194139995668 #inst "2016-09-16T18:05:40.830-00:00"] 
[17592186484088 13194140023845 #inst "2016-09-23T15:54:30.171-00:00"] 
[17592186484088 13194140037787 #inst "2016-09-28T21:34:13.794-00:00"] 
[17592186484088 13194140040978 #inst "2016-09-29T20:59:27.388-00:00"]]

bmays15:10:35

I suppose what I really want is to group all these txs by attribute and take the max/min of each of those to get created/updated times

bmays15:10:49

interesting, didn’t know you could nest

djjolicoeur15:10:22

neither did I, found out a while ago by accident, really. And again, I have no idea if its a good idea or not!

bmays15:10:38

I guess we’ll see, I’ll give it a run — thanks

djjolicoeur15:10:06

I didn’t run that so it might need a few tweaks to run, but I think you get the idea

djjolicoeur15:10:37

a :with clause might also get you what you need, here instead of a nested query

bkamphaus15:10:10

@bmays I believe the reason it doesn’t is because of the instant. You’re returning the max-tx for all unique [entity, instant] pairs, which w/roughly 1->1 instant correspondence to tx means no meaningful max.

bkamphaus15:10:28

I would pull the instant using the tx after the query and take that out of the find clause.

bmays15:10:15

interesting

bmays15:10:22

that did indeed work

bkamphaus15:10:22

in general I try to structure queries that way — provide just enough information to limit your results to the tuples of interest, then use pull or entity to retrieve more information. Roughly splitting what would be the select and project portions of a query in the relational algebra world.

bmays15:10:38

Definitely. It’s making sense — I was binding the ?state value and getting back two groupings

bmays16:10:14

@djjolicoeur the subquery did the job, here is the working version:

(d/q '[:find ?e ?state-name ?state-tx-max-inst
        :in $ ?e
        :where
        ;; get the max-tx for the attribute :serviceRequest/state by running a subquery for all
        ;; txs affecting the attribute and taking the max
        [(datomic.api/q '[:find (max ?state-tx) .
                          :in $ ?e
                          :where
                          [?e :serviceRequest/state _ ?state-tx true]] $ ?e) ?state-tx-max]
        ;; get the value of the :serviceRequest/state at the latest tx
        [?e :serviceRequest/state ?state ?state-tx-max true]
        [?state :db/ident ?state-name]
        ;; get the last updated time for the :serviceRequest/state attribute
        [?state-tx-max :db/txInstant ?state-tx-max-inst]]
      hist-db 17592186484088)

bmays16:10:45

thanks for your help

djjolicoeur16:10:07

no prob, glad it worked out for you!