I can't figure out what's wrong with the following query: (d/q '[:find ?tx ?e :in ?log ?t1 ?t2 :where [(tx-ids ?log ?t1 ?t2) [?tx ...]] [(tx-data ?log ?tx) [[?e]]] [?tx :db/txInstant ?time]] (d/log conn) t1 t2)

It tells me IllegalArgumentExceptionInfo :db.error/invalid-data-source Nil or missing data source. Did you forget to pass a database argument? and it works fine if I take out the [?tx :db/txInstant ?time]

not sure but what if you add $ '[:find ?tx ?e :in $ ?log ?t1 ?t2 :where

@ibarrick d/log looks suspicious, are you sure that is serializeable?

I suspect that just won't work with client api

wait you're not talking about client api anymore

the clause where you go [?tx :db/txInstant ?time] requires a "data source" (i.e. a datomic db) to satisfy

so you need to provide a db as args too

@jazzytomato That told me it was expecting 4 arguments but only received 3. I was able to get results by adding (d/db conn) right before (d/log conn) but I'm not positive why I need to include the db and the log or even if I'm getting the results I want with that.

Maybe @favila just answered my question though. I switched from the client API because I couldn't get tx-ids or tx-data to work at all with queries from the client api

@ibarrick didn't test this but maybe this works:

(d/q '[:find ?tx ?e :in ?log ?t1 ?t2 :where [(tx-ids ?log ?t1 ?t2) [?tx ...]]  [(tx-data ?log ?tx) [[?e :db/txInstant ?time]]]] (d/log conn) t1 t2)

So why can I query over the return of (d/log conn) (using just tx-ids and tx-data) if it isn't technically a "data source"?

look at your query clauses: none of them are actual clauses, they are just functions and destructuring

the data flows through just fine

it's when you have [$ ?e ?a ?v ?tx] clauses that a data source is involved

note also that ?tx = ?e

I think I follow you on the first part but I'm not sure why ?tx would be equal to ?e and in my results they appear to not be equal

I was looking at @thegeez 's example

@ibarrick what is it you really want? tx, entities mentioned, tx-instant?

your end goal, not your impl

Oh okay. what about the "$"? should my [?tx :db/txInstant ?time] have been [$ ?tx :db/txInstant ?time]?

And yes I want exactly what you described

I am reminding you that clauses like that have a source var, which can be omitted (and defaults to $ if you do)

so if a clause could take a source-var at the beginning, it's a clause that needs a db ("data source")

This seems like the easiest way to get a log of what changes were made to which entities during a time period, ordered by time.

you want what changes, or just the entities?

why not use the log directly and ignore query?

I guess I could just use the log, buy how would I extract the timestamps of the transactions? Would I have to do a query for each unique transaction returned from the log?

(:db/txInstant (d/entity db ?tx))

or you could extract it from the tx data itself

(probably not worth the trouble)

E.g.

(let [db (d/db conn)
      txs (-> (d/log conn) (d/tx-range 0 (inc (d/basis-t db))))]
  (->> txs
       (take 1)
       (map (fn [{:keys [t data] :as tx-info}]
              (conj tx-info (find (d/entity db (d/t->tx t)) :db/txInstant))))))

pretty much equivalent, but also lazy

What would the implementation of the first option look like? the ?tx make me think that goes in the query itself but I didn't think you could do all that inside a query (and also you mentioned not doing a query)

first option?

you said: (:db/txInstant (d/entity db ?tx)) or extract it from the tx data itself. I assumed the code sample was the implementation of the latter

no, the code sample is doing the first

so I gave you an example of reading the tx log and adding a :db/txInstant key to the returned maps, without doing a query

That is in this line: (conj tx-info (find (d/entity db (d/t->tx t)) :db/txInstant))))))

we're just grabbing the tx entity, then pulling out the :db/txInstant value and adding it to the tx-info map

Oh I see. The code segment makes perfect sense I just misunderstood which one that referred to

as a query it would look like this:

(d/q '[:find ?tx ?tx-data ?time
       :in $ ?log ?t1 ?t2
       :where
       [(tx-ids ?log ?t1 ?t2) [?tx ...]]
       [(tx-data ?log ?tx) ?tx-data]
       [?tx :db/txInstant ?time]
       ]
  (d/db conn) (d/log conn) 1000 1001)

but queries are not lazy

and this is a hack that doesn't use a db:

(d/q '[:find ?tx ?tx-data ?time
       :in ?log ?t1 ?t2
       :where
       [(tx-ids ?log ?t1 ?t2) [?tx ...]]
       [(tx-data ?log ?tx) ?tx-data]
             ;; 50 is the :db/txInstant id
       [(identity ?tx-data) [[?tx 50 ?time _ true]]]
       ]
  (d/log conn) 1000 1001)

here we look in the tx-data itself for the tx instant

(I wouldn't recommend this approach, but it's just to show what's possible)

a safer version needs to determine what the :db/txInstant attribute's eid is

Yep, that's what the query I got working looked like exactly. Is the only intrinsic advantage to not querying laziness?

it's presorted

I was just asking about the last query and then you fixed it 😅

query assembles a set

I really appreciate you taking the time to go over all of this for me.

so if you want to sort by time, the non-query version will already be sorted

the query version you will need to sort after

the query version in theory can make use of parallelism, not sure it matters here

honestly some things are really just easier to express without a query too

In my case the version without querying gives me a more manageable shape for the data anyways

when I just need to read an entire index segment or tx log segment I find the non-query approach to be faster, use less memory, and be more straightforward

it's only when I need to do actual pattern matching or walking across entities (i.e. real queries) that using tx-log in a query makes sense

I always have the history db as input too in those queries

probably a tx-log query that doesn't have a history db as input is a sign that it's probably not an ideal query--just map over tx-range

I think this is all starting to click, thanks!

Datomic 0.9.5561.56 is now available https://groups.google.com/d/topic/datomic/ZO4lW8wI2MI/discussion