Clojurians Log v2

I expect either a seq of maps or an exception. Instead, I got:

[[:user/email "foo@example.com"]]

Produced with [com.datomic/peer "1.0.6726"]. Let me know if you want more details.

I was also surprised that find-scalar queries like

[:find ?email .
 :keys user/email
 :where [_ :user/email ?email]]

(def conn ,,,)

@(d/transact conn [{:db/ident :test/id
                    :db/valueType :db.type/string
                    :db/cardinality :db.cardinality/one
                    :db/unique :db.unique/identity}

                   {:db/ident :test/value
                    :db/valueType :db.type/string
                    :db/cardinality :db.cardinality/one}])

(def nil-db (d/db conn))
(def nil-t (d/basis-t nil-db))

@(d/transact conn [{:test/id "id" :test/value "foo"}])

(def foo-db (d/db conn))
(def foo-t (d/basis-t foo-db))

@(d/transact conn [{:test/id "id" :test/value "bar"}])

(def bar-db (d/db conn))
(def bar-t (d/basis-t bar-db))

(:test/value (d/entity bar-db [:test/id "id"])) ;; => "bar"
(:test/value (d/entity foo-db [:test/id "id"])) ;; => "foo"
(:test/value (d/entity nil-db [:test/id "id"])) ;; => nil

(def as-of-foo-db (d/as-of (d/db conn) foo-t))
(:test/value (d/entity as-of-foo-db [:test/id "id"])) ;; => "foo"

(def with-baz-db (:db-after (d/with (d/db conn) [{:test/id "id" :test/value "baz"}])))
(:test/value (d/entity with-baz-db [:test/id "id"])) ;; => "baz"

(def with-baz-as-of-foo-db (:db-after (d/with as-of-foo-db [{:test/id "id" :test/value "baz"}])))

(:test/value (d/entity with-baz-as-of-foo-db [:test/id "id"]))
;; => "foo", expecting "baz"

In your mind, why does the db given to d/with need to be filtered?

That's not what's going on here. I'm using as-of much further out to reproduce an earlier result. "What did this look like back in time?" However, since some function downstream is now using with , this pattern breaks. My code is no longer able to answer questions about databases back in time.

I'm trying to figure out your overall intention re: d/with with filtered dbs. D/with (like d/transact) ignores filters, because the consistency properties transactions guarantee (eg unique values, implicit retractions) are about the unfiltered db value and tx log. Eg what if you add a unique value at t=5, as-of to 3, then add the same value to a different entity? D/as-of is not a fork, just a filter. https://docs.datomic.com/reference/filters.html#as-of-not-branch

Here's my overall intention: I would like for a function f of db to be able to also answer questions about the database as a value back in time. What was (f db) yesterday, or the day before? This has been working wonderfully with Datomic, but breaks the moment f at any point uses with to come up with an answer.

What questions are you asking that require with?

That would be a big detour of an explanation, and not very relevant to the question at hand, since at the end we would still be at a place where d/with could have been useful, but ends up being a foot gun if you also would like to use as-of to answer questions about previous versions of the database.

Here's what I gather so far: Since with and as-of does not compose, and fails silently / in interesting ways, one should probably not rely on both of these Datomic features. In my case I ended up with an infinite loop, a couple kill -9 and a few lost hours of debugging - others might experience different outcomes.

I'm still not sure what you expected it to do? Did you expect as-of to be a branch?

Yes, that's probably a good estimation of my mental model. But that's also not what happened in practice. I have feature A, which was implemented using with . And I have feature B which was implemented using as-of . And when these intersected, program go boom.

The boom was because…?

The boom was due to a loop expecting the db to change (using with) such that it would eventually exit the loop. When the db was as-of 'ed (entering the loop from a rarely used branch of code), with did nothing, and the loop kept on going.

The db-after also had as-of set?

Yes, the :db-after keeps the filter.

Ok well that's something we could consider changing, but it's still not going to give you the behavior you want

I agree. That would just fail in a different manner. So the problem boils down to one of mental model / expectations about as-of being a branch. I don't think just removing the filter from the :db-after from d/with would do anyone any favors.

What would have saved me a couple hours, at least, was an exception when the "nonsensical" operation of using with where all the datoms are filtered out happened.

To be clear the d/with didn't “do nothing”, it did add a transaction

But to the db basis-t not the as-of t

Datomic doesn't let you change the past

Yes, I understand - but in the context of "database as a value", then nothing changed.

The database value includes everything up to the basis-t

Only if you don't consider the filter a part of the database value.

No, even so

Filters influence what datoms are visible, not what are present

Yes, I'm not confused as to those details, but from a usage perspective, that is just a matter of semantics / implementation details. Datoms that are not visible to me, might as well not be present.

Arbitrary filtered dbs can't be a sound basis for a transaction because there are no consistency guarantees for the visible datom set

If this were some other filter, such as history or since, I don't think you would have been surprised. But as of by itself is superficially similar to a branch (changing basis-t), thus confusion

But it's only similar from a query perspective, not accumulation

Yes, true, I can see how the implementation of as-of and the implementation of with collide. I guess what I really needed in this case was a branch. Consider my report a vote for such a feature (albeit unlikely), and perhaps one for an exception when using with on an as-of 'ed database.

That definitely looks broken. I re-created on my end. Will investigate further.