Fork me on GitHub
#datomic
<
2024-02-08
>
dazld09:02:11

Valcache seems to take a very long time to complete its startup - on my local, it’s around 3 minutes. I’m wondering if it’s a quirk of my local setup, or a known issue?

2024-02-08 09:29:50 INFO  d.valcache-direct {:event :valcache-direct/start, :root "/Users/dazld/valcache/", :pid 12712, :tid 21}
2024-02-08 09:29:50 INFO  d.valcache {:event :valcache/direct-init, :datomic.valcache/path "/Users/dazld/valcache/", :datomic.valcache/eviction-threshold-mb 4000, :pid 12712, :tid 21}
2024-02-08 09:29:51 INFO  d.async {:event :daemon/thread-started, :name "valcache-eviction-loop-1", :pid 12712, :tid 54}
2024-02-08 09:29:51 INFO  d.valcache {:event "ValcacheEvictPlan", :path "/Users/dazld/valcache/", :threshold-mb 4000, :interval-secs 10, :file-window 10000, :pid 12712, :tid 54}
2024-02-08 09:32:36 INFO  d.c.datomic :connected
MacOS, M1 Pro with 16GB.

dazld09:02:03

our DB isn’t very big - and it doesn’t have so many updates day to day

dazld10:02:40

I suppose it’s just how fast the upstream db can provide the new blocks..

favila17:02:55

We use a pattern for filtering queries (built in code by user input). To make this tractable we have a pattern of a rule per filter, e.g.

[(filter-entity-by-rel1 [?test-value ?entity-id])
     [(!= :none ?test-value)]
     [?entity-id :entity/rel1 ?test-value]]

    [(filter-entity-by-rel1 [?test-value ?entity-id])
     [(= :none ?test-value)]
     (not [?entity-id :entity/rel1])]
In this pattern ?test-value may be either an entity id or the :none sentinel (which is not an ident in the database). I am finding cases where the :none guard clause seems to be moved around unpredictably so that evaluates after the pattern clause, throwing an exception. This filter in a query by itself runs fine, but some combination with other clauses in the query can (reliably, but unpredictably) cause it to throw java.lang.IllegalArgumentException: Cannot resolve key: :none , which looks a lot like it evaluated [?entity-id :entity/rel1 ?test-value] first. Is there a reliable way to implement this pattern short of using a negative integer as a none sentinel?

Joe Lane18:02:18

Hate to ask, but any chance you have a minimal repro ? Would go a long way in speeding up the analysis.

favila21:02:45

Lucky for you I now can’t make a minimal which doesn’t fail

1
favila21:02:04

Only workaround I could find was to rebind

favila21:02:16

[(filter-entity-by-rel1 [?test-value ?entity-id])
 [(!= :none ?test-value)]
 [(identity ?test-value) ?tv2]
 [?entity-id :entity/rel1 ?tv2]]

favila21:02:08

Interestingly, if I destructure the test-value, it does work as expected

favila21:02:46

(d/query {:query '[:find (pull ?e [:entity/id])
                   :in $ % [?filter ...] ;; the only change
                   :where
                   [?e :entity/id]
                   (filter-entity-by-rel1 ?filter ?e)]
          :args  [(d/db c)
                  '[[(filter-entity-by-rel1 ?test-value ?entity-id)
                     [(!= :none ?test-value)]
                     [?entity-id :entity/rel1 ?test-value]]
                    [(filter-entity-by-rel1 ?test-value ?entity-id)
                     [(= :none ?test-value)]
                     (not [?entity-id :entity/rel1])]]
                  [:none]] ;; now wrapped in a vector
})

favila22:02:01

The Expect in that file is wrong, should be [[#:entity{:id 1}]]

favila13:02:18

Any advice here? Are guards something that is supposed to work, or is reordering as preds a fact of life that isn’t going to change? If the latter, what do you suggest for patterns like this that are entity + sentinel? Wrap in a tuple to tag the variant?

Joe Lane15:02:16

I haven’t had the bandwidth to look into this yet, but the exception appears to be trying to resolve your :none keyword as an entity/ident

favila16:02:12

Yes, correct, I assume that is because it became a :pred and evaluates after the pattern clause now

favila16:02:28

but the point of the predicate is to guard against evaluating the pattern clause

favila16:02:30

I use guard predicates like this a lot in rules with multiple implementations, and it had been reliable, but now I’m wondering if it was meant to be reliable and I should do something else.

favila16:02:47

i.e. maybe it was just working by accident

Joe Lane16:02:49

I’ll try to find a time to look at this soon. Again, sorry for the delay.

Keith16:02:17

Hey Francis, A few things seem to be going on here. For one, the the query engine is attempting to resolve ?test-value in the clause [?entity-id :entity/rel1 ?test-value] as if it's an ident because :entity/rel1 is a :db.type/ref. The next question I would ask is: why are we reaching that clause if [(!= :none ?test-value)] returns false? This actually appears to be related to how datalog rules are processed - a clause that returns an empty result set doesn't cause the rule to short-circuit. You can see this behavior with query-stats. Notice how :rows-out is 0 for all clauses inside of this rule, yet all of the clauses are still present and have been processed. I imagine this normally isn't noticeable when your clauses don't result in an error, but in the case you presented, the failure to resolve :none as an ident is making this behavior apparent. Does that make sense?

favila17:02:10

Sure, but why does it sometimes work and sometimes not? I made a trivial change (adding destructuring to the input) and it behaves as expected.

favila17:02:08

Really I’m looking for guidance on how one ought to structure rules that have this sentinel-or-entity-id pattern

favila17:02:10

I don’t know if this is something I shouldn’t expect to work reliably ever (i.e. a binding mentioned in a pattern clause can only safely be an entity ref), or if this is a bug.

Keith18:02:02

The behavior of a collection binding and how that doesn't manifest the same error is very interesting. I'll have to get back to you with more, but wanted to respond so as to not leave you hanging in the dark 🙂

Mateusz Mazurczak17:02:10

For datomic pro (with transactor deployed with app, connecting to postgresqldb) I'm getting error AMQ219007 when I'm starting second app (with transactor) trying to connect to the same datomic-db (with the same name and that transactor connects to the same postgresqldb) How can I solve this, to be able to have two transactors at the same time? Is there some configuration I'm missing?

favila17:02:21

transactor host and alt-host values from the transactor.properties file are written into storage. They need to be different values for each transactor

favila17:02:07

only one transactor will be active at a time; the second will be a hot spare (https://docs.datomic.com/pro/operation/ha.html)

favila17:02:30

The transactors “elect” an active one by racing writes to storage to claim it.

favila17:02:45

peers find and connect to transactors by reading the same host and alt-host values from storage

favila17:02:23

so, everything depends on host and alt-host values: make sure they are different for each transactor, and make sure peers and transactors can all resolve those values, or at least one of them, to a transactor instance.

favila17:02:55

If you are trying to isolate peers to different sets of transactors, know that that is not possible

favila17:02:03

peers don’t get to choose the active transactor

Mateusz Mazurczak15:02:51

@U09R86PA4 Thank you very much! It is all clear now

uwo17:02:06

I just want to double check that it's not possible to pass in a separate database, other than the implicit db, to a pull inside the :find specification. Judging by the grammar (https://docs.datomic.com/pro/query/query.html#grammar) this is not a possibility. In my case, the where clause uses a filtered-db and I'd like the pull expression to use an unfiltered db. No biggie if this doesn't work of course -- I'll just just pull-many after the fact

favila17:02:58

I think it’s possible but not documented: :find (pull $ ?e pattern)

favila17:02:26

I would double-check that it’s actually using the correct db and not just GIGO

favila17:02:38

but that for sure doesn’t throw an error

uwo17:02:43

Yeah! Thanks Favila -- right after I asked I figured I might as well give it a shot, and like you said, no errors are coming out. Gonna see if it works as expected

favila17:02:19

I was using this syntax for quite a while before I realized it wasn’t supported--so I discovered it by accident

😂 1
favila17:02:51

I was just aping d/pull (and btw, super annoying that pattern and entity id are switched, that trips me up constantly)

uwo17:02:26

exactly! I swap those args literally every time when I write pull inside a :find