Will you be recreating the ES index as a batch? I've done a lot of work on this problem but with Lucene and it may be easier to add all assertions and retractions, then filter via the es query itself.

Yes, I'll be recreating the ES index as a batch. I thought about that too, this will be a problem if I ever want to use the ES suggestions (because they lack good filtering capabilities). Other possibility is filtering by :added while traversing? But this goes back to my original question as I'm not sure what EAVT returns.

only the history database includes retractions

(d/history db)

if you never called d/history on a db it will only ever have assertions

Ok thanks! So if I transact and then retract an entity e, it won't be included in the EAVT index?

correct

(and easily verified by trying it)

Thank you. Yes, that was a lazy question, sorry.

Hi @UHJH8MG6S are you using datomic pro? What version? I believe we resolved this issue after clojure 1.10.1 release in the latest Datomic pro.

0.9.5951

Currently I’m on datomic pro 0.9.5786. will check out latset version

Thanks! bumping to 0.9.5951 fixed it :thumbsup::skin-tone-2:

if you know what you need ahead of time pull is likely to be faster, or at least can be made faster (whereas entity will always have a “should I prefetch this? will you need it?” problem)

there’s another advantage that pull gives you Real Maps and can do some key renaming for you, and you know for sure that IO is done

so you can isolate potentially blocking/latency-sensitive code to its own threads

(entity has unpredictable latency because there’s always a chance it has to perform blocking io)

thanks, we did some benchmarks and got results that match your description:

(crit/with-progress-reporting
    (crit/report-result
      (crit/quick-bench
        (->> (d/q '{:find  [[?e ...]]
                    :where [[?e :artist/name _]]}
               db)
             (mapv (comp #(select-keys % [:artist/name])
                         #(d/entity db %)))))))
                         
Evaluation count : 48 in 6 samples of 8 calls.
             Execution time mean : 12.594899 ms
    Execution time std-deviation : 1.845767 ms
   Execution time lower quantile : 10.651151 ms ( 2.5%)
   Execution time upper quantile : 14.538328 ms (97.5%)
                   Overhead used : 1.820422 ns


==============================================================


(crit/with-progress-reporting
    (crit/report-result
      (crit/quick-bench
        (->> (d/q '{:find  [[(pull ?e [:artist/name]) ...]]
                    :where [[?e :artist/name _]]}
               db)))))

Evaluation count : 36 in 6 samples of 6 calls.
             Execution time mean : 18.813521 ms
    Execution time std-deviation : 298.939459 µs
   Execution time lower quantile : 18.551782 ms ( 2.5%)
   Execution time upper quantile : 19.209279 ms (97.5%)
                   Overhead used : 1.820422 ns


==============================================================


(crit/with-progress-reporting
    (crit/report-result
      (crit/quick-bench
        (->> (d/q '{:find  [?e ?name]
                    :where [[?e :artist/name ?name]]}
               db)))))
Evaluation count : 300 in 6 samples of 50 calls.
             Execution time mean : 2.156460 ms
    Execution time std-deviation : 143.362504 µs
   Execution time lower quantile : 1.990183 ms ( 2.5%)
   Execution time upper quantile : 2.312776 ms (97.5%)
                   Overhead used : 1.820422 ns

also, it seems like using the datalog matching is much faster (10x) than both of the other options

this was run against the demo mbraiz database

just to be sure, new benchmarks with full tests:

(crit/with-progress-reporting
    (crit/report-result
      (crit/bench
        (->> (d/q '{:find  [[?e ...]]
                    :where [[?e :artist/name _]]}
               db)
             (mapv (comp #(select-keys % [:artist/name])
                         #(d/entity db %))))
        :verbose)))
        
Warming up for JIT optimisations 10000000000 ...
  compilation occurred before 1 iterations
  compilation occurred before 347 iterations
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
x86_64 Mac OS X 10.14.1 12 cpu(s)
Java HotSpot(TM) 64-Bit Server VM 25.181-b13
Evaluation count : 6420 in 60 samples of 107 calls.

      Execution time sample mean : 9.898978 ms
             Execution time mean : 9.895288 ms
Execution time sample std-deviation : 536.776513 µs
    Execution time std-deviation : 544.280358 µs
   Execution time lower quantile : 9.142990 ms ( 2.5%)
   Execution time upper quantile : 10.901935 ms (97.5%)
                   Overhead used : 1.820422 ns


==============================================================


(crit/with-progress-reporting
    (crit/report-result
      (crit/bench
        (->> (d/q '{:find  [[(pull ?e [:artist/name]) ...]]
                    :where [[?e :artist/name _]]}
               db))
        :verbose)))
        
Warming up for JIT optimisations 10000000000 ...
  compilation occurred before 103 iterations
  compilation occurred before 307 iterations
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
x86_64 Mac OS X 10.14.1 12 cpu(s)
Java HotSpot(TM) 64-Bit Server VM 25.181-b13
Evaluation count : 3300 in 60 samples of 55 calls.

      Execution time sample mean : 18.169708 ms
             Execution time mean : 18.174275 ms
Execution time sample std-deviation : 516.347064 µs
    Execution time std-deviation : 522.849633 µs
   Execution time lower quantile : 17.626452 ms ( 2.5%)
   Execution time upper quantile : 19.723233 ms (97.5%)
                   Overhead used : 1.820422 ns

Found 6 outliers in 60 samples (10.0000 %)
	low-severe	 2 (3.3333 %)
	low-mild	 4 (6.6667 %)
 Variance from outliers : 15.7926 % Variance is moderately inflated by outliers


==============================================================


(crit/with-progress-reporting
    (crit/report-result
      (crit/bench
        (->> (d/q '{:find  [?e ?name]
                    :where [[?e :artist/name ?name]]}
               db))
        :verbose)))
        
Warming up for JIT optimisations 10000000000 ...
  compilation occurred before 5846 iterations
Estimating execution count ...
Sampling ...
Final GC...
Checking GC...
Finding outliers ...
Bootstrapping ...
Checking outlier significance
x86_64 Mac OS X 10.14.1 12 cpu(s)
Java HotSpot(TM) 64-Bit Server VM 25.181-b13
Evaluation count : 29940 in 60 samples of 499 calls.

      Execution time sample mean : 1.945301 ms
             Execution time mean : 1.945398 ms
Execution time sample std-deviation : 32.284694 µs
    Execution time std-deviation : 32.687682 µs
   Execution time lower quantile : 1.908870 ms ( 2.5%)
   Execution time upper quantile : 2.033389 ms (97.5%)
                   Overhead used : 1.820422 ns

Found 3 outliers in 60 samples (5.0000 %)
	low-severe	 3 (5.0000 %)
 Variance from outliers : 6.2567 % Variance is slightly inflated by outliers

these tests seem to show that I am wrong? d/entity appears 2x faster to me

than pull in a query expression

did you try d/pull or d/pull-many after the query?

maybe there’s a large fixed cost to compiling the pull expression

and yes, pulling directly from the query itself (no entity or pull) is always going to be much faster, but the shape is often wrong and it can’t represent nils

wilker, did you try extracting the pull part from your query, into its own function?

quite interested in your findings

I didn't ran the tests with the pull out yet, have to test that one in some other time