datomic 2020-02-16 | Slack Archive

odie08:02:05

Hi all, One of the queries I’m trying to run seemed “slow”, at around 2+ seconds on my local machine. The query is just trying to match on an exact value for an attribute. It looks something like this:

'[:find [(pull ?e [*]) ...]
	:in $ ?target-val
	:where [?e :some-attr ?target-val]]

It turns out, there are ~10M entities with this particular attribute. I then tried to speed this up by asking for the attribute to be indexed. The attribute was updated to look like:

{:db/ident :some-attr
 :db/valueType :db.type/string
 :db/cardinality :db.cardinality/one
 :db/index true}

However, the query speed did not change. I then tried looking things up directly through the AVET index like so.

(d/datoms db :avet :some-attr "12345")

This results in an error. Saying the attribute isn’t indexed.

Execution error (Exceptions$IllegalArgumentExceptionInfo) at datomic.error/arg (error.clj:79).
:db.error/attribute-not-indexed attribute: :some-attr is not indexed

I also tried running

(d/request-index (db/get-conn)) ;=> true

and

(deref (d/sync-index conn (d/basis-t (d/db conn)))) ;=> blocks forever

I’m pretty lost on how to go about getting the index built. Is there something obvious I’m missing here? I’m running 0.9.6014 btw.

favila14:02:24

You are on the right track but indexing isn’t instant

favila14:02:07

transactor logs and metrics will show if it’s actively indexing

favila14:02:16

Use a basis t exactly corresponding to the t when you issued request-index (or a little older if you are unsure) to your d/sync-index call

favila14:02:50

If it’s still blocking, then you are definitely still indexing

favila14:02:02

You just need to wait for it to finish

favila14:02:20

And you can confirm by looking at tx logs

odie06:02:08

@U09R86PA4 I left everything running over night. The index still isn’t available. Digging a bit in the log, I found that whenever I ran

(d/request-index (db/get-conn))

The following lines would soon show up in the log file:

2020-02-17 14:19:10.773 WARN  default    o.a.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue admin.request5e4a305e-a0cd-4a28-a539-6e209b4d53ec in AddressSettings
2020-02-17 14:19:10.778 WARN  default    o.a.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue admin.request5e4a305e-a0cd-4a28-a539-6e209b4d53ec in AddressSettings
2020-02-17 14:19:10.860 WARN  default    o.a.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue admin.response5e4a305e-bd13-4fe6-905e-d3a0a0ace172 in AddressSettings
2020-02-17 14:19:10.860 WARN  default    o.a.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue admin.response5e4a305e-bd13-4fe6-905e-d3a0a0ace172 in AddressSettings
2020-02-17 14:19:10.909 INFO  default    datomic.update - {:event :transactor/admin-command, :cmd :request-index, :arg "stock-insight-d1b785f9-4994-485c-960f-45cc94ebced8", :result {:queued "stock-insight-d1b785f9-4994-485c-960f-45cc94ebced8"}, :pid 97181, :tid 97}
2020-02-17 14:19:11.328 INFO  default    datomic.update - {:index/requested-up-to-t 25104804, :pid 97181, :tid 54}

odie06:02:56

I’m guessing the last line means all the index has been brought up to t:25104804.

odie06:02:34

Indexing on said field was enabled at t:25104802. So I guess that means the transactor thinks all index as been brought up to date?

odie06:02:32

However, trying to use the AVET index fails in the same way.

odie06:02:03

Would those activemq warnings be some indication that something isn’t working the way it is supposed to?

odie06:02:52

Also, I noticed that I was starting the transactor with java 11. I’ve since switched to running it with java 8. Could that have messed something up?

favila12:02:51

The last line means the index has been requested up to that T, not that it is finished

favila12:02:19

Try sync-index with that t. If it blocks, you are not finished

favila12:02:09

Cognitect says they support java 11 :man-shrugging:

favila12:02:30

I think the activemq warnings are red herrings

daniel.spaniel11:02:59

@steveb8n, that did not work .. seems like

java.util.regex.Pattern

daniel.spaniel11:02:15

is not allowed in the :allow section of the ion-config.edn file

favila14:02:50

Could this be a serialization issue? Try using a string for the re and constructing a pattern early in the query with re-pattern

steveb8n21:02:00

In mine I have just the fn from the namespaces I need e.g. clojure.string/starts-with?

steveb8n21:02:27

since Pattern in java interop, you might create your own fn with Pattern inside and “allow” that instead

Daouda23:02:16

Hey folks, Let say I have an entity with an attr called :entity/hash and more than one entity may share de same hash value . Now I want to perform a query where I will pass a list of hash hash1 hash2 hash3 and the query will return a tuple of hash count-of-entity-with-the-same-hash like this: [hash1 3 hash2 5 hash3 80] or {hash1 3 hash2 5 hash3 80} Don’t want to perfom that at application level. I want the query to give me back that result. Is it possible and how can I achieve that? Snippet code will be very welcome 😄

pithyless06:02:11

Unless I misunderstood the question, this sounds like an aggregate count query: https://docs.datomic.com/cloud/query/query-data-reference.html#built-in-aggregates

[:find ?hash (count ?hash)
 :in [?hash ...]
 :with ?entity
 :where [?entity :entity/hash ?hash]]

Daouda23:02:48

actually you go it right, thank you very much 😄

2020-02-16

Channels