(time
   (with-open [snap (xtdb.db/open-index-snapshot (:index-store (xdb)))]
     (let [attr-buffer (xtdb.memory/copy-to-unpooled-buffer (xtdb.codec/->id-buffer :my-string-attribute))]
       (mapv xtdb.codec/decode-value-buffer (xtdb.db/av snap attr-buffer "")))))

(like under 1ms)

compare with

(time (xt/q (xdb) '[:find v :where [_ :my-string-attribute v]]))

Accessing av like that won't do any temporal filtering, because it's across all time, and so wouldn't give repeatable consistency either. But it's certainly a neat trick and useful for cases where those things aren't a problem 🙂

The stats index tracks a hyperloglog distinct count approximation per attribute, which would be even quicker if you only want an (approx) count...but also has the same atemporal characteristics

I don't think there's a better 'correct' solution than what you came up with, though you could also model this manually or create a custom secondary index if needed (at some cost of write performance and extra storage)

yes, I adding extra docs for each distinct value at write time would be one solution.

I was thinking about a case where I would have something like “tags” where a user can write a new one, but also select from the previously added ones {:xt/id "foo" :tags ["some" "tags" "here"]} so there would need to be a quick way to get all the tags values quickly

my other idea was to get it from lucene index, but it doesn’t allow empty query string to return all… If I use raw lucene API, I guess it will have the same temporal filtering problem as av

FWIW, the Lucene module does support configuration to allow ~empty (leading) wildcard lookups, e.g. https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/modules/lucene/test/xtdb/lucene/extension_test.clj#L189 but I'm not sure if it would be any faster :thinking_face:

neat, I’ll try that as well just for completeness sake

the Lucene Java API interop seems pretty involved 😅

Yeah...Lucene is not a good benchmark for beginner friendly full-text search in 2022 🙂

ok, I got 172ms for that solution, so the hacky index version is the fastest so far

but the .setAllowLeadingWildcard is a good thing to know, as it is useful for providing a good typeahead from a set of tags that works with compound words

eg. if user types “ball” it would find both “football” and “ballet” if you surround it with earmuffs

🙂 cool, glad that was a useful pointer

1ms sounds hard to beat

can this leading wildcard be configured to the text-search available in regular queries?

sadly no, you have to register your own custom tatut-text-search function, like shown in that test file

I guess consistency is not a massive concern if this lookup is only driving a UI selector component (you can always check the selected value is visible as-of whenever before attempting any writes, like a manual bitemp filter)

I would expect tags to only ever accrete in this case

and it isn’t dangerous to show something that isn’t actually used currently