Hey, I wouldn't recommend storing that many elements in a list in a single document - in order to update that list, you'd need to supply the whole list again

I wonder whether the performance gains that you're seeing are in fact caused by Crux caching that value, and then Clojure caching the count of the collection 🙂

Restarting the repl+node should clear caches?

it should, yep

If you've done that and it's still that much faster, that'll require a bit more thought :thinking_face:

; after reset
"Elapsed time: 88.236937 msecs"
131077
; second eval
"Elapsed time: 12.391279 msecs"
131077
; the original query after few evals
"Elapsed time: 2674.383644 msecs"
#{[131077]}

In terms of what Crux will be doing differently in the two cases, the main difference will be that when there's only one entity in play, it'll only have to do one temporal resolution rather than a million - but in that case we're talking (roughly) 2 million operations down to 1 million + 1, which I doubt would yield orders of magnitude different performance

Million was a ballpark guess. Hmm how can I count all documents?

(crux/attribute-stats node) will give you an estimate

that's across all documents across all time, it doesn't tell you how many documents are visible now (although it doesn't seem it'll be too different in your test case)

:crux.db/id 480587

Shouldnt be too different

I figured that in the original query, you'd have to check those 480k documents for :membership/type :member, and in the second query you'd only need to check 1 document for one value (the vector), which would be the source if this drastic difference

I think you're inserting a list there with map, right? crux would actually break down a vector into individual EAV triples in the index, but iirc it keeps a list whole

good spot 👏 yep, this is what's happening

If you shouldn't put too many elements in a list in a single document, then can you recommend another way of speeding up attribute based queries with aggregations where you are counting thousand of documents? I seem to get about 1s query time / 30k documents when searching by attribute with my dataset. Or should I maybe be looking at a materialized view?

I'll try out lucene and see what happends with text fields

For me, basically 95%+ of queries are get an entity with this id and a graph of data relating to it. These are super fast with crux. 5% or less of the queries are "find some type of entity with these values in these these attributes". If lucene/something helps with these "search from all docs" types of queries, then I can finally sneak in Clojure at work.

I suspect you might get a small performance boost by performing the count using a Datalog aggregate, as in :find [(count ?e)] However a user-space materialized view of some kind (either in-memory, or persisted alongside the data in your transactions) will undoubtedly give the best performance. Edit: I should add that implementing a materialized view and keeping it consistent and up-to-date is not a trivial undertaking though. e.g. transaction functions can help but then you will be trading off against write-speed, and the temporal semantics can get tricky

Even the Crux built-in count will still necessarily have to find all the entities with that type, but yes, it will do less work than getting the entities back and counting in user-space.

@UH9091BLY: your requirement to 'search from all docs' may well not need to scan all the documents every time - if you have any kind of filter available (say, 'created in the last week', or 'name = "John"' - your example to 'get an entity with this id' is obv the absolute best case 🙂) this will significantly limit the amount of data Crux has to search

You may well also not need Lucene, unless you're looking to search within a string field, fuzzy match, or any of the other specific benefits of Lucene's full-text search.

Crux also queries lazily - if you only need to return a page of members at a time, say, you should find that this is relatively fast

Thanks this clears things up. For some reason I assumed there wouldn't be any way to know which document had which attribute. With lucene, checkpoints and limits I think crux will work way better than what we have now

Great to hear - let us know if you have any further questions or issues 🙂

> I think crux will work way better than what we have now @UH9091BLY Out of curiosity, do you mind telling us loosely what you have now? (Feel free to turn down that request if it's none of my business, though. 🙂) I'm always curious about the legacy system in the case of folks migrating a production system to Crux.

@U01AVNG2XNF Wouldn't call it legacy yet but we're using Apollo Server + MySQL mostly.