Hey @U0D8J9T5K - thanks for letting us know! I'm afraid that blog is out of date, we now no longer return any details about evicted entities, even to confirm that they ever existed. I'll ensure it gets updated 🙂

Hi 🙂 it's not a bug as such, though it does deserve clearer documentation. What's happening here is that 19 is treated as a Long by the query engine whereas the :density values are ingested and indexed as Doubles, and the engine doesn't attempt to do any coercion. You can see here that Longs & Doubles are stored in distinct index spaces: https://github.com/juxt/crux/blob/master/crux-core/src/crux/codec.clj#L79-L80 I'll update the docs with a note of caution - thanks for pointing it out!

thanks for the explanation

I should add that you could use [(clojure.core/>= d 19)] to get the coercive behaviour, although it will result in scanning, rather than efficient range filtering

Hey @U6M9V2735, thanks for getting in touch 🙂 At first glance it doesn't seem related to 1238 I'm afraid. Could you share a small repro? (Feel free to DM if you wouldn't be comfortable sharing it publicly)

I’ll try to isolate the issue and share a snippet.

As for fixing my current trouble, I’m trying to rebuild the index. Though not sure if that will even work? Is it the case that the index stores keys and values of the docs?

As a side note, I get this error when trying to rebuild the index.

Execution error (IllegalArgumentException) at crux.tx/index-docs (tx.clj:235).
Missing required attribute :crux.db/id: ({:crux.db.fn/failed? true, :crux.db.fn/exception java.lang.NullPointerException, :crux.db.fn/message nil, :crux.db.fn/ex-data nil})

Hi @U6M9V2735 I have a hunch that something is wrong with your document store. Please could you paste the config map passed to start-node?

Rebuilding (after deleting the index store) should work fine if the tx-log and document store are working properly

Thanks @U899JBRPF. Here is the config. I'm trying to recreate the issue on a fresh setup, not able to yet.

{:crux/index-store
  {:kv-store
   {:crux/module crux.rocksdb/->kv-store
    :db-dir "index-data"}}
  :crux/document-store
  {:kv-store
   {:crux/module crux.rocksdb/->kv-store
    :db-dir "document-data"}}
  :crux/tx-log
  {:kv-store
   {:crux/module crux.rocksdb/->kv-store
    :db-dir  "tx-log-data"}}}

Hmm, that looks okay, the document store is probably not the problem then. The next most likely candidate would be the use of transaction functions (maybe you've found an edge-case?), but I'm not sure what else to suggest for debugging at your end :thinking_face:

Roger that. Would you ever expect the transaction log to skip a crux.tx/tx-id? I'm trying to look through the tx-log to find the txs causing the anomaly and looks like there are a few skipped tx-ids around the anomaly

Ah, interesting. Looking at the code for the "kv-tx-log" I think it should only ever be inc-reasing smoothly: https://github.com/juxt/crux/blob/master/crux-core/src/crux/kv/tx_log.clj#L51 Did you start/stop the node at all between transactions?

Yeah, very likely that I restarted the node. I'm trying to remember as best as I can the steps to get into that state, but there were several. It is likely I even restarted the JVM.

That's fair enough. Well it's probably a good idea to make sure writes always hit the disk (just in case) by adding :sync true like so:

{:crux/index-store
  {:kv-store
   {:crux/module crux.rocksdb/->kv-store
    :db-dir "index-data"}}
  :crux/document-store
  {:kv-store
   {:crux/module crux.rocksdb/->kv-store
    :db-dir "document-data"
    :sync true}}
  :crux/tx-log
  {:kv-store
   {:crux/module crux.rocksdb/->kv-store
    :db-dir  "tx-log-data"
    :sync true}}}

Got it, thanks. I'm about out of time for troubleshooting this so I'll have to move on. Though if I get to the point of reproducing it, I'll let you know.