I don't think so, at least not with queries. I think queries should never be concerned with the value of :crux.db/id at all, at least because the way facts are distributed across documents has operational consequences that should be transparent to the application

unfortunately queries still do need to care about the question of "is this attribute in the same document as that attribute?" but I don't think you ever need to worry about "what is the value of :crux.db/id in a document?

i was concerned more with puts than queries: "wups, an ordering bug overwrote {:friendship [2 1]} when it was supposed to be [1 2] ? maybe that's less likely than i'm making it seem in my head.

yeah then you would have duplicate entities with the same attributes, which I think is another dangerous problem in general, but I think in practice you'd have the document generator as a function rather than typing it out by hand anyway. It could be a little more likely to mess up typing it out by hand with the vector stuff but typos are always a possibility

hehe... i wasn't imagining typing attributes out by hand. i just don't trust any of the code i write. 😉

I think the only care is actually in the kv stores which want to know how many bits to store. Unfortunately I have a pair of longs instead of a single long, so I can't fit that into a single long. I'll probably do some truncation for now and assume that things won't go too badly... Still interested in input from the core team though.

Yep, you're on the right lines there - we certainly care that it's ascending. The KV stores do assume it's 8 bytes - in theory this isn't a hard requirement but will likely involve a decent amount of changes throughout the indexer and query engine.

@U050V1N74 Oh, I thought it might not be ascending. Hmph. I've done some bit fiddling to get the 2 longs into a single long (under the assumption that 2070 is a long way away and someone will have fixed this by then...). But (I think) that will break the long from being ascending, even though there's linearity when I convert it back.

Yeah, it definitely breaks ascending under the current impl. I can probably do some bit shifts to solve that, but I'd be curious to know if I need to :)

I cannot do bit shifting to solve that, that isn't how longs work 😀 So yeah, if they don't need to be ascending inside of Crux, that'd be helpful.

You might not see issues with that, necessarily - the index in question is ordered first by transaction-time, then by transaction-id (for historical reasons, although I wouldn't rely on that - we're considering solely using transaction-id there) - so if you don't have any transaction-time millisecond ties the ordering of the transaction-ids won't be noticeable in that area. Another place you will notice it is in your implementation of open-tx-log which takes an after-tx-id - this doesn't require it to be ascending, strictly speaking, only that it's totally ordered I can't think of others, off-hand :thinking_face:

entity-history may well sort internally just by transaction-id (on the assumption that sorting by tx-time and tx-id results in the same ordering)

certainly an interesting use-case, good to challenge our assumptions, and a possible driver of further unbundling 🙂

open-tx-log is fine, as I can take the long and split it back into consituent parts, and those are ascending (I'm taking the current time millis (essentially) and a tie-breaker which are both normally longs, and assuming the date won't get too high, and neither will the number of tie breakers. Then I'm writing the short into the 16 bits left spare by the assumption that the date won't get too high.

I think it should always be incrementing when there's a tie. So it's possibly something I should get away with. Although I'm still not seeing any results from my query :D

blugh. I'd done ::tx/id instead of ::tx/tx-id

Still no luck though :) Must be more bugs

I got it! Woohoo

I also did tx-event instead of tx-events. Basically: pay attention to details.

At one point there actually was a document api that could return documents based their content-hash, but we took the decision that it was probably a bad idea to expose applications to Crux-internal hashing. There are currently no public APIs to retrieve the doc in a query or otherwise. Assuming you are already pulling the content-hash from the entity-history API, is it not sufficient to store the coordinate of [eid valid-time tx-time] and use that as a key?

@U899JBRPF the content-hash actually comes from a request in this case, (like "/:some-resource-id/:content-hash"). I might be better off adding my own content-hash then. I worked around it for now by fetching the entity history, group-by :content-hash then get it from there. Thanks for the answer!

Does it have any performance issue, adding each fact about an entity in its own document? Also, what does PII stands for?

definitely, since you'll have to do more joins in the query if you wanted the document together. The upside is you can reduce duplication if you're only updating that fact, since you can only submit whole documents, and maybe save a decent amount of space if your documents are big. PII is personally identifiable information, which as anyone who's worked in a high-compliance environment will likely tell you, is like toxic waste to handle

or, as anyone who's worked in a low-compliance environment will tell you, actually feels like swimming in toxic waste.