Fork me on GitHub
#xtdb
<
2021-09-10
>
rschmukler18:09:25

Hey all. Just got an interesting error while putting some data into a xtdb instance (well, technically crux 1.17) where I get the following message:

Exception in thread "crux-polling-tx-consumer" java.lang.IllegalStateException: missing docs:
Followed by a ton of doc ID’s. Anybody seen it before? Is there anything that I should do besides restart the node?

refset18:09:07

Hey @UEC8W94AE ! Which tx-log & doc-store combo is this?

refset18:09:49

reports of S3 doc-store flakiness lead us to make some changes in 1.18.1, which should help if you are able to upgrade https://github.com/xtdb/xtdb/releases/tag/1.18.1

rschmukler18:09:42

Hey sorry on the delay. It’s kafka for both, w/ a rocks local document store on top of the document store. Restarting the node fixed it

🙂 2
rschmukler18:09:27

However it did block processing the tx log until I restarted it. Stuck on 1.17 until I get time to update crux-geo to 1.18

rschmukler19:09:54

But I did have multiple threads going so the fix looks promising 🙂

👍 2
refset19:09:24

Ah, cool, well hopefully your upgrade to crux-geo isn't too rough when you get to it 🤞 Blocking the tx processing is an intentional consistency safety feature, if a bit crude.

lgessler21:09:51

hi--about a year ago I posted a deftx macro that tried to make transaction functions more convenient to work with. It was half-baked then, but I've gotten back to it and it's now about 3/5 baked, or at least useful enough for me to start using throughout my codebase. Sharing it here in case it's useful or interesting for others: https://gist.github.com/lgessler/19f24dc89eb0a2a1a6a77d96dc1325b2

👍 6
refset22:09:39

Hey, I remember! But it's a very hazy memory 🙂 I am definitely going give this a go tomorrow. Thanks for sharing 🙏

🙂 2
lgessler01:09:32

I should ask as I'm posting this--for anyone else using tx functions heavily, what if anything have you been doing to make things more ergonomic?

lgessler06:09:03

and re: the macro i'll say the jankiest part of it is probably how i resolve symbols--writing fq symbols in tx fns is kind of a drag so i wanted to try to solve that so you can just write code how you normally do, and the machinery of a syntax quote is just the thing for it, but i couldn't figure out how to use a syntax quote in that macro so i have to give a bad impl of it that probably has many subtle bugs, a few of which i can already think of

rschmukler22:09:38

Is there a faster way to get distinct values for a given attribute from an index directly somehow? Specifically a scan over AVE perhaps if it exists?

(q *node*
 '{:find    [?vals]
   :where   [[_ :my-attr ?v]
            [(distinct ?v) ?vals]]})
The above times out with ~495K records coming from attribute-stats

refset23:09:29

Hmm, try using the distinct aggregate, i.e. :find [(distinct ?vals)] instead. The clojure.core/distinct predicate function you used will, I think, be trying to process the ?v relation all in one go, whereas the aggregate is lazy/streaming

refset23:09:49

It still may not be fast enough though, so you could either increase the timeout or, as a last resort, manually maintain a materialised index (or secondary index) to keep track

rschmukler23:09:24

I’m trying to build the materialized index, but it’s taking minutes to populate

rschmukler23:09:29

I’ve tried both way

rschmukler23:09:53

Is there any way to get into the raw indices?

rschmukler23:09:46

(and, is there a raw index that would be of any use here)

refset10:09:06

> Is there any way to get into the raw indices? Officially, no 🙂 see https://github.com/xtdb/xtdb/discussions/1514 However, there is an AV index, and you can access it directly like this https://gist.github.com/refset/357699aeeb59d605f0d005bf65f672aa

refset11:09:29

You can see the index definition here: https://github.com/xtdb/xtdb/blob/8bcbf1eb03b459edd0d4c25c588eed0ae2b174c4/core/src/xtdb/codec.clj#L55 How it gets encoded: https://github.com/xtdb/xtdb/blob/8bcbf1eb03b459edd0d4c25c588eed0ae2b174c4/core/src/xtdb/kv/index_store.clj#L115 That it gets encoded for every field-value in each document: https://github.com/xtdb/xtdb/blob/8bcbf1eb03b459edd0d4c25c588eed0ae2b174c4/core/src/xtdb/kv/index_store.clj#L957 How it gets prefix-scanned: https://github.com/xtdb/xtdb/blob/8bcbf1eb03b459edd0d4c25c588eed0ae2b174c4/core/src/xtdb/kv/index_store.clj#L673-L681 And that the query engine does make use of the AV scan here (exclusively) https://github.com/xtdb/xtdb/blob/8bcbf1eb03b459edd0d4c25c588eed0ae2b174c4/core/src/xtdb/query.clj#L560 ...but I don't yet understand the ins-and-outs of how the n-ary-join-layered-virtual-index executes enough to explain why the initial solution I tried in that gist doesn't work 😅

rschmukler19:09:05

Amazing, I’ll give this all a read. Thanks for the tips 🙂

🙏 2
rschmukler19:09:40

Looking through the discussion - it seems like the distinct aggregate could also just be smarter in-terms of using this index…. Or am I missing something?

rschmukler19:09:31

(and seems somewhat similar to the idea proposed in GH#1515 too)

refset09:09:09

Yep! Although I think the triple clause could use it whenever the e is _ (and v is a logic var) and it would have the ~same effect

refset12:09:25

Hey again, any joy with the speed up?

rschmukler15:09:00

Hey! Yes, using the AV index directly worked perfectly for my usecase.

🙌 2
rschmukler15:09:00

That being said, I still think that being forced to materialize the attributes (as opposed to the EID or even the sorted serialized representation) in an index really is difficult to overcome

rschmukler15:09:25

Specifically if I have a bounding box constraint, it might load up 500K records that then need to be deserialized from geojson, encoded into the attribute binary, and then joined… I really think that being able to lazily provide the serialized (and sorted) representation would be a huge win for third-party indices…

rschmukler15:09:36

Is there any chance I could convince you all to consider throwing those capabilities on your roadmap?

refset15:09:14

Well, the direction we've taken with Lucene is almost in the opposite direction...where we've de-emphasized usage of the predicate function integration (which is still potentially useful in basic scenarios) in favour of running lots of small queries and performing the temporal resolution in userspace, e.g. https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/modules/lucene/test/xtdb/lucene/extension_test.clj#L38-L54 - you can potentially then hide all this in a user-defined predicate to achieve the same end result in Datalog (as per https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/modules/lucene/test/xtdb/lucene/extension_test.clj#L137-L150) Do you think that could work for your scenario also? I wonder if there's some way to avoid the excessive serialization work by either using ByteBuffers or extending this codec protocol (or similar) https://github.com/xtdb/xtdb/blob/5e1d822602a39fcc6a448f959c294def21988aa1/core/src/xtdb/codec.clj#L211

rschmukler15:09:43

Thanks for the tips as always @U899JBRPF - I will take a look and see whether I can figure out how it can best be applied. Maybe I can work it into some of the work of getting up to speed w/ v18 too

refset17:09:34

My pleasure, as always! I'd be happy to talk it through on a call / pair on it also