Fork me on GitHub
#xtdb
<
2020-06-03
>
worlds-endless17:06:24

What's the fastest way to count all entries in a node?

dvingo03:06:56

you can try: https://opencrux.com/docs#_attribute_stats This was from earlier in the slack log: > Note that you are able to add :full-results? true to the query map to easily retrieve the source documents relating to the entities in the result set. For instance to retrieve all documents in a single query:

{:find [e]
   :where [[e :crux.db/id _]]
   :full-results? true}

jarohen08:06:37

Yep - (:crux.db/id (crux.api/attribute-stats node)) will get you a good estimation of the number of documents indexed (all versions of all entities). If you want a count of the entities the above query's good - if you're just going to (count (crux/q ...)) this query I'd recommend omitting the full-results? true as this will make it significantly slower

royalaid21:06:32

Hey all, ran into a bit of an issue while using the JBDC backend

jarohen08:06:03

Hey @U0S3YK6HK, thanks for raising this - could you post a stack trace?

royalaid08:06:21

I have since blown away the DB as everytime I ran the node it would blow up with that error

royalaid08:06:51

Should be easy to replicate, just transact over 100k items at once

jarohen08:06:15

Ah, np - will give it a go

royalaid08:06:26

I am running on top of postgres if that help specifically

jarohen08:06:39

Thanks 🙂

royalaid08:06:41

If you don't have time I can get around to it later this week

jarohen08:06:48

That'd be great, thanks - I've raised a Github issue: https://github.com/juxt/crux/issues/902

royalaid02:06:45

So interestingly I have played around with this trying to replicate it and I can't. I have transacted 1 million super small docs in 1 transaction and it succeeds. Granted before I was using Windows Native postgres and this postgres is in a docker container but that really shouldn't affect things

royalaid02:06:01

I will continue to dig as I have time over the next few days

royalaid02:06:15

and I will copy this over to the github issue for posterity

royalaid18:06:00

In order to dig into to this more I want to replicate the transaction that triggered this by using the original project but I don't want to bork my DB again. What would be the best way for me to backup and restore my DB and Crux index? Just backup my RocksDB folder and dump the db using PGbackup?

refset17:06:55

Hey @U0S3YK6HK sorry to keep you waiting for an official response. I think your plan is/was good, assuming your tx-log really is stored only in the postgres. However dumping postgres db alone should be enough, as Rocks will always rebuild itself from the log, and it would be a good test to verify your dump and restore works correctly. I.e. check the fully re-built node looks roughly the same as the old one (comparing the output from attribute-stats is a safe bet)

royalaid20:06:32

Following up on this thread again I have managed to trigger the bug again, this time in a different project. I have added a stacktrace and will not touch the project until I get direction on how to best preserve or debug the problem.

royalaid21:06:50

.IOException: Tried to send an out-of-range integer as a 2-byte value: 123647

royalaid21:06:19

My assumption is that the 123647 is the number of docs that I am trying to send in a single transaction

royalaid21:06:38

I have no problem submitting them via a map call

royalaid21:06:59

but it seems like limits such as this should be documented with the JBDC backend config

royalaid21:06:21

Admittedly I haven't done a ton of digging if this is or is not documented.

royalaid18:06:00

In order to dig into to this more I want to replicate the transaction that triggered this by using the original project but I don't want to bork my DB again. What would be the best way for me to backup and restore my DB and Crux index? Just backup my RocksDB folder and dump the db using PGbackup?

royalaid20:06:32

Following up on this thread again I have managed to trigger the bug again, this time in a different project. I have added a stacktrace and will not touch the project until I get direction on how to best preserve or debug the problem.