Fork me on GitHub
#xtdb
<
2021-11-01
>
Tuomas11:11:15

Anyone encountered erros similar to?

; Execution error (RocksDBException) at org.rocksdb.RocksDB/open (RocksDB.java:-2).
; While open a file for random read: /tmp/rocksdb/document/000017.ldb: No such file or directory
Not sure how to recover without wiping golden stores

Tuomas11:11:09

Happened on my laptop after I forgot to shutdown everything properly and closed the lid

refset13:11:06

Hi @UH9091BLY can you share your start-node config?

refset13:11:45

Rocks should be resiliant assuming you haven't disabled the WAL

refset13:11:57

Which version is this?

refset13:11:55

From the directory in that message it looks like it was using /tmp ...which doesn't usually survive OS restarts

Tuomas14:11:02

/tmp ...which doesn't usually survive OS restarts oh good to know 😅.

{:xtdb/index-store {:kv-store
                    {:xtdb/module xtdb.rocksdb/->kv-store
                     :db-dir "/tmp/rocksdb/index"}
                    :checkpointer {:xtdb/module xtdb.checkpoint/->checkpointer
                                   :store {:xtdb/module xtdb.checkpoint/->filesystem-checkpoint-store
                                           :path "/tmp/xtdb-checkpoint"}
                                   :approx-frequency "PT1H"}}
 :xtdb/document-store {:kv-store
                       {:xtdb/module xtdb.rocksdb/->kv-store
                        :db-dir "/tmp/rocksdb/document"}}
 :xtdb/tx-log {:kv-store
               {:xtdb/module xtdb.rocksdb/->kv-store
                :db-dir "/tmp/rocksdb/tx"}}
 :xtdb.lucene/lucene-store {:db-dir "/tmp/lucene"
                            :indexer xtdb.lucene.multi-field/->indexer}}

refset22:11:58

ah yes, that looks like your problem 😬 have you lost anything important?

refset22:11:06

which OS are you using?

Tuomas05:11:12

Only dev time sample data that is easy to regenerate. Thought I'd try to understand what happened to save time in the future. I'm using macOS Big Sur 11.2.3

👍 1
🙂 1
Tomas Brejla22:11:16

Hello. I have a question. I want to submit a tx containing a custom ::xt/fn transaction function. I have found that if my custom tx function fails, then after I perform tx/await on the transaction and ask xt/tx-committed? , it returns false. This is fine. My question is : can I somehow get the details regarding why that transaction failed? Sorry if it's covered in the docs, I might have missed it.

refset22:11:09

Hi 🙂 debugging transaction functions generally requires adding logging and looking through logs, since the transaction processing is happening on a dedicated ingestion thread. If you have a debugger handy then there may well be some nicer approach, but I've not tried to do anything advanced like that myself yet. However, you should also be able to get quite far by just running the body of the transaction function elsewhere in the REPL, to simulate how it will execute, since the `ctx` parameter is near enough equivalent to the actual `node` interface > Sorry if it's covered in the docs, I might have missed it. Not at all, I am sure we can definitely add some better details / recommendations here.

Tomas Brejla23:11:05

Maybe I'm trying to do something fundamentally silly and there may be way better way to implement something like this. I just felt that if try to submit some transaction operation and it fails, there might be some API to find out the exception details. I'll probably have to study tx docs more thoroughly to find out what options I have when it comes to figuring out which of the transaction's "custom pre-conditions" were not met.

Tomas Brejla23:11:35

For example suppose that I only want to place my order into db, if there's appropriate stock level for each of the order's items' quantities. If the stock level for some item is too low, I'd like to be able to find out, which of the items have low stock level.

Tomas Brejla23:11:45

similarly, if I submit multiple ::xt/match in my transaction there's probably no easy way of figuring out which of them didn't succeed, is there?

refset23:11:51

Ah, I see. Well if you care about that kind of logic and "error" from a business use-case perspective, then instead of making the underlying transaction abort, you could simply write a "transaction failed" document instead which encodes exactly what you want - could something like that make sense?

Tomas Brejla23:11:08

I guess it's a possible solution. I actually got the same idea - but it feels weird - should I "pollute" my db with basically some sort of "exception representation", just so that I get out the "error details"? I mean.. this kind of error documents would only be useful for a really small amount of time - so why should such document remain in db store? (or I'd have to manually evict such document after using it). Feels a bit strange.

Tomas Brejla23:11:55

But perhaps it's not too "hacky" idea after all and perhaps it's not just a "workaround" but proper acceptale solution (and re-usable "pattern") to such situations. Anyways, thank you for the verification that such idea might work. At least now it's not just my idea, but there are at least 2 people thinking it might just work 🙂 I guess I'll have to sleep on that and maybe try it tomorrow. Thanks again.

🙏 1
refset23:11:39

It's definitely an interesting debate, like what counts as important data that should be archived carefully, vs. what can be quickly forgotten in log files never to see the light of day. Perhaps writing logging statements in your transaction functions is enough. There is also some logging available for capture in the transaction processing code that might be of use, e.g. https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/core/src/xtdb/tx.clj#L143 (though we may decide to change this arbitrarily between releases without much warning) Always happy to chat ☺️

Tuomas16:11:31

like what counts as important data that should be archived carefully, vs. what can be quickly forgotten in log files never to see the light of day . I've had similar thoughts relating to specifically transaction failures. Let's say you have an api for a resource and you are enforcing a uniqueness restraint. When your api gets a request to create a new resource that violates said unique constraint, then you'd normally respond with something like 400 :user/name already taken . With xtdb you would enforce unique constraints in a tx function (you can check before submitting tx but it's not enough). The only way to get data from a tx function to a response is through :xt/put so you would need to create an error document instead of returning false, await for the tx and query for a possible error before returning a response. In an alternate world I could see a tx function returning a map that you can inspect after await-tx

#:xtdb.api{:tx-id
             100361,
           :tx-time
             #inst
              "2021-11-02T16:23:35.106-00:00"
           :tx-details {:ops [...] :errors [...]}

Tuomas16:11:15

The difference between choosing to create new docs or return data from a tx function is the difference between having something to query afterwards or just use during a request/response cycle

truestory 1
👏 1
Tomas Brejla16:11:23

I feel the same. Although it's probably a valid solution, having to create and persist an error document feels like a bit too heavy way for such task. But perhaps there's some good architectural decision behind not providing a way of programatically figuring out the cause of failed transaction (?).

Tomas Brejla16:11:35

btw @UH9091BLY so you ended up creating that error document in your app?

Tuomas16:11:03

For the moment being I only query for errors before submitting a tx, although I could get tx functions failing without users knowing. To be sure a tx function was succesful you’d query afterwards for verification

Tuomas16:11:05

not providing a way now that I think about it there certainly are ways. A tx function is just a function that is ran in a dedicated thread

Tomas Brejla16:11:03

... or remote xtdb node (?)

Tomas Brejla16:11:01

I mean.. someone might be using https://docs.xtdb.com/clients/http/#submit-tx for example..

Tomas Brejla16:11:45

maybe the question is: when failure happens inside custom tx function in xtdb, does xtdb itself somehow persist the information about that error? Or does it just log it somehow internally and forgets it? If it persist the error info somehow, can I dig it up somehow (via tx log or something similar?) My "project" is just a toy project, so I don't mind bringing up that "create error document instead" pattern and it might be the best solution with what xtdb offers at the moment. I'm just thinking whether there should (?) be some alternative way in the future which would allow you to get the information about the failed transaction (its ex-info?) via a standardized way, without having to manually introduce that "error doc pattern".

👍 1
refset09:11:10

XT does keep a list of failed transaction IDs (see https://github.com/xtdb/xtdb/blob/e2f51ed99fc2716faa8ad254c0b18166c937b134/core/src/xtdb/kv/index_store.clj#L387-L395) but doesn't currently store any other information about what happened...maybe it should one day though. Please feel free to open a GitHub Discussion if you feel like proposing an idea or having a longer-term debate 🙂