Fork me on GitHub
#xtdb
<
2022-10-13
>
Alexis Vincent00:10:32

Hi there, we’re running into an issue where open-tx-log on a kafka backed node gives us a nil seq, but local rocks works fine. Tested on v21 and v22. Is this supported with Kafka as tx-log?

Alexis Vincent12:10:36

Or is there perhaps an alternative way we can get our hands on the transactions (we want to basically get a group of the documents that were written over time)

refset12:10:00

Hey @UJWLUPW13 can you share the node config map? open-tx-log should surface the transactions properly (the ~same as using Rocks), assuming they have been indexed. You could also try creating https://docs.xtdb.com/clients/1.22.0/clojure/#_new_submit_client and use the open-tx-log method on that, which should give you an unfiltered view of all transactions (whether they have been successfully indexed or not)

Alexis Vincent13:10:29

{:xtdb.kafka/kafka-config (gx/ref :xtdb/kafka-config)

 :local-rocksdb
 {:xtdb/module "xtdb.rocksdb/->kv-store"
  :db-dir "/tmp/rocks"

  :checkpointer {:xtdb/module "xtdb.checkpoint/->checkpointer"
                 :store {:xtdb/module "xtdb.google.cloud-storage/->checkpoint-store"
                         :path ""}
                 :approx-frequency "PT30M"}}

  :xtdb/tx-log
  {:xtdb/module "xtdb.kafka/->tx-log"
   :kafka-config :xtdb.kafka/kafka-config
   :tx-topic-opts {:topic-name "net"
                   :replication-factor 3}}

  :xtdb/document-store
  {:xtdb/module "xtdb.kafka/->document-store"
   :local-document-store  {:kv-store :local-rocksdb}
   :kafka-config :xtdb.kafka/kafka-config
   :doc-topic-opts {:topic-name "net-a"
                    :replication-factor 3}}

    :xtdb/index-store
    {:kv-store :local-rocksdb}}

👍 1