Clojurians Log v2
xtdb
(ref: XTDBv1)
I was about to upgrade from 1.23.0 to latest 1.24.x but had difficulties of configuring Google Cloud Storage checkpointing in a way that would re-use existing checkpoint files. The result is that while deploying the google cloud startup probe fails because the startup takes too long when XTDB tries to build it's kv-storage, this is the last log line before container gets restarted until retries are exhausted.
INFO xtdb.tx - XT tx ingester starting... Latest completed tx 'null', latest submitted to tx-log '2219256'
Old config for checkpoints was
{:checkpointer {:xtdb/module 'xtdb.checkpoint/->checkpointer
:store {:xtdb/module 'xtdb.google.cloud-storage/->checkpoint-store
:path (str "gs://" bucket)}
:approx-frequency (java.time.Duration/ofHours 1)}}bucket is a configurable name for the bucket (eg. "pilot-xtdb-checkpoints"). I tried the new version with
{:checkpointer {:xtdb/module 'xtdb.checkpoint/->checkpointer
:store {:xtdb/module 'xtdb.google.cloud-storage/->checkpoint-store
:project-id (:xtdb-project-id config)
:bucket bucket}
:approx-frequency (java.time.Duration/ofHours 1)}}:prefix "gs://" to store config with no visible effect. Are there other reasons for not using existing checkpoint files (like changed format) or am I just configuring it wrong?
(just returned to work from vacay, so brains might be a tad bit slow to work, sry if this something really silly pebkac stuff)
Hey @jussi.mononen 🙂 did you switch buckets here also? wondering if https://github.com/xtdb/xtdb/issues/2553 might apply
Nope, same buckets, I did not make any changes to our cloud infra.
Did a routine "bump dependencies" rounds and run into this 😎
The only change was version and then config changes
Kk, I'll keep thinking thinking-face
🙏🏻
this is PR where things changed https://github.com/xtdb/xtdb/pull/2621/files
assume you didn't have prefix in your bucket path previously?
No, we did not. Bucket name is a simple string without any directories.
Wondering also whether my assumption / deduction of the cause is correct. if nothing has changed physically in google cloud storage, all paths and files are as they were and I make the configuration change when updating version to 1.24.5, everything should just work as before?
A screencap of the bucket view in GCP in question
My understanding is that the old files should remain usable from the new storage module, but I haven't verified it with my own eyes/REPL
Do you have REPL access to the bucket? Could be worth checking that list-blobs in the GCS ns actually returns something (and isn't hitting authz issues etc.)
If you have any bandwidth to pair on Monday I can make the time - can't today unfortunately
Not sure if the bucket is accessible from REPL, we do have bastion host / ssh access to our VPC but havent' checked if the bucket is actually accessible.