Clojurians Log v2
xtdb
I'm not totally getting how the clojure API, using xtql, works with XTDB.
I can connect to XTDB with next.jdbc, but how do I do it with the Clojure API? With xt/client? Am I able to use connection pooling with that?
Edit: NVM, I see you can query a hikari datasource directly.
> Is XTQL just for reading, and you use the SQL api for inserting?
in addition to XTQL reading, you can also use limited writes - e.g. :put-docs, :delete-docs - but for anything more complicated, yes, I'd use SQL
> My understanding is that the XTDB team have rebuilt the Clojure API with XTQL to convert it to SQL under the hood anyway
yes - also it's not converted to 'SQL' SQL - just a string inlined within the query: (format "XTQL $$ %s $$" (pr-str your-xtql)
i.e. it's an XTDB SQL extension to support XTQL inlined within the query string
Note that you can use XTQL via the SQL API too.
Is XTQL just for reading, and you use the SQL api for inserting?
Personally, I wouldn't use XTQL -- I'm solidly in the SQL camp. I don't know about whether you can run XTQL insert/update ops via SQL -- I haven't tried it but it feels like it ought to work. My understanding is that the XTDB team have rebuilt the Clojure API with XTQL to convert it to SQL under the hood anyway -- the SQL API is the primary one these days.
Got it
I mean, some folks love the XTQL approach but I don't really see the point.
I guess the biggest draw would just be making conditional queries easier. I don't have fun with that in SQL, though I do it a lot.
HoneySQL.
Perfect for constructing conditional SQL queries.
(and it supports XTQL in SQL!)
https://cljdoc.org/d/com.github.seancorfield/honeysql/2.7.1310/doc/getting-started/xtdb-support
I'm trying to get a better idea of how the log, storage, and postgres database all work together.
In the docker setup image XTDB, it sets the storage to !Local under /var/lib/xtdb/buffers. There appears to be no data in here even when I insert data. Isn't XTDB supposed to persist to this storage when the Log is flushed?
The log also has some weird encoding. Is there a special way I am supposed to read the log file?
> Is it accurate to say the log is the first entrypoint for transactions? Does XTDB get updates from the log? yes and yes 🙂 the log is only considered a transient store - the XTDB nodes then consume that log and write the transaction results to the more durable object store
Re: persisting to Storage, you may be seeing this https://clojurians.slack.com/archives/CG3AM2F7V/p1748788917113179?thread_ts=1748742196.687199&channel=CG3AM2F7V&message_ts=1748788917.113179
I’m not sure where/if it’s documented but I believe the configuration options https://github.com/xtdb/xtdb/blob/main/core/src/main/kotlin/xtdb/api/IndexerConfig.kt#15:
indexer:
rowsPerBlock: # of rows
flushDuration: duration string Not 100% on the local log format exactly, but it’s going to be some sequence of protobuf encoded messages with the actual trie data and block flush notifications, not really meant for direct consumption by anything but the xtdb nodes.
Those are the kinds of things I wanted to understand better. For instance, there's docs on backing up the storage vs the logs, and it sounds like there are cases where the logs have a lot of stuff in them. I thought observing them would be the easiest way to understand.
Is it accurate to say the log is the first entrypoint for transactions? Does XTDB get updates from the log? I am just wondering because Kafka is an option, and assume the reason for that is Kafka has high throughput as an append only log.
You might like this article if you haven’t seen it. https://xtdb.com/blog/building-a-bitemp-index-3-storage The predecessors also, but I think this one is more immediately on-topic for what you’re interested in.