Clojurians Log v2

Hey @nikwarke you could try swapping clojure.string/includes? for println in that expression and see what comes out, but it sounds like either m or q are not actually the strings you're expecting them to be (or you need to cast them, e.g. using`(name sym)`)

I was able to fix it, q was defined somewhere and going in as value. Thanks @taylor.jeremydavid

awesome 🙂

thank you! I'm guessing then (also based on the diagram mentioned below) that the HA story for XTDB would be that for the document store and the transaction log? continuing on: I just stumbled onto this architecture diagram at https://v1-docs.xtdb.com/concepts/what-is-xtdb/. Is this an accurate diagram? One question (for now) arising from this: if writes occurring from a transaction go to both the document store and the transaction log, what is the transaction log for? I see inflows from both the document store and the transaction log going into the indexer. Do they provide different data for the indexer? maybe there is a separate box called the query engine that pulls the documents from the document store, and it's not actually going to the indexer? I know this is getting into internals now, but I'm curious. I would love to get more details about the content and reasons for the flows in the diagram. I imagine also that the writes to the document store and the transaction log might be sequential, with the writes to the log happening only once we're sure about the results of the write to the store? (btw, I know that's a lot of the questions in there. My apologies! I'm just genuinely curious and wanting to find out more)

> Is this an accurate diagram? Yes, it's still essentially correct. There are of course some nuances to how more advanced operations work, like transaction functions, and also how pull and entity APIs make use of in-memory caches in front of the document store, which aren't represented in that diagram > what is the transaction log for? It ensures a single, 100% linear and totally immutable history with tight latency bounds. Whereas the document store can have documents mutably replaced with 'tombstones' (to support eviction) and is more of a random-access affair > Do they provide different data for the indexer? The indexer only discovers which documents exist in the first place because the log stores the references. The log entries can be thought of like metadata for a groups of documents that are committed (ACID) together. The log entries hold the :put or :delete or other operations, but then only ever hashed references to documents/IDs and never the user's actual data. > I imagine also that the writes to the document store and the transaction log might be sequential, with the writes to the log happening only once we're sure about the results of the write to the store? This is correct ✅ > My apologies! Not at all, it's been a while since this particular topic came up (no pun intended 🙂)

🙂 topic as in Kafka, I assume

Hey @jf.slack-clojurians are you looking at v2 in particular? We added SSL support for the pgwire endpoint last week https://github.com/xtdb/xtdb/issues/3637

hey @jf.slack-clojurians - just to check, because there's quite a range of answers here - when you say "secure", are there any threat models you're particularly considering?

I am looking at v1 (not looking to go to v2 yet). Re "secure" 🙂 I'm sorry, I understand it's kinda vague. I'm really just looking for an answer I can pass along that would make a mostly non-technical audience happy. I think given this case, the threats we would consider would be outside of hired devs. We can start with threats coming from the network. I was also asked a question about encryption; but I'm not entirely certain that there is any practical value (feel free to educate and correct me) in encryption at rest. I'm thinking is that u would have to have the key stored - in plaintext - somewhere; and then what's protecting that key?

For v1 the story can be broadly summarised as "please refer to the respective upstream documentation for the S3/Kafka/Postgres/etc. storage modules and treat securing XTDB like securing any other JVM application" The main security area we spent time on explicitly for v1 is the "Predicate Allowlist" which is intended to prevent untrusted queries from doing anything other than read data in the database (e.g. doing other things in the JVM or even reaching out beyond the JVM), see https://v1-docs.xtdb.com/administration/configuring/#fn-allowlist

RocksDB has encryption support which is likely much more efficient than relying on filesystem-level encryption, but XTDB does do anything explicit to support/enable it - you should definitely be able to configure it without XT getting in the way