Fork me on GitHub
Michael W23:02:02

I've had an idea for an app. I want to use xtdb as a sort of cache for data from my network gear. I have switches with a list of mac addresses and ip addresses. I can use an api to pull out those ips and macs. I would like to create a document in xtdb with that data, say 1 document per switch. I'll run a job that pulls the data every X minutes and inputs it into xtdb. Is there a way to compare the data I get from the api, to the xtdb document without pulling the entire document to compare them in memory? I would only like to update the document if things change.


I guess the comparison will still have to happen in memory ultimately, but you can pass the data from the API into a query with :in and use something like [(xtdb.api/entity $ ?id) ?e][(= ?e ?data)]

Michael W23:02:17

So there is no way to have xtdb say run a checksum function against the document, and I can run the checksum against the data from the api, and just compare the checksum value?


Internally there are hashes that could do this, but I can't think of a clean way to access them via a query right now. I mean you could use internal APIs...but that's not ideal


The document hashes are visible in the open-tx-log API

Michael W23:02:31

Ok thanks, if this is doable, even if I'm using internal apis then I can scale my solution up to thousands of switches which is what I need.

Michael W23:02:54

I don't want to be writing the full document every 5 minutes 5000 times, when data only changed on some much smaller subset of switches.


Actually the entity-history API surfaces the hashes also, and is probably usable for this.

Michael W23:02:47

I'll look, just didn't want to pick xtdb if this was impossible. Thanks much for the info.


You can use xtdb.codec/new-id to generate your hashes and compare

Michael W00:02:19

Excellent, I'll get a small project setup and start playing. Thanks again and xtdb looks pretty amazing so far.

🙏 2

Note that open-entity-history will allow you to only retrieve the latest version


Happy to help :)