Clojurians Log v2

This might be a bit of a broad and subjective question, but I have some data on Postgres on which I need to run an expensive SQL query, whose results I need to paginate, sort and filter. The resulting data is deeply nested and the schema is dynamic. This query is slow and I'd like to save it to disk periodically so it can be served quicker (for reporting purposes). I also want to do the pagination, sorting and filtering in memory (I know, bad idea!), because there are various combinations that can be applied and persisting them all to disk seems inefficient. A couple of questions: - Can I use tech.ml.dataset to do these operations efficiently in memory? (It looks like the purpose of the library, but I'm not sure if my use case fits) - What is the best data format for me to use? After experimentation, nippy works out of the box (as expected), whereas parquet and arrow keep giving me various errors (Unsupported types like :decimal and :object, doesn't handle nil values, etc). My first thought was a simple CSV but I ruled it out because of the nested data. Perhaps I'm in the wrong space and I shouldn't be doing any of this at all 🙂