Fork me on GitHub

Hey. Has anyone ever tried to excise large amounts of data? We currently have 15B datoms, and are thinking of dropping roughly 1/3 - 1/2 of them. Doing it in one go clearly would not hold, so just wondering whether there are any guidelines/benchmarks than could help us tackle this


May I ask what is driving your desire to remove 1/3 to half your DB? Is there an underlying problem causing you to consider this option? In general, excision was not designed to cleanup old data. It was specifically designed to meet puts substantial burden on indexing. If you were to create a too large excision job without proper batching/testing and understanding you can potentially render the system unusable while indexing finishes. That being said, we have greatly improved the performance of excision in the most recent release and it may be possible to design a batched excision job to address. If you want to discuss further, I'd encourage you to open a support case and perhaps we can meet to chat about this topic. <mailto:[email protected]|[email protected]> or the website