Fork me on GitHub
#datalevin
<
2023-02-02
>
Huahai21:02:03

I am working towards 0.9.0: compressed data. The goal is not just to reduce the storage space, but also hopefully speed up read/write more, by the virtue dealing with less data.

Huahai22:02:35

The plan is the following: 1. correct implementation of iterators on LMDB dupsorted dbi, this is a dbi where a list of sorted values can be associated with the same key, basically, a two level B+tree. (this is now done), so that we can do 2. store triples in dupsort dbi, so the first element of the triple is the key, the remaining elements are in the value lists. This saves us from repeating the first element needlessly, achieving some compression. 3. we will implement order-preserving compression algorithm to compress keys, so that index scan can be done directly on compressed data.

👍 8