Doing a garbage collect of the Sha256 based databases means we remove
all the records that have been deleted from our file.
We also sort the file to have all the jump-tables at the end, making it
much cheaper on memory-locality to find (or not) items in the DB.
The downsides are that this prune step takes time, writes dozens of MBs
and that we lose checkpoints. The latter means we no longer can rollback
to a safe position, simply because we flushed those records.
So we want to do this often enough to avoid fragmentation but not too
often because it creates a greater risk on data consistency.
This algoritm checks the actual data and calculates the fragmentation of
the jump-tables to decide if we want to start a GC.
When we do GC, we try to do as many files as makes sense, to make sure we
can wait quite a long time before we need to do a new GC.
When pruning we sort leafs by txid / output and refrain from
writing the txid repeatedly for outputs of the same tx.
Additionally, use the fact that we only ever get to a leaf via
a bucket and since the first 64 bits of a txid is there, skip
repeating them when writing to disk.
Last, make pruning have different strategies.
This should shrink the utxo by about 40%
tomFlowee
TomZ