As we moved most of the creation of a BufferPool to be via the
Streaming::pool() method, which uses a thread-local, it makes sense
to start cleaning up the design and make it more modern C++.
The above mentioned method would return a reference and you'd see
loads of places use `auto &pool =` which is less than ideal.
As the number of places where we actually instantiate a BufferPool
goes down, the usage of some sort of smart pointer makes more sense.
This now makes all APIs use BufferPool be wrapped in a shared_ptr.
This allows the pruner to be used on the 'tip' DB file, at which point it
will set the filesize to be the default 2GB one.
Previously pruning the tip would confuse the Hub with a smaller file
size.
Doing a garbage collect of the Sha256 based databases means we remove
all the records that have been deleted from our file.
We also sort the file to have all the jump-tables at the end, making it
much cheaper on memory-locality to find (or not) items in the DB.
The downsides are that this prune step takes time, writes dozens of MBs
and that we lose checkpoints. The latter means we no longer can rollback
to a safe position, simply because we flushed those records.
So we want to do this often enough to avoid fragmentation but not too
often because it creates a greater risk on data consistency.
This algoritm checks the actual data and calculates the fragmentation of
the jump-tables to decide if we want to start a GC.
When we do GC, we try to do as many files as makes sense, to make sure we
can wait quite a long time before we need to do a new GC.
This means that for apps like 'pos' no longer link against libSSL
The streams no longer zero-after-free, there are no secrets transported
in datastreams so this is useless and avoids linking in one OpenSSL
call.
The insecure_rand() method depended on the openssl code to seed it with
randomness. Now replaced with a proven current-time-milliseconds.
This is enough in those cases because it was always meant to be an
insecure random.
Now multiple different targets use this database, the messages
were confusing. Next to that "pruning" is not really correct
and it lead to confusing. More correct is the term GC.
The 'server' library has always been a catch-all and
ideally only the hub links it in (far future goal).
In line with this I move a list of files out of server
into the utils lib.
I choose 'utils' because all these are plain old data
objects that many crypto apps will find useful.
now in utils/primitives/
* CScript
* CPubKey
* CTransaction
* CBlock
* FastTransaction
* FastBlock
* CScript
streams.h is now in utils/streaming/
hash.h is now in utils/
When pruning we sort leafs by txid / output and refrain from
writing the txid repeatedly for outputs of the same tx.
Additionally, use the fact that we only ever get to a leaf via
a bucket and since the first 64 bits of a txid is there, skip
repeating them when writing to disk.
Last, make pruning have different strategies.
This should shrink the utxo by about 40%
tomFlowee
TomZ