As the "normal" machine that runs this kind of software gets more
memory, it is Ok to increase the numbers in order to aim to keep more
data cached in memory between check-points, and have less check-points.
This mostly helps speeding up initial block download, it also helps keeping
the utxo datadir small and non-fragmented. All good for speed.
As we moved most of the creation of a BufferPool to be via the
Streaming::pool() method, which uses a thread-local, it makes sense
to start cleaning up the design and make it more modern C++.
The above mentioned method would return a reference and you'd see
loads of places use `auto &pool =` which is less than ideal.
As the number of places where we actually instantiate a BufferPool
goes down, the usage of some sort of smart pointer makes more sense.
This now makes all APIs use BufferPool be wrapped in a shared_ptr.
Doing a garbage collect of the Sha256 based databases means we remove
all the records that have been deleted from our file.
We also sort the file to have all the jump-tables at the end, making it
much cheaper on memory-locality to find (or not) items in the DB.
The downsides are that this prune step takes time, writes dozens of MBs
and that we lose checkpoints. The latter means we no longer can rollback
to a safe position, simply because we flushed those records.
So we want to do this often enough to avoid fragmentation but not too
often because it creates a greater risk on data consistency.
This algoritm checks the actual data and calculates the fragmentation of
the jump-tables to decide if we want to start a GC.
When we do GC, we try to do as many files as makes sense, to make sure we
can wait quite a long time before we need to do a new GC.
When, on loading, the blockindex and the UTXO don't agree then try to find an older UTXO
state where they do agree.
The most typical state issue is where a block stored in the blocksdb is not available in
the index due to corruption or similar.
To allow the UTXO to actually use the power of checkpoints we need to
make sure that the block-validation state is not stored separately from
it.
The goal is that when we have some curruption we can just go back to an
earlier state of the UTXO and re-validate the blocks to get to the
current tip. The often seen problem is that corruption will instead
leave the block-index (leveldb) with an incorrect state so the replay
fails.
This change solves that by no longer reading the block-validation-state
and no longer writing it on a state change.
The UTXO keeps outdated records around in an append-only file, which
means we need to do a garbage collect regularly.
This new algo uses the commulatative amount of changes since last GC
(aka prune) as an indicator to plan a new one.
The effect should be much smaller files to keep in memory and the data and
jump tables being much more localized which should result in higher
throughput.
Move some globals and all stuff pulling in crypto to the
server/serverutil.cpp file
Remove dead code.
Move several items that were used in only one place to the respective
files using them.
Move the class WaitUntilFinishedHelper into its own file.
Made sure no header includes utils.h
remove a handfull of files including utils.h for nothing.
Write a changesSincePruning integer which has the following effects;
after a restart;
* We will do a prune faster and/or more often for nodes that are not
running very long stretches of time.
* We will skip pruning files that don't need it.
Context; pruning of the UTXO is throwing away already spent outputs
that are still in our dirty files.
Checking if my database file is full can not be assumed to only happen
on end-of-block if blocks get really large.
I'm taking a small performance hit to check this on insert and
flushSomeNodesToDisk on a more regular basis.
The reason there are no standard library versions of lock-free
containers is because you want to always take full advantage of
the details in question.
In this case (read millions of times for each modification) it makes
no sense to use anything other than a standard container, but put in
a copy-on-write block. Simple and easy.
Replace the m_buckets unsorted map with a lock-free version
based on atomic pointers. (BucketMap)
remove the m_leafs and move those into the bucket struct.
Make the access to the jumptable transactional to avoid one big lock
over all datastructures.
On my threadripper 2990WX the entire 150GB BCH blockchain was
parsed and imported in under 3 hours.
At least 50% of the UTXO entries reuse the posOnDisk
because they are outputs on the same tx. This changes
such entries from using 2 bytes to 1 byte only when saved
in a bucket.
On low core-count machines the request to save could end up being
delayed since all threads are busy doing things like validating
transactions.
So when a really big block came in I ended up throttling while there
was no save method running in parallel at all.
This fixes this and also makes throtteling be a per-thread thing
instead of doing it inside the mutex.
This change makes the limits be static variables, which means they are
only settable once for a process. The only usecase so far is to use much
smaller limits in testing situations.
This allows queries like RPC and mempool-accept to continue
even during a prune (we just use the 'old' DB) and when its
done the last active query will delete the old DB.
This assumes a sane filesystem where you can rename a file that
is in use.
When pruning we sort leafs by txid / output and refrain from
writing the txid repeatedly for outputs of the same tx.
Additionally, use the fact that we only ever get to a leaf via
a bucket and since the first 64 bits of a txid is there, skip
repeating them when writing to disk.
Last, make pruning have different strategies.
This should shrink the utxo by about 40%
* remove unused code
* reorg code to make the access to disk be outside the mutex
* add detection of slow disk-writes and slow down data coming in
* Update and fix the rmHint
together with blockFinished() as 'commit' this introduces the ability
to modify the in-memory utxo which can be rolled-back with ease.
This allows us to "modify" the utxo while validating a block in an
optimistic manner and only spend extra resources doing a rollback
should the block end up not being Ok.
The new UnspentOutputDatabase classes are only very loosly a database, we
purely register and store unspent outputs there. But unline a DB we don't
allow modification (just insert and delete).
This replaces the coin-db (which was based on leveldb) and as first goal
this gives us a higher level of stability. The level-database was known
to give corruption issues.
Notice that users will need to do a manual reindex on first update.
The old was based on levelDB which has scalability issues and stability
issues. (most often cited problem is corrupted database..)
This unspent output database I wrote is based on the idea that we need
never actually update any rows, which makes most old fashioned databases
a bad fit.
All we do is create rows and we forget rows. So lets design a DB to fit
that pattern.
tomFlowee
TomZ