As we moved most of the creation of a BufferPool to be via the
Streaming::pool() method, which uses a thread-local, it makes sense
to start cleaning up the design and make it more modern C++.
The above mentioned method would return a reference and you'd see
loads of places use `auto &pool =` which is less than ideal.
As the number of places where we actually instantiate a BufferPool
goes down, the usage of some sort of smart pointer makes more sense.
This now makes all APIs use BufferPool be wrapped in a shared_ptr.
This adds a unit test to see if the combination of invalidateBlock and
reconsiderBlock do what we expect to do.
The main issue was that we store an invalidated block in the UTXO and we
forgot to re-validate that on reconsider.
Additionally I avoid writing a lot of unchanged data to the block-index.
This fixes that the .info file is sometimes not written when nothing
changed in the db file.
This fixes that the info file with .1.info extension is reused again
and again, effectively reducing the usefulness of the snapshots.
This only really affects the indexer as pruning avoids this most
of the time.
Closes issue flowee-issues#6
When the UTXO saved checkpoints this change makes sure we also store the
index-db changes.
Since we stopped saving simple state changes fromt the index-db, the
only real data we save is the 'undo-block-index', as such this will be
relatively cheap to save.
Without an undo block position we will currently fail to verify those
blocks and as such it is useful to save all at the same time to actually
have a state we can start from.
When, on loading, the blockindex and the UTXO don't agree then try to find an older UTXO
state where they do agree.
The most typical state issue is where a block stored in the blocksdb is not available in
the index due to corruption or similar.
To allow the UTXO to actually use the power of checkpoints we need to
make sure that the block-validation state is not stored separately from
it.
The goal is that when we have some curruption we can just go back to an
earlier state of the UTXO and re-validate the blocks to get to the
current tip. The often seen problem is that corruption will instead
leave the block-index (leveldb) with an incorrect state so the replay
fails.
This change solves that by no longer reading the block-validation-state
and no longer writing it on a state change.
This avoids an internal error from failing one block due
to a perceived missing input while the block may be fine
but other issues need looking at (for instance disk errors).
Replace the m_buckets unsorted map with a lock-free version
based on atomic pointers. (BucketMap)
remove the m_leafs and move those into the bucket struct.
Make the access to the jumptable transactional to avoid one big lock
over all datastructures.
On my threadripper 2990WX the entire 150GB BCH blockchain was
parsed and imported in under 3 hours.
The current system writes based on the amount of changes, which is a bit
risky as long as we have small blocks as the amount of changes may for
hours be below the "lets write" limit.
So also write every 5 minutes, just to make sure we won't keep data in
memory longer than needed. Also this allows pruning to happen for people
that don't run their node all the time.
This change makes the limits be static variables, which means they are
only settable once for a process. The only usecase so far is to use much
smaller limits in testing situations.
When pruning we sort leafs by txid / output and refrain from
writing the txid repeatedly for outputs of the same tx.
Additionally, use the fact that we only ever get to a leaf via
a bucket and since the first 64 bits of a txid is there, skip
repeating them when writing to disk.
Last, make pruning have different strategies.
This should shrink the utxo by about 40%
* remove unused code
* reorg code to make the access to disk be outside the mutex
* add detection of slow disk-writes and slow down data coming in
* Update and fix the rmHint
together with blockFinished() as 'commit' this introduces the ability
to modify the in-memory utxo which can be rolled-back with ease.
This allows us to "modify" the utxo while validating a block in an
optimistic manner and only spend extra resources doing a rollback
should the block end up not being Ok.
The new UnspentOutputDatabase classes are only very loosly a database, we
purely register and store unspent outputs there. But unline a DB we don't
allow modification (just insert and delete).
This replaces the coin-db (which was based on leveldb) and as first goal
this gives us a higher level of stability. The level-database was known
to give corruption issues.
Notice that users will need to do a manual reindex on first update.
The old was based on levelDB which has scalability issues and stability
issues. (most often cited problem is corrupted database..)
This unspent output database I wrote is based on the idea that we need
never actually update any rows, which makes most old fashioned databases
a bad fit.
All we do is create rows and we forget rows. So lets design a DB to fit
that pattern.
tomFlowee
TomZ