This makes it possible for the most recent file to not be the default
filesize but much smaller. This fixes us to writing outside of the
allocated membuffer and causing a kernel signal.
Additionally we start a new file for writes in that case.
This case is seen in the unit tests where we have a perfect-fitting file
in git which we then try to write to when adding a new block.
As such this is tested in the upcoming unit test.
The code that connects blocks together uses the block-hash (blockId) for
that, which we always just created on the fly.
Instead store the blockId on the state and in that way make adding
headers not entirely in-sequence be 10 times as fast.
Additionally, we wait with actual validation after a reindex until the
files are all found because otherwise we can't save the meta blocks.
The API throws when the directories already exist, which is quite
unexpected.
This now changes the code to silently ignore these issues in the
knowledge that the immediately following opening of files will
fail with a nice error message.
This adds a unit test to see if the combination of invalidateBlock and
reconsiderBlock do what we expect to do.
The main issue was that we store an invalidated block in the UTXO and we
forgot to re-validate that on reconsider.
Additionally I avoid writing a lot of unchanged data to the block-index.
If we are past at least 2000 blocks of the items stored in a data file we
can delete the revert file and save a couple hundred megabytes of
on-disk storage for every blk file.
Small refactor; move the partition check away from main and the obsolete
global variables we used to use, instead now just add it to the DB which
owns the data it works on.
This fixes the bug that in various cases we'd incorrectly get a warning
about no blocks being found in the last N hours.
To share the downloaded blocks between instances on Linux it is common
to sym-link the blk files from read-only storage.
The Hub would fail to write to the last file due to the file being on
read-only storage and the Hub would shut down.
This change makes sure we instead create a new file instead of trying to
write to a symlinked one.
Boost throws an exception when the resize fails, which would cause
a total shutdown of the client. So make sure we catch it in the
scheduled task to avoid this problem.
When reindexing we now always first try to open a file read-write because
the algo to only open the last file in a sequence as RW failed in such
situations where we don't know the amount of files yet.
To allow the UTXO to actually use the power of checkpoints we need to
make sure that the block-validation state is not stored separately from
it.
The goal is that when we have some curruption we can just go back to an
earlier state of the UTXO and re-validate the blocks to get to the
current tip. The often seen problem is that corruption will instead
leave the block-index (leveldb) with an incorrect state so the replay
fails.
This change solves that by no longer reading the block-validation-state
and no longer writing it on a state change.
Move some globals and all stuff pulling in crypto to the
server/serverutil.cpp file
Remove dead code.
Move several items that were used in only one place to the respective
files using them.
Move the class WaitUntilFinishedHelper into its own file.
Made sure no header includes utils.h
remove a handfull of files including utils.h for nothing.
On all mature operating systems and filesystems creation of a 1GB file
will not actually use that data, only when we start using blocks does
the disk space get used.
As such this is a premature optimization that Core added and I ported to
Flowee which this commit removes.
Additionally, increase the block file size to 1GiB
Instead of having a magic 10 open files, we now have a scheduled
run that closes all mapped files 30 seconds after last usage.
This allows many files to be opened with extremely low overhead and
cost (they are not actually loaded until needed, and even then only
per 4K page).
After we extract all block-headers from the files and we then shut down
the node this makes sure that we avoid thinking we finished because
there are no more jobs.
This so that a restarting node has the opportunity to continue indexing.
The new UnspentOutputDatabase classes are only very loosly a database, we
purely register and store unspent outputs there. But unline a DB we don't
allow modification (just insert and delete).
This replaces the coin-db (which was based on leveldb) and as first goal
this gives us a higher level of stability. The level-database was known
to give corruption issues.
Notice that users will need to do a manual reindex on first update.
tomFlowee
TomZ