This fixes that the .info file is sometimes not written when nothing
changed in the db file.
This fixes that the info file with .1.info extension is reused again
and again, effectively reducing the usefulness of the snapshots.
This only really affects the indexer as pruning avoids this most
of the time.
Closes issue flowee-issues#6
When the UTXO saved checkpoints this change makes sure we also store the
index-db changes.
Since we stopped saving simple state changes fromt the index-db, the
only real data we save is the 'undo-block-index', as such this will be
relatively cheap to save.
Without an undo block position we will currently fail to verify those
blocks and as such it is useful to save all at the same time to actually
have a state we can start from.
This allows the pruner to be used on the 'tip' DB file, at which point it
will set the filesize to be the default 2GB one.
Previously pruning the tip would confuse the Hub with a smaller file
size.
Doing a garbage collect of the Sha256 based databases means we remove
all the records that have been deleted from our file.
We also sort the file to have all the jump-tables at the end, making it
much cheaper on memory-locality to find (or not) items in the DB.
The downsides are that this prune step takes time, writes dozens of MBs
and that we lose checkpoints. The latter means we no longer can rollback
to a safe position, simply because we flushed those records.
So we want to do this often enough to avoid fragmentation but not too
often because it creates a greater risk on data consistency.
This algoritm checks the actual data and calculates the fragmentation of
the jump-tables to decide if we want to start a GC.
When we do GC, we try to do as many files as makes sense, to make sure we
can wait quite a long time before we need to do a new GC.
When, on loading, the blockindex and the UTXO don't agree then try to find an older UTXO
state where they do agree.
The most typical state issue is where a block stored in the blocksdb is not available in
the index due to corruption or similar.
To allow the UTXO to actually use the power of checkpoints we need to
make sure that the block-validation state is not stored separately from
it.
The goal is that when we have some curruption we can just go back to an
earlier state of the UTXO and re-validate the blocks to get to the
current tip. The often seen problem is that corruption will instead
leave the block-index (leveldb) with an incorrect state so the replay
fails.
This change solves that by no longer reading the block-validation-state
and no longer writing it on a state change.
The UTXO keeps outdated records around in an append-only file, which
means we need to do a garbage collect regularly.
This new algo uses the commulatative amount of changes since last GC
(aka prune) as an indicator to plan a new one.
The effect should be much smaller files to keep in memory and the data and
jump tables being much more localized which should result in higher
throughput.
Move some files back to the server "library".
Merge the 'console' lib with server, as it doesn't really make sense with
just one file and nobody exclusively linking to it.
The server "libary" is not really a library, its the place we put all
the files shared by hub-qt hub-cli and hub.
We no longer depend on these files from other places (mostly due to
moving to the new logging framework) and as such we can move the files
back.
This means that for apps like 'pos' no longer link against libSSL
The streams no longer zero-after-free, there are no secrets transported
in datastreams so this is useless and avoids linking in one OpenSSL
call.
The insecure_rand() method depended on the openssl code to seed it with
randomness. Now replaced with a proven current-time-milliseconds.
This is enough in those cases because it was always meant to be an
insecure random.
Move some globals and all stuff pulling in crypto to the
server/serverutil.cpp file
Remove dead code.
Move several items that were used in only one place to the respective
files using them.
Move the class WaitUntilFinishedHelper into its own file.
Made sure no header includes utils.h
remove a handfull of files including utils.h for nothing.
Now multiple different targets use this database, the messages
were confusing. Next to that "pruning" is not really correct
and it lead to confusing. More correct is the term GC.
In the rare case when we already created a new DB file, but never
managed to write a .info file, the constructor would end up rejecting
the whole DB.
This just drops off the last useless file and goes from there.
Fix a bug that could make the wait between saves too long and now aim to
make the saving more often because this helps us determine in time that
the file is full so we don't go over the end.
Write a changesSincePruning integer which has the following effects;
after a restart;
* We will do a prune faster and/or more often for nodes that are not
running very long stretches of time.
* We will skip pruning files that don't need it.
Context; pruning of the UTXO is throwing away already spent outputs
that are still in our dirty files.
This avoids an internal error from failing one block due
to a perceived missing input while the block may be fine
but other issues need looking at (for instance disk errors).
Checking if my database file is full can not be assumed to only happen
on end-of-block if blocks get really large.
I'm taking a small performance hit to check this on insert and
flushSomeNodesToDisk on a more regular basis.
The 'server' library has always been a catch-all and
ideally only the hub links it in (far future goal).
In line with this I move a list of files out of server
into the utils lib.
I choose 'utils' because all these are plain old data
objects that many crypto apps will find useful.
now in utils/primitives/
* CScript
* CPubKey
* CTransaction
* CBlock
* FastTransaction
* FastBlock
* CScript
streams.h is now in utils/streaming/
hash.h is now in utils/
This is really only relevant for people using btrfs on Linux, we will
set the utxo dir to be no-copy-on-write which causes all new databases
to follow this flag and be faster due to lack of copying on write.
The reason there are no standard library versions of lock-free
containers is because you want to always take full advantage of
the details in question.
In this case (read millions of times for each modification) it makes
no sense to use anything other than a standard container, but put in
a copy-on-write block. Simple and easy.
TomZ