we only save a subsection of stuff in memory on each round of saving,
but I set the counter to zero on how long to wait for the next save.
This makes us save more often till we actually saved most stuff, avoiding
buildups of caches until we do the periodic big flush.
This change makes the limits be static variables, which means they are
only settable once for a process. The only usecase so far is to use much
smaller limits in testing situations.
This allows queries like RPC and mempool-accept to continue
even during a prune (we just use the 'old' DB) and when its
done the last active query will delete the old DB.
This assumes a sane filesystem where you can rename a file that
is in use.
On all mature operating systems and filesystems creation of a 1GB file
will not actually use that data, only when we start using blocks does
the disk space get used.
As such this is a premature optimization that Core added and I ported to
Flowee which this commit removes.
Additionally, increase the block file size to 1GiB
Instead of having a magic 10 open files, we now have a scheduled
run that closes all mapped files 30 seconds after last usage.
This allows many files to be opened with extremely low overhead and
cost (they are not actually loaded until needed, and even then only
per 4K page).
We inherited the design from Core that you can put in your config
file which chain you will follow (testnet, regtest).
This has the effect that you can't have different config files for
different chains and that we always read the fallback location for
all chains. Especially the last one is just harmful.
So, regtest/testnet can only be selected from commandline.
If you choose one of those, it will look for a flowee.conf in the
related subdir (flowee/regtest/flowee.conf for instance) and NOT
for the global one. The global one is only ever read for mainnet.
When pruning we sort leafs by txid / output and refrain from
writing the txid repeatedly for outputs of the same tx.
Additionally, use the fact that we only ever get to a leaf via
a bucket and since the first 64 bits of a txid is there, skip
repeating them when writing to disk.
Last, make pruning have different strategies.
This should shrink the utxo by about 40%
Actually delete the asio io_service when we are done with it since that
will delete any left over tasks in the queue.
This makes deletion order more predictable.
After we extract all block-headers from the files and we then shut down
the node this makes sure that we avoid thinking we finished because
there are no more jobs.
This so that a restarting node has the opportunity to continue indexing.
TomZ