Our database uses memmapping and each blk file allocates a file-
descriptor (FD). As such we should increase the number at the start of
the app, if we can.
Notice that the 2000 is chosen to work with a full node that still uses
tiny blk files (128 typically) instead of 1GB files as the Hub makes by
default.
Also fix a bug in the handing code. Using the hardcoded default
FD_SETSIZE made no sense there..
If for some reason there are a massive amount of orphaned headers, we
don't just keep them around until they are resolved as that is
detrimental to the operation of the node.
Special case this for a reindex and simply skip all orphans fur the
duration.
Doing a -reindex on a datadir of a different client gives us thousands
of small blk files, which may result in us hitting the max simultaneous
open files limit.
This change forces us to flush the list a bit more aggressively and
close open files.
The download from a peer is now limited to just 4000 blocks, at which
point the download proceeds from another peer.
Naturally it can go back to the first afterwards too.
The idea is that we download a month worth of blocks at a time and then
fetch the same blocks from a backup peer in order to be certain we
actually see the real state.
The problem we've seen is that after the first downloads we end up
starting the second and if the initial sync or a similar large download
happens, many people won't leave it running until the backup actually
completes since the UI looks like its finished.
This is also a good starting point for rating peers at download speed by
having predictable chunks and that means we could time and aim to use
the faster peers.
This is becoming relevant as we can see a huge difference where peers
give you 10x performance compared to some of the slower ones.
If between the first peer finishing downloading and the second
finishing downloading a block was mined, we need to pick a different
peer for the block the first one didn't download.
With the expansion of the database more agressive, checking a
larger set of peers for one that works becomes more important, as
such open more connections ever cycle if we can't find good ones
after half a minute.
With the expansion of the database more agressive, checking a
larger set of items for the best one becomes more important, as
such do a 10x for finding the best sccoring item.
The observed effect is finding previously useful peers within seconds
instead of a minute.
When our addresses database is filled with old or wrong addresses, we
end up trying to connect to a large number of IPs before we find actual
peers.
When we see that is the case, let's try to find more IP addresses from
our peers.
When we didn't personally verify the services, don't judge the IP based
on them and connect anyway.
In addition to not trusting the services we receive from the net, be
more careful with the usage of the 'lastConnected' field.
Notice that we don't actually use the result of that variable in the end
if the everConnected is likewise incorrect.
this introduces a new version of the address-db, as stored in the db
file itself. Causing a one-time 'upgrade'.
Main issue solved is that the 'everSeen' bool was restored incorrectly
and from then on out saved wrong too.
This may have caused bad selection of peers to connect to as the DB
grew.
Together with the various bugs in the last months update we also reset
the punishment to not avoid connecting to possibly perfectly fine peers.
A connection that is 'shutdown' now ensures one more time we'll call
onDisconnected() (is it was..) allowing the application level logic to
get notified and avoid memory leaks.
tomFlowee