The download from a peer is now limited to just 4000 blocks, at which
point the download proceeds from another peer.
Naturally it can go back to the first afterwards too.
The idea is that we download a month worth of blocks at a time and then
fetch the same blocks from a backup peer in order to be certain we
actually see the real state.
The problem we've seen is that after the first downloads we end up
starting the second and if the initial sync or a similar large download
happens, many people won't leave it running until the backup actually
completes since the UI looks like its finished.
This is also a good starting point for rating peers at download speed by
having predictable chunks and that means we could time and aim to use
the faster peers.
This is becoming relevant as we can see a huge difference where peers
give you 10x performance compared to some of the slower ones.
Every peer that we connect to should get the bloom filter set and it
should also get the 'mempool' call sent once which will make that peer
respond with all matches of the bloom filters which are in the mempool.
The tricky part is that we should have the latest bloom filter set on
each of those peers before the mempool is sent, since they work
together.
Also if we are behind, the mempool should not be sent. (It can cause
problems in the wallet if we receive unconfirmed transactions before
mined transactions)
So, when a peer starts downloading merkle blocks, the bloom filter of all
the other peers becomes invalid the moment a match is found by an actual
wallet.
This is Ok on one peer because the merkleblock automatically updates the
filter on match on the server side, but obviously not on the other peers
we have for that wallet.
The approach we follow is that as soon as a sync-run is done on a single
peer (we do a main and also a backup sync), we tell the wallet to re-
build its bloom filter for _all_ peers and if the sync that peer did
leads us to be at the chain-tip, we also send each peer the mempool
command.
Tuesdays idea of adding some code into the SyncSPVAction didn't feel
right.
A second look made clear that bloom filter updates make much more sense
to go hand in hand with sending a mempool message. Especially since they
depend on each other on the server side.
To-rehash:
the wallet may decide at any time that a new bloom filter is needed. It
then uses the superclass (code in p2plib) PrivacySegment, to build that
filter. As part of that we get a lock object which, when going out of
scope, makes the peers that are subscribed to the privacySegment send
out the filter.
This separation of concerns means that the subclass wallet in the app
doens't know about peers or messages, only its superclass PrivacySegment
does.
What we did in this change is make the PrivacySegment class decide to
combine a bloom update with a mempool call. Typicall only once per
connection.
This means I can remove hacks in the SyncSPVAction which forced the
sending of the mempool message separately.
This is done in several steps:
1. Separate my height from the remote peer heights.
Instead of assuming we have one height, recognize that a peer may
not be at the tip at the same time we are. We monitor headers/invs
to update the 'peerHeight' variable.
2. Ask for merkle blocks from a peer to the maximum height of that
peer (but not later than what we validated to be true).
This avoids us asking past the remotes tip which they didn't like.
3. Redo the SyncSPVAction to use all this and make it much more
reliable in finding peers to download from and getting all the
changes as fast as possible.
To send out transactions in the p2p net is quite a lot of work,
you need to find multiple peers to send the transaction to. First
you send an INV, then you respond to a getData to actually send
the transaction and last you wait for 'reject' messages that may
indicate that there is something wrong with the transaction.
This introduces the BroadcastTxData class that wraps a transaction
and gets callbacks for sending and for rejects, abstracting away
all the complexity for the user.
Instead of forwarding one transaction at a time as the peer sends them
to us, bundle them in a group of transactions known to be merkle-checked
and all belonging together in one block.
Since the peer has no obligation (and with CTOR even less) to send the
transactions in natural order, we should get them per block so we know
all transactions forwarded have parents.
This avoids a peer once sending acceptable headers and never
being bothered again. Instead we now check regularly and keep
track of when the peer was known to follow the same chain as us.
The NetworkManager usage was mostly for low connection counts and this
made defaults selection easy.
With more usages it is important to allow the NWM-connection to be more
configurable about memory usage and leaner in general.
This changes the headers-buffers (used to create envelopes) to not be per
connection anymore but per thread using the tread_local keyword.
This changes the ring-buffers to become configurable using
NetworkConnections::setMessageQueueSizes().
Also removing some include statements where they were not really needed
in the P2PNet lib.
This avoids a race condition on remove/delete of peer where
the connection manager decides to delete a peer while in its own thread
the peer is processing a package.
This moves deletion of the peer to its own strand.
We reuse the NetworkManager lower level code in order to connect
to the Bitcoin P2P network.
This implements the basics for anyone wanting to be a player on
this network.
tomFlowee
TomZ