We always need two random peers to download a certain merkle block, to
avoid the withholding attack.
It occurred to me that there is no need to do them one after the other,
they can both run in parallel without issues. Saving time.
This is an abstract class that the application using this library needs
to subclass. Ownership and lifetime don't change, it still lies with the
app using the library and they still need to add and remove it from the
connectionManager, but this makes it much more stable for multi-
threading environments and avoids issues on misuse.
The download from a peer is now limited to just 4000 blocks, at which
point the download proceeds from another peer.
Naturally it can go back to the first afterwards too.
The idea is that we download a month worth of blocks at a time and then
fetch the same blocks from a backup peer in order to be certain we
actually see the real state.
The problem we've seen is that after the first downloads we end up
starting the second and if the initial sync or a similar large download
happens, many people won't leave it running until the backup actually
completes since the UI looks like its finished.
This is also a good starting point for rating peers at download speed by
having predictable chunks and that means we could time and aim to use
the faster peers.
This is becoming relevant as we can see a huge difference where peers
give you 10x performance compared to some of the slower ones.
If between the first peer finishing downloading and the second
finishing downloading a block was mined, we need to pick a different
peer for the block the first one didn't download.
With the expansion of the database more agressive, checking a
larger set of peers for one that works becomes more important, as
such open more connections ever cycle if we can't find good ones
after half a minute.
When our addresses database is filled with old or wrong addresses, we
end up trying to connect to a large number of IPs before we find actual
peers.
When we see that is the case, let's try to find more IP addresses from
our peers.
This changes the SPV action to not actually exit when sync
is completed, but instead keep running in the background
so it will detect when a wallet loses a peer and reinstate one.
The PeerAddressDB keeps a score for each IP/port we have about how
useful that peer is.
This change re-visits the way we use the DB where we now are more fluid
in our approach to scores. We no longer simply forever-ban a peer after
it made a mistake once.
This rewrites the select-a-peer algo to take timing into account,
preferring a peer that we connected to recently while putting a higher
score on a peer that failed a long time ago so if we start running out
of good peers we start on the not-so-great-but-maybe-improved first.
Good behavior is likewise rewarded so a peer can become higher prio
again, but generally a new IP still gets preferred. Again, the balance
is to keep not-so-great options in the running.
Every peer that we connect to should get the bloom filter set and it
should also get the 'mempool' call sent once which will make that peer
respond with all matches of the bloom filters which are in the mempool.
The tricky part is that we should have the latest bloom filter set on
each of those peers before the mempool is sent, since they work
together.
Also if we are behind, the mempool should not be sent. (It can cause
problems in the wallet if we receive unconfirmed transactions before
mined transactions)
So, when a peer starts downloading merkle blocks, the bloom filter of all
the other peers becomes invalid the moment a match is found by an actual
wallet.
This is Ok on one peer because the merkleblock automatically updates the
filter on match on the server side, but obviously not on the other peers
we have for that wallet.
The approach we follow is that as soon as a sync-run is done on a single
peer (we do a main and also a backup sync), we tell the wallet to re-
build its bloom filter for _all_ peers and if the sync that peer did
leads us to be at the chain-tip, we also send each peer the mempool
command.
When the SyncChainAction was written, various interactions we do
today in the ConnectionManager did not happen yet. Features in
Peer didn't exist yet.
This updates the SyncChainAction to take those items into account
and be more responsive and conclue we are 'up to date' faster,
while also leaving behind a better state.
Tuesdays idea of adding some code into the SyncSPVAction didn't feel
right.
A second look made clear that bloom filter updates make much more sense
to go hand in hand with sending a mempool message. Especially since they
depend on each other on the server side.
To-rehash:
the wallet may decide at any time that a new bloom filter is needed. It
then uses the superclass (code in p2plib) PrivacySegment, to build that
filter. As part of that we get a lock object which, when going out of
scope, makes the peers that are subscribed to the privacySegment send
out the filter.
This separation of concerns means that the subclass wallet in the app
doens't know about peers or messages, only its superclass PrivacySegment
does.
What we did in this change is make the PrivacySegment class decide to
combine a bloom update with a mempool call. Typicall only once per
connection.
This means I can remove hacks in the SyncSPVAction which forced the
sending of the mempool message separately.
The job description of this component is to sync a wallet. Its means to
that end is to connect to peers.
Problem is that when we startup and the whole system is already in-sync,
then we forget to connect to peers. (that wasn't in the job description,
it just happened to be done as a sideeffect).
This changes the SyncSPVAction's job description to also make sure that
each wallet has the required number of peers.
We send a 'mempool' p2p command to a peer after we did a series of
merkleblock downloads.
This code adds the mempool p2p command to be sent to peers that didn't
participate in the merkleblock download and thus get the best out of all
our connected peers.
This is done in several steps:
1. Separate my height from the remote peer heights.
Instead of assuming we have one height, recognize that a peer may
not be at the tip at the same time we are. We monitor headers/invs
to update the 'peerHeight' variable.
2. Ask for merkle blocks from a peer to the maximum height of that
peer (but not later than what we validated to be true).
This avoids us asking past the remotes tip which they didn't like.
3. Redo the SyncSPVAction to use all this and make it much more
reliable in finding peers to download from and getting all the
changes as fast as possible.
We connect to the "first" priority stated segments first, in order to
allow the UX to be made much better since it may take a bit of time to
find peers.
We reuse the NetworkManager lower level code in order to connect
to the Bitcoin P2P network.
This implements the basics for anyone wanting to be a player on
this network.
tomFlowee
TomZ