This is an abstract class that the application using this library needs
to subclass. Ownership and lifetime don't change, it still lies with the
app using the library and they still need to add and remove it from the
connectionManager, but this makes it much more stable for multi-
threading environments and avoids issues on misuse.
The download from a peer is now limited to just 4000 blocks, at which
point the download proceeds from another peer.
Naturally it can go back to the first afterwards too.
The idea is that we download a month worth of blocks at a time and then
fetch the same blocks from a backup peer in order to be certain we
actually see the real state.
The problem we've seen is that after the first downloads we end up
starting the second and if the initial sync or a similar large download
happens, many people won't leave it running until the backup actually
completes since the UI looks like its finished.
This is also a good starting point for rating peers at download speed by
having predictable chunks and that means we could time and aim to use
the faster peers.
This is becoming relevant as we can see a huge difference where peers
give you 10x performance compared to some of the slower ones.
When we didn't personally verify the services, don't judge the IP based
on them and connect anyway.
In addition to not trusting the services we receive from the net, be
more careful with the usage of the 'lastConnected' field.
Notice that we don't actually use the result of that variable in the end
if the everConnected is likewise incorrect.
This allows us to use a shared pointer while at the same time not having
the problem that a peer de-registration hits an already deleted
PrivacySegment.
Last year we started wrapping the Peer object in a shared pointer,
which is used now in the managers.
This continues the idea by making the PrivacySegment use smart
pointers too and the P2PNetInterface is changed to do the same
for downstream applications.
As we moved most of the creation of a BufferPool to be via the
Streaming::pool() method, which uses a thread-local, it makes sense
to start cleaning up the design and make it more modern C++.
The above mentioned method would return a reference and you'd see
loads of places use `auto &pool =` which is less than ideal.
As the number of places where we actually instantiate a BufferPool
goes down, the usage of some sort of smart pointer makes more sense.
This now makes all APIs use BufferPool be wrapped in a shared_ptr.
Should the peer have been rejected on status or similar, we don't
register a successful connection and avoid the peer from being selected
again since then we don't remove the punishment done on connection
start.
We removed the default value in order to avoid misusage of the method
and mistaking the amount with the connection-id.
From Peer we now se the faster overload of punish(), avoiding looping
through the peer list.
Also check return-code when needed.
Every peer that we connect to should get the bloom filter set and it
should also get the 'mempool' call sent once which will make that peer
respond with all matches of the bloom filters which are in the mempool.
The tricky part is that we should have the latest bloom filter set on
each of those peers before the mempool is sent, since they work
together.
Also if we are behind, the mempool should not be sent. (It can cause
problems in the wallet if we receive unconfirmed transactions before
mined transactions)
So, when a peer starts downloading merkle blocks, the bloom filter of all
the other peers becomes invalid the moment a match is found by an actual
wallet.
This is Ok on one peer because the merkleblock automatically updates the
filter on match on the server side, but obviously not on the other peers
we have for that wallet.
The approach we follow is that as soon as a sync-run is done on a single
peer (we do a main and also a backup sync), we tell the wallet to re-
build its bloom filter for _all_ peers and if the sync that peer did
leads us to be at the chain-tip, we also send each peer the mempool
command.
When we receive a known message (currently just avahello) we now
intantly ban and disconnect the peer, no need to
test compliance when they openly greet us saying they are not following
the same chain.
Tuesdays idea of adding some code into the SyncSPVAction didn't feel
right.
A second look made clear that bloom filter updates make much more sense
to go hand in hand with sending a mempool message. Especially since they
depend on each other on the server side.
To-rehash:
the wallet may decide at any time that a new bloom filter is needed. It
then uses the superclass (code in p2plib) PrivacySegment, to build that
filter. As part of that we get a lock object which, when going out of
scope, makes the peers that are subscribed to the privacySegment send
out the filter.
This separation of concerns means that the subclass wallet in the app
doens't know about peers or messages, only its superclass PrivacySegment
does.
What we did in this change is make the PrivacySegment class decide to
combine a bloom update with a mempool call. Typicall only once per
connection.
This means I can remove hacks in the SyncSPVAction which forced the
sending of the mempool message separately.
Version.h held mostly stuff for protocol.h, which is a hub-specific file.
The only thing that we actually use is the PROTOCOL_VERSION in our code
and as such that one moved to the interfaces dir.
In HD wallets we get an issue where not-yet-generated keys are needed in
the merkle-block request and we notice this as a chunk of blocks is
being downloaded.
This is solved with some extra code.
Also avoid re-uploading the bloom filter during initial sync.
This is done in several steps:
1. Separate my height from the remote peer heights.
Instead of assuming we have one height, recognize that a peer may
not be at the tip at the same time we are. We monitor headers/invs
to update the 'peerHeight' variable.
2. Ask for merkle blocks from a peer to the maximum height of that
peer (but not later than what we validated to be true).
This avoids us asking past the remotes tip which they didn't like.
3. Redo the SyncSPVAction to use all this and make it much more
reliable in finding peers to download from and getting all the
changes as fast as possible.
To send out transactions in the p2p net is quite a lot of work,
you need to find multiple peers to send the transaction to. First
you send an INV, then you respond to a getData to actually send
the transaction and last you wait for 'reject' messages that may
indicate that there is something wrong with the transaction.
This introduces the BroadcastTxData class that wraps a transaction
and gets callbacks for sending and for rejects, abstracting away
all the complexity for the user.
Instead of forwarding one transaction at a time as the peer sends them
to us, bundle them in a group of transactions known to be merkle-checked
and all belonging together in one block.
Since the peer has no obligation (and with CTOR even less) to send the
transactions in natural order, we should get them per block so we know
all transactions forwarded have parents.
This avoids a peer once sending acceptable headers and never
being bothered again. Instead we now check regularly and keep
track of when the peer was known to follow the same chain as us.
The NetworkManager usage was mostly for low connection counts and this
made defaults selection easy.
With more usages it is important to allow the NWM-connection to be more
configurable about memory usage and leaner in general.
This changes the headers-buffers (used to create envelopes) to not be per
connection anymore but per thread using the tread_local keyword.
This changes the ring-buffers to become configurable using
NetworkConnections::setMessageQueueSizes().
Also removing some include statements where they were not really needed
in the P2PNet lib.
tomFlowee
TomZ