193 Commits

Author SHA1 Message Date
TomZ 3b553fd80b Make sure we handle a close() correctly.
When we close() the receiveStream is cleared and we should not be
reading from it after. So detect and handle this.
2020-09-17 21:51:37 +02:00
TomZ 7afefefa41 Improve NWM flow-control
The NetworkManager now has more mature flow-control which means that we
send our buffers more regularly and we delay parsing received data if the
sending takes too long.

The direct effect should be that creating of a large number of messages
in response to incoming messages is no longer a problem in many cases.
2020-09-17 21:04:28 +02:00
TomZ 84ee1bc3fe Minor: comments and whitespace. 2020-09-17 20:53:56 +02:00
TomZ f0375c09a7 Fix bad memory access on sending prio message
Don't ask the size of an unitialized buffer.
2020-09-17 20:51:06 +02:00
TomZ 713697525a A network buffer maximum size of 6MB sounds fine for a server.
This is for the connection to the hub/indexer, so not for the incoming
clients.
Increasing the size allows much larger traffic without delay.
2020-09-02 13:47:32 +02:00
TomZ bd2a06a9ff Add a save to match the load Addr. 2020-05-11 13:24:06 +02:00
TomZ 79a92cf2ce NWM: work better with large concurrency
The asio stuff is quite broken when it comes to, ironically,
concurrency.
Connecting to peers lead to lots of 'canceled' (sic) errors being
generated internally in asio and other such issues.
Even though they were called on separate objects.

Adding a mutex around the async_connect calls made the amount of
failures go down dramatically.

This patch also includes upgrading boost::mutex to the std-lib
versions.
2020-05-11 13:19:10 +02:00
TomZ ed5b100ca5 Delete ununsed connections on shutdown. 2020-05-10 22:03:22 +02:00
TomZ 0b3dc7a8f5 NWM: fix reconnects/reuse of connection 2020-05-10 16:02:14 +02:00
TomZ 00453d52c4 NWM: Reuse connection objects
Avoid deleting and creating connections, which is really a rather rare
event anyway, and simply remember the connection object and reuse it
when a new one is requested.
2020-05-10 16:01:08 +02:00
TomZ d49b1ae5e2 Make NetworkManager connections leaner
The NetworkManager usage was mostly for low connection counts and this
made defaults selection easy.
With more usages it is important to allow the NWM-connection to be more
configurable about memory usage and leaner in general.

This changes the headers-buffers (used to create envelopes) to not be per
connection anymore but per thread using the tread_local keyword.

This changes the ring-buffers to become configurable using
NetworkConnections::setMessageQueueSizes().

Also removing some include statements where they were not really needed
in the P2PNet lib.
2020-05-09 20:23:05 +02:00
TomZ 5bb095f8a2 Minor fixes in usage of shared pointers. 2020-05-07 18:26:41 +02:00
TomZ 9a78e770ac P2PNet: handle errors better
This changes handling to also remove peers that fail with:
  "Cannot assign requested address"
An error seen in a Docker container on a system supporting IPv6.

Additionally we now always assign a error level so an unknown error will
still cause the node to be banned after 5 attempts.
2020-05-07 16:07:12 +02:00
TomZ 488e9b4322 Add missing dependency 2020-05-07 14:04:57 +02:00
TomZ fb6f91bf79 Simplify p2pnet ownership
Make the Peer object be owned by a shared pointer.
2020-05-05 22:53:25 +02:00
TomZ bf76c5ab36 Write all bytes after command to zero.
Instead of just a single trailing zero, write all bytes of the
command-section. This is required by the protocol.
2020-05-05 10:54:26 +02:00
TomZ 3f16f6224a Fix the threading model in the P2PNet
This allows deletion of peers also from other threads than their own
assigned ones.
2020-05-03 21:02:45 +02:00
TomZ e5ee37fd87 Fix larger messages in LegacyP2P mode.
Splitting up the bigger message in chunks is not supported on the legacy
p2p network, so don't attempt.
2020-04-16 17:12:57 +02:00
TomZ 436535647f Add several new features to networkmanager
Allow user to shutdown a connection, making it instantly invalid.
Allow user to register a callback for errors.
And fix pinging to be disabled on legacyP2P style connections.
2020-04-01 20:24:04 +02:00
TomZ 24e557f566 Stop double applying of offset.
We already removed the bytes in Message::header()
2020-04-01 20:22:55 +02:00
TomZ 655179e660 Add EndPoint::from() from a byte-array
This is used on the p2p layer and this allows simple
integration.
2020-04-01 20:22:55 +02:00
TomZ 7b1b742036 Remove workaround for old boost version
The API changes in boost between 1.66 and later was the need
for the boost_compat.h header file.
Its been a long time since Flowee started demanding 1.67 minimum
for boost, making this compat obsolete.
2020-03-29 12:55:12 +02:00
TomZ bf24ed042b Add support for legacy p2p network envelope in networkmanager 2020-03-29 12:55:12 +02:00
TomZ e042fb90f5 Cancel timer on shutdown 2020-01-12 18:05:09 +01:00
TomZ aef124e523 Merge branch '2019.10' 2019-12-17 17:58:34 +01:00
TomZ 1143a71a92 Fix assignment typo 2019-12-17 17:57:08 +01:00
TomZ f4d8870001 NWM: tweaks to logs. 2019-12-17 17:36:15 +01:00
TomZ 8c842fa108 Catch exceptions on usage of socket
This should fix the issue where the NWM stopped listening due to us not
calling the proper setupCallback()

Also add some logging.
2019-12-11 23:47:21 +01:00
TomZ 152efb2c5d Print more details on making NWM connection. 2019-11-29 20:09:28 +01:00
TomZ a471d919eb Improve logging of not-acceptable connections
Log IP of remote that tries to talk to us but doesn't know the protocol.
2019-11-29 10:59:58 +01:00
TomZ e7ad998d57 Add -fPIC to installed libs 2019-11-04 14:20:50 +01:00
TomZ 83d455a5d4 Help devops with more info. 2019-08-29 22:33:57 +02:00
TomZ 191bc3758d Add a simpliefied NetworkManager::bind() method
One that doesn't require a callback to be passed in.
2019-08-24 22:25:10 +02:00
TomZ df7de2ecd2 Clean up the hairy ball of util.cpp
Move some globals and all stuff pulling in crypto to the
server/serverutil.cpp file

Remove dead code.

Move several items that were used in only one place to the respective
files using them.

Move the class WaitUntilFinishedHelper into its own file.

Made sure no header includes utils.h
remove a handfull of files including utils.h for nothing.
2019-08-24 22:25:09 +02:00
TomZ 9faf50029e Fix typo 2019-08-24 22:25:09 +02:00
TomZ 2bb06afa9e Cleanup the link-lines 2019-08-24 22:25:09 +02:00
TomZ f311073312 Avoid DNS lookups and better logging
When users type an IP address as hostname we optimistically try
to parse them in order to avoid asking the DNS anyway.
2019-08-19 21:43:08 +02:00
TomZ a56b67caf2 NWM; Create a new socket before reconnect 2019-06-22 10:59:47 +02:00
TomZ fca434b1a6 Make Indexer connect to Hub after loading the conf
Doing it before could lead to incoming data being
processed before the SQL database connection was setup.
2019-06-13 17:12:28 +02:00
TomZ e5233fba32 On close, reset the chunked internal data. 2019-06-13 12:21:38 +02:00
TomZ 0a916c21e9 Add UTXO support to APIs
Generically this looks like this;

IsUnspent
[
  { "txid": "0x111111", "output": 1 },
  { "txid": "0x111111", "output": 11 }
]
IsUnspentReply
[
  { "blockHeight": 1111, "offsetInBlock": 1111, "unspent": true },
  { "unspent": false }
]

GetUnspentOutput
[
  { "txid": "0x111111", "output": 1 },
  { "txid": "0x111111", "output": 0 }
]
GetUnspentOutputReply
[
  { "blockHeight": 1111, "offsetInBlock": 1111, "unspent": true,
    "amount": 1111, "outputscript": "bytearray" },
  { "unspent": false }
]

note: txids are naturally always sha256s.
2019-06-10 21:02:15 +02:00
TomZ e168e3904d Respond to error. 2019-06-07 09:53:05 +02:00
TomZ 99e7f8656c Make headers go to nice subdirs 2019-06-07 09:53:05 +02:00
TomZ ec2610139e Allow setting indexer connections in conf file
This adds the ability to read from the config file the network
connections. Both the listening address and port as well as the upstream
'hub' address.
2019-06-06 19:44:16 +02:00
TomZ 0c9f415eb8 Several NetworkManager fixes
This adds all the header ints to chunked messages, solving the problem
of losing RequestId on roundtrips if the answer was too big.

Added a unit test for this "new" features.

This makes callbacks all use shared_from_this() in order to avoid
callbacks being done on deleted instances (thanks boost!).

Last, special case when the user doesn't connect but just sends messages
which caused a send of data, only to realize the connection wasn't open,
and then a connect.
This makes sure we immediately start a connect on queue of a message.
2019-06-04 21:40:02 +02:00
TomZ 57b0b58aa7 Add code to stream a NetworkEndPoint to Logger. 2019-06-03 21:41:04 +02:00
TomZ ca2d6b041a fix connection() not finding an existing connection 2019-06-03 14:37:08 +02:00
TomZ a00938a118 Fix race condition in shutdown of NetworkConnection 2019-06-03 14:37:08 +02:00
TomZ 50870520df Fix NetworkConnection::isConnected() behaviour
This brings it closer to what we expect.
2019-06-03 14:36:59 +02:00
TomZ a1dfd1cde9 Work on installable headers 2019-06-03 10:59:33 +02:00