Commit Graph

103 Commits

Author SHA1 Message Date
TomZ 896f5e7b68 Merge branch 'master' into MessageBuilder::Reply 2019-06-26 22:20:44 +02:00
TomZ 98b3e7cece use new MessageBuilder::reply() 2019-06-26 22:19:28 +02:00
TomZ cb5ff404d3 Refactor the Indexer application
This moves each indexer to live in its own thread, making blocking
operations allowed and Ok and separate from incoming requests.
2019-06-26 21:39:07 +02:00
TomZ 6337b1b93b Create indexes 2019-06-19 16:55:51 +02:00
TomZ 9f041cf926 Move handling of address-db requests to another thread
A slow SQL database means we should not do this on the networkmanager
thread as that causes issues.
2019-06-19 16:55:47 +02:00
TomZ a6888b74d8 Need 2 connections to do multithreading
Lets open 2 SQL connections to allow the bulk inserts
to happen in a different thread from the selects we do
resulting from incoming questions.
2019-06-19 16:55:39 +02:00
TomZ fca434b1a6 Make Indexer connect to Hub after loading the conf
Doing it before could lead to incoming data being
processed before the SQL database connection was setup.
2019-06-13 17:12:28 +02:00
TomZ 00963dd31b Make parsing indexer.conf a bit more verbose
And init our new variables.
2019-06-12 23:53:26 +02:00
TomZ 230611b98e Tweak the numbers to be better on IBD 2019-06-12 23:53:24 +02:00
TomZ d172dc4063 Don't fetch data when the node has an older chain.
For instance when its reindexing.
2019-06-11 21:43:50 +02:00
TomZ 9e69a3fef9 Add new indexer; spentoutput
where the UTXO allows you to find an output based on txid+output, the
spentoutput DB turns that around and allows you to find an input that
spends a certain output.

This also fits perfectly find into the UTXO database class, so this
creates a new dir "spent" which is filled as expected.
2019-06-11 17:38:30 +02:00
TomZ a7f4e942f9 Add README for indexer 2019-06-06 20:15:01 +02:00
TomZ ec2610139e Allow setting indexer connections in conf file
This adds the ability to read from the config file the network
connections. Both the listening address and port as well as the upstream
'hub' address.
2019-06-06 19:44:16 +02:00
TomZ 3c659855db Merge branch '2019.05' 2019-06-06 17:47:19 +02:00
TomZ c5aefb0722 Various small fixes in indexer-cli 2019-06-06 17:43:27 +02:00
TomZ 0c9f415eb8 Several NetworkManager fixes
This adds all the header ints to chunked messages, solving the problem
of losing RequestId on roundtrips if the answer was too big.

Added a unit test for this "new" features.

This makes callbacks all use shared_from_this() in order to avoid
callbacks being done on deleted instances (thanks boost!).

Last, special case when the user doesn't connect but just sends messages
which caused a send of data, only to realize the connection wasn't open,
and then a connect.
This makes sure we immediately start a connect on queue of a message.
2019-06-04 21:40:02 +02:00
TomZ feb73bd1a8 Improve loglevel and naming. 2019-06-03 21:38:21 +02:00
TomZ a1dfd1cde9 Work on installable headers 2019-06-03 10:59:33 +02:00
TomZ 7203a058a3 Change API call SplitHostPort slightly
Instead of taking an int for the port, take a short.
This matches better with the networkmanager EndPoint struct.
2019-06-02 20:16:49 +02:00
TomZ 1f52ae7a7f FloweeServiceApplication: support net wildcard
For service applications that bind to networks (currently
only indexer) this allows users to pass the address:
    0.0.0.0
to bind to. The effect will be that we find all the network
interfaces in the system and bind to all of them.
Notice that we already had "localhost" as an alias to bind to both
v4 and v6 loopback interfaces.

This new feature depends on optional dependency QtNetwork.
2019-05-17 12:32:11 +02:00
TomZ ffb4196434 Add conf and systemd files to install 2019-05-13 12:20:57 +02:00
TomZ 5b155600d9 Make app arguments more consitent
Now everyone auto-connects to localhost unless passed the
option --connect=

Also make indexer find the logs.conf from the same dir as the
commandline passed config file.

This makes running it as a service more useful.
2019-05-09 12:16:11 +02:00
TomZ 8d81cc5471 Create new lib apputils
This lib depends on QtCore, as such it is separated from other libs. As
many apps depend on QtCore we can share classes here to avoid
duplication.
2019-05-07 20:53:22 +02:00
TomZ 67865e4615 Add support to print the version. 2019-04-13 16:23:02 +02:00
TomZ edd4cada30 Make compile on older Qt
The method I choose was introduced in Qt 5.8.
This change makes it work on any Qt5 based release.
2019-04-11 19:59:54 +02:00
TomZ ed7ca33d6a Add decoding and printing addresses work.
This now supports cashaddress, legacy addresses and it prints
basic info about the usages of said address.
2019-04-11 18:45:28 +02:00
TomZ dad38b66c6 Flush indexer to SQL database when we reach Tip() 2019-04-11 00:01:54 +02:00
TomZ cec0c28be3 Make indexer subscribe to block updates
When an indexer is in-sync it will stop actively asking for
new blocks, so the block updates (Hub saying they found a new block)
will then allow the indexer to keep up-to-date
2019-04-10 22:17:13 +02:00
TomZ 78219d5bba Make the indexer-cli and indexer interact well
This adds lots of little things;
* Add GetTransaction API call
* I refactored the GetBlock API a little to reuse code.
* a new 'Version' API call for the hub
* API for the logging manager, so we can set a default
  setup with just C++-APIs
* various (usability) fixes in the FloweeServiceApplication
* Binding to localhost attempts to bind to both IPv4 and v6
* Print the actual transaction hex from indexer-cli (which really
  is just a testing app)
2019-04-10 20:52:04 +02:00
TomZ aea9c25c15 Make indexer-cli have more suitable output 2019-04-10 15:38:37 +02:00
TomZ 3e58679108 Add a new app indexer-cli
This is frankly just a testing application at the moment, but you
can connect to a Indexer server and get the information out, so its
at least useful.
2019-04-10 14:32:46 +02:00
TomZ 76fdd25b51 Avoid exit without save
Only throw at startup when SQL problems occur.
When they occur later ask for a clean shutdown so we can allow other
parts to shutdown cleanly.
2019-04-09 20:22:35 +02:00
TomZ 20694d4f33 Make indexer be a server too 2019-04-09 19:34:55 +02:00
TomZ 50b65d054b Cleanups and fixlets 2019-04-09 14:49:08 +02:00
TomZ 07f3a57f73 Fix misuse of std::sort API
Its not clear from my reading of the docs, but the std::sort version
that takes a struct actually makes a deep copy of the struct for each
sort. Which means that this usage is a really bad idea if you were
actually storing the data on this struct.
2019-04-09 12:02:45 +02:00
TomZ df83dc8701 Upgrade HashStorage design phase2
Instead of merging a small list into a bigger and bigger list,
write lots of small lists (20MB each or so) and then when we hit
the wanted final size, merge all of those into one.
2019-04-09 11:19:03 +02:00
TomZ 1ce6a93984 Redo setup of HashStorage, phase 1 2019-04-08 13:19:54 +02:00
TomZ a3a456cdcd Try to get more speed out of the SQL DB
we chunk the data and make sure we create a transaction so autocommit
is turned off.
2019-04-06 19:00:55 +02:00
TomZ e14254f33d Make indexer load a config file for setup
Using standard ini file format.
This is the easiest way to configure it to connect to an external SQL
database.
2019-04-06 15:16:37 +02:00
TomZ ef5dafe6e0 Add private header warnings 2019-04-06 12:28:34 +02:00
TomZ dfb28874ca Adjust design by inverting hash
Additionally I split the sql data over multiple tables
to avoid a forever-growing one.
2019-04-05 23:11:37 +02:00
TomZ 5ed3419d9b Add beginnings of the AddressIndexer as well 2019-04-04 22:46:38 +02:00
TomZ 285b27d3b8 Add multiple databases concept to HashStorage 2019-04-04 22:46:38 +02:00
TomZ 88dd81d090 Make find() work properly 2019-04-03 18:55:33 +02:00
TomZ e5d5b9ec91 Merge branch 'master' into addressdb 2019-04-03 18:09:56 +02:00
TomZ 0447d3c9f6 Correctly detect and print finishing of indexer 2019-04-03 16:16:44 +02:00
TomZ 3afd867454 Add binary search to the HashStorage 2019-04-02 12:07:08 +02:00
TomZ 5767012a9b Make second finalize() work 2019-04-01 22:53:39 +02:00
TomZ 836a558bd2 Make hashstorage level 2 work a little 2019-04-01 21:15:24 +02:00
TomZ 4155557c2f Start working on the HashStorage class 2019-04-01 18:34:39 +02:00