Commit Graph

118 Commits

Author SHA1 Message Date
TomZ 425ee250cb Fixlets from static code inspector 2019-09-12 15:21:48 +02:00
TomZ 83d455a5d4 Help devops with more info. 2019-08-29 22:33:57 +02:00
TomZ b6c4ac6f88 Fixes in SQL speed
This explicitly detects the initial sync and refuses to create
any indexes on the SQL databases during this massive insert stage.

Some other minor issues got fixed here too.
2019-08-29 22:33:57 +02:00
TomZ 191bc3758d Add a simpliefied NetworkManager::bind() method
One that doesn't require a callback to be passed in.
2019-08-24 22:25:10 +02:00
TomZ c3ddab9cd4 Make Indexer bind to localhost as default 2019-08-24 22:25:10 +02:00
TomZ 2bb06afa9e Cleanup the link-lines 2019-08-24 22:25:09 +02:00
TomZ da272d7d7f Fix SQL usage for postgres & better logging 2019-08-19 21:42:43 +02:00
TomZ d26386965a Fix defaults
This solves the issue that unless you explicitly turn off the txdb you
still get it.
2019-08-18 16:23:32 +02:00
TomZ 30bee6bf08 Make connecting a higher prio log 2019-08-16 23:39:40 +02:00
TomZ 691ff90014 Avoid uninit data from entering DB
Also fix typo in variable name
2019-08-15 10:55:41 +02:00
TomZ d89f38e4fb More fixlets in indexer after to the rewrite 2019-08-13 23:19:08 +02:00
TomZ 0fdba5a404 Add special postgres SQL statements
When the user selected postgres, use specific SQL statements that are
optimised for it.
2019-08-13 22:21:22 +02:00
TomZ d03be86166 Make indexer follow the tip again
The rewrite of the synching code broke this, so fixing.
2019-08-13 16:19:31 +02:00
TomZ 53c9badc51 Use atomics in the proper way. 2019-08-09 19:53:13 +02:00
TomZ a1ee211968 make compile with older Qt 2019-06-26 22:50:20 +02:00
TomZ 896f5e7b68 Merge branch 'master' into MessageBuilder::Reply 2019-06-26 22:20:44 +02:00
TomZ 98b3e7cece use new MessageBuilder::reply() 2019-06-26 22:19:28 +02:00
TomZ cb5ff404d3 Refactor the Indexer application
This moves each indexer to live in its own thread, making blocking
operations allowed and Ok and separate from incoming requests.
2019-06-26 21:39:07 +02:00
TomZ 6337b1b93b Create indexes 2019-06-19 16:55:51 +02:00
TomZ 9f041cf926 Move handling of address-db requests to another thread
A slow SQL database means we should not do this on the networkmanager
thread as that causes issues.
2019-06-19 16:55:47 +02:00
TomZ a6888b74d8 Need 2 connections to do multithreading
Lets open 2 SQL connections to allow the bulk inserts
to happen in a different thread from the selects we do
resulting from incoming questions.
2019-06-19 16:55:39 +02:00
TomZ fca434b1a6 Make Indexer connect to Hub after loading the conf
Doing it before could lead to incoming data being
processed before the SQL database connection was setup.
2019-06-13 17:12:28 +02:00
TomZ 00963dd31b Make parsing indexer.conf a bit more verbose
And init our new variables.
2019-06-12 23:53:26 +02:00
TomZ 230611b98e Tweak the numbers to be better on IBD 2019-06-12 23:53:24 +02:00
TomZ d172dc4063 Don't fetch data when the node has an older chain.
For instance when its reindexing.
2019-06-11 21:43:50 +02:00
TomZ 9e69a3fef9 Add new indexer; spentoutput
where the UTXO allows you to find an output based on txid+output, the
spentoutput DB turns that around and allows you to find an input that
spends a certain output.

This also fits perfectly find into the UTXO database class, so this
creates a new dir "spent" which is filled as expected.
2019-06-11 17:38:30 +02:00
TomZ a7f4e942f9 Add README for indexer 2019-06-06 20:15:01 +02:00
TomZ ec2610139e Allow setting indexer connections in conf file
This adds the ability to read from the config file the network
connections. Both the listening address and port as well as the upstream
'hub' address.
2019-06-06 19:44:16 +02:00
TomZ 3c659855db Merge branch '2019.05' 2019-06-06 17:47:19 +02:00
TomZ c5aefb0722 Various small fixes in indexer-cli 2019-06-06 17:43:27 +02:00
TomZ 0c9f415eb8 Several NetworkManager fixes
This adds all the header ints to chunked messages, solving the problem
of losing RequestId on roundtrips if the answer was too big.

Added a unit test for this "new" features.

This makes callbacks all use shared_from_this() in order to avoid
callbacks being done on deleted instances (thanks boost!).

Last, special case when the user doesn't connect but just sends messages
which caused a send of data, only to realize the connection wasn't open,
and then a connect.
This makes sure we immediately start a connect on queue of a message.
2019-06-04 21:40:02 +02:00
TomZ feb73bd1a8 Improve loglevel and naming. 2019-06-03 21:38:21 +02:00
TomZ a1dfd1cde9 Work on installable headers 2019-06-03 10:59:33 +02:00
TomZ 7203a058a3 Change API call SplitHostPort slightly
Instead of taking an int for the port, take a short.
This matches better with the networkmanager EndPoint struct.
2019-06-02 20:16:49 +02:00
TomZ 1f52ae7a7f FloweeServiceApplication: support net wildcard
For service applications that bind to networks (currently
only indexer) this allows users to pass the address:
    0.0.0.0
to bind to. The effect will be that we find all the network
interfaces in the system and bind to all of them.
Notice that we already had "localhost" as an alias to bind to both
v4 and v6 loopback interfaces.

This new feature depends on optional dependency QtNetwork.
2019-05-17 12:32:11 +02:00
TomZ ffb4196434 Add conf and systemd files to install 2019-05-13 12:20:57 +02:00
TomZ 5b155600d9 Make app arguments more consitent
Now everyone auto-connects to localhost unless passed the
option --connect=

Also make indexer find the logs.conf from the same dir as the
commandline passed config file.

This makes running it as a service more useful.
2019-05-09 12:16:11 +02:00
TomZ 8d81cc5471 Create new lib apputils
This lib depends on QtCore, as such it is separated from other libs. As
many apps depend on QtCore we can share classes here to avoid
duplication.
2019-05-07 20:53:22 +02:00
TomZ 67865e4615 Add support to print the version. 2019-04-13 16:23:02 +02:00
TomZ edd4cada30 Make compile on older Qt
The method I choose was introduced in Qt 5.8.
This change makes it work on any Qt5 based release.
2019-04-11 19:59:54 +02:00
TomZ ed7ca33d6a Add decoding and printing addresses work.
This now supports cashaddress, legacy addresses and it prints
basic info about the usages of said address.
2019-04-11 18:45:28 +02:00
TomZ dad38b66c6 Flush indexer to SQL database when we reach Tip() 2019-04-11 00:01:54 +02:00
TomZ cec0c28be3 Make indexer subscribe to block updates
When an indexer is in-sync it will stop actively asking for
new blocks, so the block updates (Hub saying they found a new block)
will then allow the indexer to keep up-to-date
2019-04-10 22:17:13 +02:00
TomZ 78219d5bba Make the indexer-cli and indexer interact well
This adds lots of little things;
* Add GetTransaction API call
* I refactored the GetBlock API a little to reuse code.
* a new 'Version' API call for the hub
* API for the logging manager, so we can set a default
  setup with just C++-APIs
* various (usability) fixes in the FloweeServiceApplication
* Binding to localhost attempts to bind to both IPv4 and v6
* Print the actual transaction hex from indexer-cli (which really
  is just a testing app)
2019-04-10 20:52:04 +02:00
TomZ aea9c25c15 Make indexer-cli have more suitable output 2019-04-10 15:38:37 +02:00
TomZ 3e58679108 Add a new app indexer-cli
This is frankly just a testing application at the moment, but you
can connect to a Indexer server and get the information out, so its
at least useful.
2019-04-10 14:32:46 +02:00
TomZ 76fdd25b51 Avoid exit without save
Only throw at startup when SQL problems occur.
When they occur later ask for a clean shutdown so we can allow other
parts to shutdown cleanly.
2019-04-09 20:22:35 +02:00
TomZ 20694d4f33 Make indexer be a server too 2019-04-09 19:34:55 +02:00
TomZ 50b65d054b Cleanups and fixlets 2019-04-09 14:49:08 +02:00
TomZ 07f3a57f73 Fix misuse of std::sort API
Its not clear from my reading of the docs, but the std::sort version
that takes a struct actually makes a deep copy of the struct for each
sort. Which means that this usage is a really bad idea if you were
actually storing the data on this struct.
2019-04-09 12:02:45 +02:00