137 Commits

Author SHA1 Message Date
TomZ 7d445b445b Add address indexer index. 2019-11-29 17:56:24 +01:00
TomZ 9e3b145b4e AddressIndexer: Wait for SQL DB
Instead of exiting when no SQL DB is found, sleep for 5 seconds and try
again.
2019-11-29 14:47:36 +01:00
TomZ b157ffba5f Update to also list new indexer 2019-11-29 11:07:54 +01:00
TomZ d573e307c3 Refactor; move SplitHostPort to utilstrencodings 2019-11-12 15:35:54 +01:00
TomZ 6cc749627d Merge branch '2019.09' 2019-11-11 21:12:18 +01:00
TomZ 0e8a4da185 Find address from newest to oldest. 2019-11-11 19:37:04 +01:00
TomZ 8dde1c86ed Avoid unneeded copying 2019-11-06 23:20:17 +01:00
TomZ 748cd54a8e Fix log line limiter
Avoid logging being the slow part.
2019-11-06 23:20:17 +01:00
TomZ 3c25232751 Exit when nothing is found. 2019-10-20 22:12:09 +02:00
TomZ 79e4974d19 Port API getblock::filter to use script-hash
The API call for GetBlock has a filter-on-address functionality which is
now ported to no longer use ripe160 addresses but instead uses output-
script hashes.
This avoids problems for transactions not using p2pkh and generally is a
cleaner solution.

This also adds a unit test to test this feature.
2019-10-20 22:12:09 +02:00
TomZ c4e255560b Port indexer client to use script-hash
This allows us to support p2sh properly.
2019-10-20 12:19:00 +02:00
TomZ 2396938cfb New version and only connect to new versions 2019-10-20 11:57:01 +02:00
TomZ 00814e24f9 API: Use hash of outscript instead of address
The usage of a ripe160 for bitcoin addresses in the API and in the
Indexer loses some info, specifically what kind of script it is.
Additionally not all types of scripts fit this mold. At best that means
its not future-proof.

This adds a method to the API in order to select from a Tx the hashed
outscript (thats singlehashed sha256) and refactor the address indexer
to use that instead of the ripe160 address.

The API enums broke a little, so I used the opportunity to break it a
lot and clean up the enums in order to make them more future-proof.
But, yeah, software from before this commit is protocol incompatible
with software after this commit.
2019-10-16 22:57:50 +02:00
TomZ e2209abcbb Make the slowness of the SQL DB show up in everyday logs. 2019-10-16 22:32:47 +02:00
TomZ 680437f16f Merge branch '2019.07' 2019-10-05 17:12:15 +02:00
TomZ 0e62640ba9 SQL DISTINCT is a keyword, make it DB agnostic 2019-10-02 17:41:36 +02:00
TomZ 5c5967c9cf Use Message::reply() and remove some boilerplate code. 2019-10-02 17:39:11 +02:00
TomZ 89c0f15955 Use Message::reply() and remove some boilerplate code. 2019-10-02 16:44:20 +02:00
TomZ 6b50a75053 Fix typo in comments 2019-09-13 15:12:54 +02:00
TomZ 425ee250cb Fixlets from static code inspector 2019-09-12 15:21:48 +02:00
TomZ 83d455a5d4 Help devops with more info. 2019-08-29 22:33:57 +02:00
TomZ b6c4ac6f88 Fixes in SQL speed
This explicitly detects the initial sync and refuses to create
any indexes on the SQL databases during this massive insert stage.

Some other minor issues got fixed here too.
2019-08-29 22:33:57 +02:00
TomZ 191bc3758d Add a simpliefied NetworkManager::bind() method
One that doesn't require a callback to be passed in.
2019-08-24 22:25:10 +02:00
TomZ c3ddab9cd4 Make Indexer bind to localhost as default 2019-08-24 22:25:10 +02:00
TomZ 2bb06afa9e Cleanup the link-lines 2019-08-24 22:25:09 +02:00
TomZ da272d7d7f Fix SQL usage for postgres & better logging 2019-08-19 21:42:43 +02:00
TomZ d26386965a Fix defaults
This solves the issue that unless you explicitly turn off the txdb you
still get it.
2019-08-18 16:23:32 +02:00
TomZ 30bee6bf08 Make connecting a higher prio log 2019-08-16 23:39:40 +02:00
TomZ 691ff90014 Avoid uninit data from entering DB
Also fix typo in variable name
2019-08-15 10:55:41 +02:00
TomZ d89f38e4fb More fixlets in indexer after to the rewrite 2019-08-13 23:19:08 +02:00
TomZ 0fdba5a404 Add special postgres SQL statements
When the user selected postgres, use specific SQL statements that are
optimised for it.
2019-08-13 22:21:22 +02:00
TomZ d03be86166 Make indexer follow the tip again
The rewrite of the synching code broke this, so fixing.
2019-08-13 16:19:31 +02:00
TomZ 53c9badc51 Use atomics in the proper way. 2019-08-09 19:53:13 +02:00
TomZ a1ee211968 make compile with older Qt 2019-06-26 22:50:20 +02:00
TomZ 896f5e7b68 Merge branch 'master' into MessageBuilder::Reply 2019-06-26 22:20:44 +02:00
TomZ 98b3e7cece use new MessageBuilder::reply() 2019-06-26 22:19:28 +02:00
TomZ cb5ff404d3 Refactor the Indexer application
This moves each indexer to live in its own thread, making blocking
operations allowed and Ok and separate from incoming requests.
2019-06-26 21:39:07 +02:00
TomZ 6337b1b93b Create indexes 2019-06-19 16:55:51 +02:00
TomZ 9f041cf926 Move handling of address-db requests to another thread
A slow SQL database means we should not do this on the networkmanager
thread as that causes issues.
2019-06-19 16:55:47 +02:00
TomZ a6888b74d8 Need 2 connections to do multithreading
Lets open 2 SQL connections to allow the bulk inserts
to happen in a different thread from the selects we do
resulting from incoming questions.
2019-06-19 16:55:39 +02:00
TomZ fca434b1a6 Make Indexer connect to Hub after loading the conf
Doing it before could lead to incoming data being
processed before the SQL database connection was setup.
2019-06-13 17:12:28 +02:00
TomZ 00963dd31b Make parsing indexer.conf a bit more verbose
And init our new variables.
2019-06-12 23:53:26 +02:00
TomZ 230611b98e Tweak the numbers to be better on IBD 2019-06-12 23:53:24 +02:00
TomZ d172dc4063 Don't fetch data when the node has an older chain.
For instance when its reindexing.
2019-06-11 21:43:50 +02:00
TomZ 9e69a3fef9 Add new indexer; spentoutput
where the UTXO allows you to find an output based on txid+output, the
spentoutput DB turns that around and allows you to find an input that
spends a certain output.

This also fits perfectly find into the UTXO database class, so this
creates a new dir "spent" which is filled as expected.
2019-06-11 17:38:30 +02:00
TomZ a7f4e942f9 Add README for indexer 2019-06-06 20:15:01 +02:00
TomZ ec2610139e Allow setting indexer connections in conf file
This adds the ability to read from the config file the network
connections. Both the listening address and port as well as the upstream
'hub' address.
2019-06-06 19:44:16 +02:00
TomZ 3c659855db Merge branch '2019.05' 2019-06-06 17:47:19 +02:00
TomZ c5aefb0722 Various small fixes in indexer-cli 2019-06-06 17:43:27 +02:00
TomZ 0c9f415eb8 Several NetworkManager fixes
This adds all the header ints to chunked messages, solving the problem
of losing RequestId on roundtrips if the answer was too big.

Added a unit test for this "new" features.

This makes callbacks all use shared_from_this() in order to avoid
callbacks being done on deleted instances (thanks boost!).

Last, special case when the user doesn't connect but just sends messages
which caused a send of data, only to realize the connection wasn't open,
and then a connect.
This makes sure we immediately start a connect on queue of a message.
2019-06-04 21:40:02 +02:00