Add domain based download

documentation/data-sourcing

@@ -0,0 +1,76 @@
+The obvious question for anyone starting with this product is:
+where do I get the data from?
+
+The basic idea of BCMRs have 3 main sources of data;
+
+1. DNS. Basically a domain name and we expect a certain file there.
+2. on-chain. Using the concept of authority-chain and comments in
+   transactions that point to URLs on the net.
+3. Payment requests from wallets and point of sale systems.
+
+
+What is important to understand that the bcmr is decentralized. We are
+not trying to create one source of truth for the whole world, we would
+fail spectacularly as the system grows. So the goal is more humble and
+the approach is matched to that.
+
+The bcmr concept gives people the ownership over their own data, which
+is the basis of that decentralization mentioned. Companies can store it
+on their own website. Companies can include it in payment requests whenever
+users visit their store. And totally anonymous users can store data on
+the cloud and refer to it on-chain.
+
+What the BCMR registry project adds is the ability to add a quick lookup
+of data for wallets and users.
+And at the same time this project adds trust indications to the data. It
+is mostly up to the operator to specify that trust, though.
+
+Trust is important since if people start to depend on this system, it's
+decentralized nature will mean the general public can't distinguish between
+a scam or the real thing.
+Wallets that ignore trust will start showing product advertisements to
+end users simply because the advertiser sent a payment to a wallet worth
+a fraction of a cent.
+
+Trust is most of the time local, someone in America won't be able to decide
+which metadata file is real when it is about some Chinese or Indian company.
+
+The conclusion is that the best way to operate a registry is to have the
+operator insert trust into the system. Even if only by marking spam as such.
+
+# DNS starting point.
+
+The DNS starting point is obvious for most bigger projects and companies.
+You create a text file (of any name) in the in/trust/ directory. In it
+you put at least 2 lines;
+  domain=flowee.org
+  trust=ultimate
+We will then check and download the metadata. Refreshing it about once a week.
+
+# on-chain.
+
+The registry does not have any direct connection to the blockchain, but
+transactions with bcmr metadata are accepted.
+Simply place the transaction raw file in the bcmrs/ directory and we will
+download and verify the referenced metadata file.
+
+Notice that this will by default have no relevant trust level (none).
+When the transaction belongs to a chain of transactions on an auth-base does
+the trust level go up.
+See the 'trust' docs on how to provide a specific trust assessment.
+
+Anyone wanting to use the BCMR as a simple "all bcmrs known" simply needs
+to find all relevant transactions on the chain with the right comment and
+drop them in the bcmrs/ dir to make that happen.
+
+# Auth-chain (from wallets of payment requests).
+
+The owner of an auth-base is expected to have the entire chain of
+transactions that they could share with us. Importing this will make
+all metadata files that use the auth-base as their identity be checked
+to actually be referred to from that auth-chain. And they will then
+automatically inherit the trust of the auth-chain. The token categories
+inherit this trust level as well.
+
+An auth-chain file (format to be determined) needs to be dropped in the
+authchains/ subdir, 

documentation/trust

@@ -38,3 +38,17 @@ owner and that we can advertise the trust to be higher.
 In hour generated metadata this is reflected by the (token) category itself
 having a trust, and each metadata file linked having it's own specific
 trust.
+
+# Config format
+
+In the trust subdir of the input directory you can place many text files
+that allows you to add trust to the system and kickstart filling the data.
+
+The file is very simple name/value pairs in standard unix line-endings.
+Fields are read from top to bottom, you can mention the same field various
+times such that the item is processed in order.
+
+* trust=ultimate
+  The trust level is given. none, marginal, good, ultimate
+* domain=bitcoincash.org  
+  Provides the source of the 

src/DownloadJob.cpp

@@ -18,7 +18,8 @@ void DownloadJob::start()
 {
     assert(m_sourceUrl.isValid());
     QNetworkRequest request(m_sourceUrl);
-    request.setHeader(QNetworkRequest::UserAgentHeader, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36");
+    if (m_sourceUrl.host() == "ipfs.io")
+        request.setHeader(QNetworkRequest::UserAgentHeader, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36");
     auto reply = m_net->get(request);
     connect (reply, &QNetworkReply::finished, this, [=]() {
         if (reply->error() != QNetworkReply::NoError) {

src/DownloadJob.h

@@ -13,7 +13,7 @@ class DownloadJob : public QObject
 {
     Q_OBJECT
 public:
-    explicit DownloadJob(QNetworkAccessManager *parent = nullptr);
+    explicit DownloadJob(QNetworkAccessManager *parent);
 
     void start();
 

src/Processor.cpp

@@ -42,12 +42,28 @@ bool Processor::run()
         logCritical() << "Failed to create target dir";
         return false;
     }
-    QDirIterator iter(m_inDir, QDir::NoDotAndDotDot | QDir::Files | QDir::Dirs, QDirIterator::Subdirectories);
+    QFileInfo in(m_inDir);
+    if (!in.isDir())
+        return false;
+
+    QDirIterator iter(m_inDir + "trust", QDir::NoDotAndDotDot | QDir::Files | QDir::Dirs, QDirIterator::Subdirectories);
     while (iter.hasNext()) {
-        const QString path = iter.next();
-        if (path.indexOf("bcmrs/", m_inDir.size()) == m_inDir.size()) {
-            parseBCMR(path);
-        }
+        auto fi = iter.nextFileInfo();
+        if (fi.isFile())
+            parseTrustFile(fi.absoluteFilePath());
+    }
+
+    QTimer::singleShot(1, this, SLOT(runDownloadQueue()));
+    return true;
+}
+
+void Processor::run2()
+{
+    QDirIterator iter(m_inDir + "bcmrs", QDir::NoDotAndDotDot | QDir::Files | QDir::Dirs, QDirIterator::Subdirectories);
+    while (iter.hasNext()) {
+        auto fi = iter.nextFileInfo();
+        if (fi.isFile())
+            parseBCMR(fi.absoluteFilePath());
     }
 
     for (auto *source : m_sources) {
@@ -139,22 +155,23 @@ bool Processor::run()
         logCritical() << "Placing or updating"
             << (mg->type == AuthBase ? "authbase" : "category") << "file" << mg->id;
         QFile out(outPath);
-        if (!out.open(QIODevice::WriteOnly)) {
+        if (!out.open(QIODevice::WriteOnly))
             logCritical() << "Failed to open file for writing:" << outPath;
-            logFatal() << "Giving up";
-            return false;
-        }
-        out.write(memData);
+        else
+            out.write(memData);
     }
 
     QTimer::singleShot(1, this, SLOT(runDownloadQueue()));
-
-    return true;
 }
 
 void Processor::runDownloadQueue()
 {
-    if (m_offline || m_downloadJobs.isEmpty()) {
+    if (m_phase == Run1 && (m_downloadJobs.isEmpty() || m_offline)) {
+        m_phase = Run2;
+        QTimer::singleShot(1, this, SLOT(run2()));
+        return;
+    }
+    if ((m_offline || m_downloadJobs.isEmpty()) && m_phase == Run2) {
         QCoreApplication::quit();
         return;
     }
@@ -319,6 +336,46 @@ void Processor::parseBCMR(const QString &path)
     }
 }
 
+void Processor::parseTrustFile(const QString &path)
+{
+    // TODO move the cache to a in/bcmrs/downloaded/trust_[filename].domain file
+    QFileInfo me(path);
+    QFileInfo cache(me.absolutePath() + "/../bcmrs/downloaded/" + me.fileName());
+    if (cache.exists()) {
+        QDateTime marker = QDateTime::currentDateTimeUtc().addDays(-7);
+        if (cache.lastModified() > marker) {
+            logInfo() << "Skipping check of" << path;
+            return;
+        }
+    }
+    QFile in(path);
+    if (!in.open(QIODevice::ReadOnly)) {
+        logCritical() << "Failed to read" << path;
+        return;
+    }
+    for (const QString &line : QString::fromUtf8(in.readAll()).split('\n')) {
+        if (line.startsWith("domain=")) {
+            // .well-known/bitcoin-cash-metadata-registry.json
+            QUrl location;
+            location.setHost(line.mid(7).trimmed());
+            location.setPath("/.well-known/bitcoin-cash-metadata-registry.json");
+            location.setScheme("https");
+            DownloadJob *job = new DownloadJob(&m_network);
+            job->setSourceUrl(location);
+            job->setTargetFilePath(cache.absoluteFilePath());
+            connect (job, &DownloadJob::finished, this, [=]() {
+                m_downloadJobs.removeAll(job);
+                QTimer::singleShot(1, this, SLOT(runDownloadQueue()));
+            });
+            m_downloadJobs.append(job);
+        }
+//       if (line.startsWith("trust=")) {
+// 				//TODO
+//       }
+    }
+
+}
+
 void Processor::parseBCHTx(QFile &file)
 {
     // TODO

src/Processor.h

@@ -23,9 +23,11 @@ public:
 
 private slots:
     void runDownloadQueue();
+    void run2();
 
 private:
     void parseBCMR(const QString &path);
+    void parseTrustFile(const QString &path);
     void parseBCHTx(QFile &file);
     struct MetaGroup;
     enum GroupType {
@@ -66,6 +68,12 @@ private:
     QNetworkAccessManager m_network;
     QList<DownloadJob*> m_downloadJobs;
     bool m_offline = false;
+
+    enum Phase {
+        Run1,
+        Run2
+    };
+    Phase m_phase = Run1;
 };
 
 #endif