wiki commit

2020-01-06 16:55:18 -05:00
parent 5ef58782d6
commit 010556df4d
5 changed files with 97 additions and 124 deletions
@@ -2,7 +2,7 @@
  BIP: 34
  Layer: Consensus (soft fork)
  Title: Block v2, Height in Coinbase
-  Author: Gavin Andresen <gavinandresen@gmail.com>
+  Author: Gavin Andresen &lt;gavinandresen@gmail.com&gt;
  Comments-Summary: No comments yet.
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0034
  Status: Final
@@ -2,8 +2,8 @@
  BIP: 37
  Layer: Peer Services
  Title: Connection Bloom filtering
-  Author: Mike Hearn <hearn@google.com>
+  Author: Mike Hearn &lt;hearn@google.com&gt;
-          Matt Corallo <bip37@bluematt.me>
+          Matt Corallo &lt;bip37@bluematt.me&gt;
  Comments-Summary: No comments yet.
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0037
  Status: Final
@@ -22,7 +22,7 @@ This document will not go into the details of how Bloom filters work and the rea
 As Bitcoin grows in usage the amount of bandwidth needed to download blocks and transaction broadcasts increases. Clients implementing ''simplified payment verification'' do not attempt to fully verify the block chain, instead just checking that block headers connect together correctly and trusting that the transactions in a chain of high difficulty are in fact valid. See the Bitcoin paper for more detail on this mode.
-Today, [clients](https://bitcoin.org/en/developer-guide#simplified-payment-verification-spv|SPV) have to download the entire contents of blocks and all broadcast transactions, only to throw away the vast majority of the transactions that are not relevant to their wallets. This slows down their synchronization process, wastes users bandwidth (which on phones is often metered) and increases memory usage. All three problems are triggering real user complaints for the Android "Bitcoin Wallet" app which implements SPV mode. In order to make chain synchronization fast, cheap and able to run on older phones with limited memory we want to have remote peers throw away irrelevant transactions before sending them across the network.
+Today, [clients](https://bitcoin.org/en/developer-guide#simplified-payment-verification-spv) have to download the entire contents of blocks and all broadcast transactions, only to throw away the vast majority of the transactions that are not relevant to their wallets. This slows down their synchronization process, wastes users bandwidth (which on phones is often metered) and increases memory usage. All three problems are triggering real user complaints for the Android "Bitcoin Wallet" app which implements SPV mode. In order to make chain synchronization fast, cheap and able to run on older phones with limited memory we want to have remote peers throw away irrelevant transactions before sending them across the network.
 ## Design rationale
@@ -111,7 +111,7 @@ To determine if a transaction matches the filter, the following algorithm is use
 4. For each input, test each data element of the input script (note: input scripts only ever contain data elements).
 5. Otherwise there is no match.
-In this way addresses, keys and script hashes (for P2SH outputs) can all be added to the filter. You can also match against classes of transactions that are marked with well known data elements in either inputs or outputs, for example, to implement various forms of [[https://en.bitcoin.it/wiki/Smart_Property|Smart property]].
+In this way addresses, keys and script hashes (for P2SH outputs) can all be added to the filter. You can also match against classes of transactions that are marked with well known data elements in either inputs or outputs, for example, to implement various forms of [Smart property](https://en.bitcoin.it/wiki/Smart_Property).
 The test for outpoints is there to ensure you can find transactions spending outputs in your wallet, even though you don't know anything about their form. As you can see, once set on a connection the filter is '''not static''' and can change throughout the connections lifetime. This is done to avoid the following race condition:
@@ -2,7 +2,7 @@
  BIP: 64
  Layer: Peer Services
  Title: getutxo message
-  Author: Mike Hearn <hearn@vinumeris.com>
+  Author: Mike Hearn &lt;hearn@vinumeris.com&gt;
  Comments-Summary: No comments yet.
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0064
  Status: Draft
@@ -10,11 +10,11 @@
  Created: 2014-06-10
 </pre>
-==Abstract==
+## Abstract
 This document describes a small P2P protocol extension that performs UTXO lookups given a set of outpoints.
-==Motivation==
+## Motivation
 All full Bitcoin nodes maintain a database called the unspent transaction output set. This set is
 how double spending is checked for: to be valid a transaction must identify unspent outputs in this
@@ -32,50 +32,39 @@ Another example of when this data can be useful is for performing floating fee c
 SPV wallet. This use case requires some other changes to the Bitcoin protocol however, so we will
 not dwell on it here.
-==Specification==
+## Specification
 Two new messages are defined. The "getutxos" message has the following structure:
-{|class="wikitable"
+
-! Field Size !! Description !! Data type !! Comments
+| Field Size | Description | Data type | Comments |
-|-
+|--|--|--|--|
-| 1 || check mempool || bool || Whether to apply mempool transactions during the calculation, thus exposing their UTXOs and removing outputs that they spend.
+| 1 | check mempool | bool | Whether to apply mempool transactions during the calculation, thus exposing their UTXOs and removing outputs that they spend.
-|-
+| ? | outpoints | vector&lt;COutPoint&gt; | The list of outpoints to be queried. Each outpoint is serialized in the same way it is in a tx message.
 | ? || outpoints || vector<COutPoint> || The list of outpoints to be queried. Each outpoint is serialized in the same way it is in a tx message.
 |}
 The response message "utxos" has the following structure:
-{|class="wikitable"
+| Field Size | Description | Data type | Comments |
-! Field Size !! Description !! Data type !! Comments
+|--|--|--|--|
-|-
+| 4 | chain height | uint32 | The height of the chain at the moment the result was calculated.
-| 4 || chain height || uint32 || The height of the chain at the moment the result was calculated.
+| 32 | chain tip hash | uint256 | Block hash of the top of the chain at the moment the result was calculated.
-|-
+| ? | hit bitmap | byte[] | An array of bytes encoding one bit for each outpoint queried. Each bit indicates whether the queried outpoint was found in the UTXO set or not.
-| 32 || chain tip hash || uint256 || Block hash of the top of the chain at the moment the result was calculated.
+| ? | result utxos | result[] | A list of result objects (defined below), one for each outpoint that is unspent (i.e. has a bit set in the bitmap).
 |-
 | ? || hit bitmap || byte[] || An array of bytes encoding one bit for each outpoint queried. Each bit indicates whether the queried outpoint was found in the UTXO set or not.
 |-
 | ? || result utxos || result[] || A list of result objects (defined below), one for each outpoint that is unspent (i.e. has a bit set in the bitmap).
 |}
 The result object is defined as:
-{|class="wikitable"
+| Field Size | Description | Data type | Comments |
-! Field Size !! Description !! Data type !! Comments
+|--|--|--|--|
-|-
+| 4 | tx version | uint32 | The version number of the transaction the UTXO was found in.
-| 4 || tx version || uint32 || The version number of the transaction the UTXO was found in.
+| 4 | height | uint32 | The height of the block containing the defining transaction, or 0x7FFFFFFF if the tx is in the mempool.
-|-
+| ? | output | CTxOut | The output itself, serialized in the same way as in a tx message.
 | 4 || height || uint32 || The height of the block containing the defining transaction, or 0x7FFFFFFF if the tx is in the mempool.
 |-
 | ? || output || CTxOut || The output itself, serialized in the same way as in a tx message.
 |}
-==Backward compatibility==
+## Backward compatibility
 Nodes indicate support by advertising a protocol version above 70003 and by setting a new
 NODE_GETUTXO flag in their nServices field, which has a value of 2 (the second bit of the field).
-==Authentication==
+## Authentication
 The UTXO set is not currently authenticated by anything. There are proposals to resolve this by
 introducing a new consensus rule that commits to a root hash of the UTXO set in blocks, however this
@@ -101,6 +90,6 @@ Querying multiple nodes and combining their answers can be a partial solution to
 nothing authenticates the Bitcoin P2P network a man in the middle could still yield incorrect
 results.
-==Implementation==
+## Implementation
 https://github.com/bitcoin/bitcoin/pull/4351/files
@@ -2,9 +2,9 @@
  BIP: 157
  Layer: Peer Services
  Title: Client Side Block Filtering
-  Author: Olaoluwa Osuntokun <laolu32@gmail.com>
+  Author: Olaoluwa Osuntokun &lt;laolu32@gmail.com&gt;
-          Alex Akselrod <alex@akselrod.org>
+          Alex Akselrod &lt;alex@akselrod.org&gt;
-          Jim Posen <jimpo@coinbase.com>
+          Jim Posen &lt;jimpo@coinbase.com&gt;
  Comments-Summary: None yet
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0157
  Status: Draft
@@ -2,8 +2,8 @@
  BIP: 158
  Layer: Peer Services
  Title: Compact Block Filters for Light Clients
-  Author: Olaoluwa Osuntokun <laolu32@gmail.com>
+  Author: Olaoluwa Osuntokun &lt;laolu32@gmail.com&gt;
-          Alex Akselrod <alex@akselrod.org>
+          Alex Akselrod &lt;alex@akselrod.org&gt;
  Comments-Summary: None yet
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0158
  Status: Draft
@@ -13,24 +13,24 @@
 </pre>
-== Abstract ==
+## Abstract
 This BIP describes a structure for compact filters on block data, for use in the
-BIP 157 light client protocol<ref>bip-0157.mediawiki</ref>. The filter
+[BIP 157 light client protocol](/protocol/forks/bip-0157). The filter
 construction proposed is an alternative to Bloom filters, as used in BIP 37,
 that minimizes filter size by using Golomb-Rice coding for compression. This
 document specifies one initial filter type based on this construction that
 enables basic wallets and applications with more advanced smart contracts.
-== Motivation ==
+## Motivation
-[[bip-0157.mediawiki|BIP 157]] defines a light client protocol based on
+[BIP 157](/protocol/forks/bip-0157) defines a light client protocol based on
 deterministic filters of block content. The filters are designed to
 minimize the expected bandwidth consumed by light clients, downloading filters
 and full blocks. This document defines the initial filter type ''basic''
 that is designed to reduce the filter size for regular wallets.
-== Definitions ==
+## Definitions
 <code>[]byte</code> represents a vector of bytes.
@@ -55,9 +55,9 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
 "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
 interpreted as described in RFC 2119.
-== Specification ==
+## Specification
-=== Golomb-Coded Sets ===
+### Golomb-Coded Sets
 For each block, compact filters are derived containing sets of items associated
 with the block (eg. addresses sent to, outpoints spent, etc.). A set of such
@@ -69,21 +69,21 @@ integer parameter <code>M</code>. The encoding is also parameterized by
 specifies values for <code>P</code> and <code>M</code>.
 At a high level, a GCS is constructed from a set of <code>N</code> items by:
-# hashing all items to 64-bit integers in the range <code>[0, N * M)</code>
+ 1. hashing all items to 64-bit integers in the range <code>[0, N * M)</code>
-# sorting the hashed values in ascending order
+ 2. sorting the hashed values in ascending order
-# computing the differences between each value and the previous one
+ 3. computing the differences between each value and the previous one
-# writing the differences sequentially, compressed with Golomb-Rice coding
+ 4. writing the differences sequentially, compressed with Golomb-Rice coding
 The following sections describe each step in greater detail.
-==== Hashing Data Objects ====
+#### Hashing Data Objects
 The first step in the filter construction is hashing the variable-sized raw
 items in the set to the range <code>[0, F)</code>, where <code>F = N *
 M</code>. Customarily, <code>M</code> is set to <code>2^P</code>. However, if
-one is able to select both Parameters independently, then more optimal values
+one is able to select both Parameters independently, then [more optimal values
 can be
-selected<ref>https://gist.github.com/sipa/576d5f09c3b86c3b1b75598d799fc845</ref>.
+selected](https://gist.github.com/sipa/576d5f09c3b86c3b1b75598d799fc845).
 Set membership queries against the hash outputs will have a false positive rate
 of <code>M</code>. To avoid integer overflow, the number of items <code>N</code>
 MUST be <2^32 and <code>M</code> MUST be <2^32.
@@ -96,8 +96,8 @@ SipHash parameters <code>c = 2</code> and <code>d = 4</code>.
 The 64-bit SipHash outputs are then mapped uniformly over the desired range by
 multiplying with F and taking the top 64 bits of the 128-bit result. This
 algorithm is a faster alternative to modulo reduction, as it avoids the
-expensive division
+[expensive division
-operation<ref>https://lemire.me/blog/2016/06/27/a-fast-alternative-to-the-modulo-reduction/</ref>.
+operation](https://lemire.me/blog/2016/06/27/a-fast-alternative-to-the-modulo-reduction/).
 Note that care must be taken when implementing this reduction to ensure the
 upper 64 bits of the integer multiplication are not truncated; certain
 architectures and high level languages may require code that decomposes the
@@ -121,15 +121,15 @@ hashed_set_construct(raw_items: [][]byte, k: [16]byte, M: uint) -> []uint64:
    return set_items
 </pre>
-==== Golomb-Rice Coding ====
+#### Golomb-Rice Coding
 Instead of writing the items in the hashed set directly to the filter, greater
 compression is achieved by only writing the differences between successive
 items in sorted order. Since the items are distributed uniformly, it can be
-shown that the differences resemble a geometric
+shown that the differences resemble a [geometric
-distribution<ref>https://en.wikipedia.org/wiki/Geometric_distribution</ref>.
+distribution](https://en.wikipedia.org/wiki/Geometric_distribution).
-''Golomb-Rice''
+[''Golomb-Rice''
-''coding''<ref>https://en.wikipedia.org/wiki/Golomb_coding#Rice_coding</ref>
+''coding''](https://en.wikipedia.org/wiki/Golomb_coding#Rice_coding)
 is a technique that optimally compresses geometrically distributed values.
 With Golomb-Rice, a value is split into a quotient and remainder modulo
@@ -138,29 +138,18 @@ encoded as ''unary'', with a string of <code>q</code> 1's followed by one 0. The
 remainder <code>r</code> is represented in big-endian by P bits. For example,
 this is a table of Golomb-Rice coded values using <code>P=2</code>:
-{| class="wikitable"
+| n | (q, r) | c
-! n !! (q, r) !! c
+|--|--|--|
-|-
+| 0 | (0, 0) | <code>0 00</code>
-| 0 || (0, 0) || <code>0 00</code>
+| 1 | (0, 1) | <code>0 01</code>
-|-
+| 2 | (0, 2) | <code>0 10</code>
-| 1 || (0, 1) || <code>0 01</code>
+| 3 | (0, 3) | <code>0 11</code>
-|-
+| 4 | (1, 0) | <code>10 00</code>
-| 2 || (0, 2) || <code>0 10</code>
+| 5 | (1, 1) | <code>10 01</code>
-|-
+| 6 | (1, 2) | <code>10 10</code>
-| 3 || (0, 3) || <code>0 11</code>
+| 7 | (1, 3) | <code>10 11</code>
-|-
+| 8 | (2, 0) | <code>110 00</code>
-| 4 || (1, 0) || <code>10 00</code>
+| 9 | (2, 1) | <code>110 01</code>
 |-
 | 5 || (1, 1) || <code>10 01</code>
 |-
 | 6 || (1, 2) || <code>10 10</code>
 |-
 | 7 || (1, 3) || <code>10 11</code>
 |-
 | 8 || (2, 0) || <code>110 00</code>
 |-
 | 9 || (2, 1) || <code>110 01</code>
 |}
 <pre>
 golomb_encode(stream, x: uint64, P: uint):
@@ -184,7 +173,7 @@ golomb_decode(stream, P: uint) -> uint64:
    return x
 </pre>
-==== Set Construction ====
+#### Set Construction
 A GCS is constructed from four parameters:
 * <code>L</code>, a vector of <code>N</code> raw items
@@ -218,7 +207,7 @@ construct_gcs(L: [][]byte, P: uint, k: [16]byte, M: uint) -> []byte:
    return output_stream.bytes()
 </pre>
-==== Set Querying/Decompression ====
+#### Set Querying/Decompression
 To check membership of an item in a compressed GCS, one must reconstruct the
 hashed set members from the encoded deltas. The procedure to do so is the
@@ -259,16 +248,16 @@ of a single item. This can be performed far more efficiently than checking each
 item individually by leveraging the sorted structure of the compressed GCS.
 First the query elements are all hashed and sorted, then compared in order
 against the decompressed GCS contents. See
-[[#golomb-coded-set-multi-match|Appendix B]] for pseudocode.
+[Appendix B](#golomb-coded-set-multi-match) for pseudocode.
-=== Block Filters ===
+### Block Filters
 This BIP defines one initial filter type:
 * Basic (<code>0x00</code>)
 ** <code>M = 784931</code>
 ** <code>P = 19</code>
-==== Contents ====
+#### Contents
 The basic filter is designed to contain everything that a light client needs to
 sync a regular Bitcoin wallet. A basic filter MUST contain exactly the
@@ -283,12 +272,11 @@ Any "nil" items MUST NOT be included into the final set of filter elements.
 We exclude all outputs that start with <code>OP_RETURN</code> in order to allow
 filters to easily be committed to in the future via a soft-fork. A likely area
 for future commitments is an additional <code>OP_RETURN</code> output in the
-coinbase transaction similar to the current witness commitment
+coinbase transaction similar to the [current witness commitment](/protocol/forks/bip-0141). By
 <ref>https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki</ref>. By
 excluding all <code>OP_RETURN</code> outputs we avoid a circular dependency
 between the commitment, and the item being committed to.
-==== Construction ====
+#### Construction
 The basic type is constructed as Golomb-coded sets with the following
 parameters.
@@ -296,8 +284,7 @@ parameters.
 The parameter <code>P</code> MUST be set to <code>19</code>, and the parameter
 <code>M</code> MUST be set to <code>784931</code>. Analysis has shown that if
 one is able to select <code>P</code> and <code>M</code> independently, then
-setting <code>M=1.497137 * 2^P</code> is close to optimal
+setting <code>M=1.497137 * 2^P</code> is [close to optimal](https://gist.github.com/sipa/576d5f09c3b86c3b1b75598d799fc845).
 <ref>https://gist.github.com/sipa/576d5f09c3b86c3b1b75598d799fc845</ref>.
 Empirical analysis also shows that was chosen as these parameters minimize the
 bandwidth utilized, considering both the expected number of blocks downloaded
@@ -314,23 +301,20 @@ complete serialization of a filter is:
 * <code>N</code>, encoded as a <code>CompactSize</code>
 * The bytes of the compressed filter itself
-==== Signaling ====
+#### Signaling
 This BIP allocates a new service bit:
-{| class="wikitable"
+||||
-|-
+|--|--|--|
-| NODE_COMPACT_FILTERS
+| NODE_COMPACT_FILTERS | <code>1 << 6</code> | If enabled, the node MUST respond to all BIP 157 messages for filter type <code>0x00</code>
 | style="white-space: nowrap;" | <code>1 << 6</code>
 | If enabled, the node MUST respond to all BIP 157 messages for filter type <code>0x00</code>
 |}
-== Compatibility ==
+## Compatibility
 This block filter construction is not incompatible with existing software,
 though it requires implementation of the new filters.
-== Acknowledgments ==
+## Acknowledgments
 We would like to thank bfd (from the bitcoin-dev mailing list) for bringing the
 basis of this BIP to our attention, Greg Maxwell for pointing us in the
@@ -341,31 +325,31 @@ Martelletto for writing the initial indexing code for <code>btcd</code>.
 We would also like to thank Dave Collins, JJ Jeffrey, and Eric Lombrozo for
 useful discussions.
-== Reference Implementation ==
+## Reference Implementation
-Light client: [https://github.com/lightninglabs/neutrino]
+Light client: https://github.com/lightninglabs/neutrino
 Full-node indexing: https://github.com/Roasbeef/btcd/tree/segwit-cbf
 Golomb-Rice Coded sets: https://github.com/btcsuite/btcutil/blob/master/gcs
-== Appendix A: Alternatives ==
+## Appendix A: Alternatives
 A number of alternative set encodings were considered before Golomb-coded
 sets were settled upon. In this appendix section, we'll list a few of the
 alternatives along with our rationale for not pursuing them.
-==== Bloom Filters ====
+#### Bloom Filters
 Bloom Filters are perhaps the best known probabilistic data structure for
 testing set membership, and were introduced into the Bitcoin protocol with BIP
 37. The size of a Bloom filter is larger than the expected size of a GCS with
 the same false positive rate, which is the main reason the option was rejected.
-==== Cryptographic Accumulators ====
+#### Cryptographic Accumulators
-Cryptographic
+[Cryptographic
-accumulators<ref>https://en.wikipedia.org/wiki/Accumulator_(cryptography)</ref>
+accumulators](https://en.wikipedia.org/wiki/Accumulator_(cryptography))
 are a cryptographic data structures that enable (amongst other operations) a one
 way membership test. One advantage of accumulators are that they are constant
 size, independent of the number of elements inserted into the accumulator.
@@ -374,17 +358,17 @@ trusted set up. Additionally, accumulators based on the Strong-RSA Assumption
 require mapping set items to prime representatives in the associated group which
 can be preemptively expensive.
-==== Matrix Based Probabilistic Set Data Structures ====
+#### Matrix Based Probabilistic Set Data Structures
 There exist data structures based on matrix solving which are even more space
-efficient compared to Bloom
+efficient compared to [Bloom
-filters<ref>https://arxiv.org/pdf/0804.1845.pdf</ref>. We instead opted for our
+filters](https://arxiv.org/pdf/0804.1845.pdf). We instead opted for our
 GCS-based filters as they have a much lower implementation complexity and are
 easier to understand.
-== Appendix B: Pseudocode ==
+## Appendix B: Pseudocode
-=== Golomb-Coded Set Multi-Match ===
+### Golomb-Coded Set Multi-Match
 <pre>
 gcs_match_any(key: [16]byte, compressed_set: []byte, targets: [][]byte, P: uint, N: uint, M: uint) -> bool:
@@ -430,14 +414,14 @@ gcs_match_any(key: [16]byte, compressed_set: []byte, targets: [][]byte, P: uint,
    return false
 </pre>
-== Appendix C: Test Vectors ==
+## Appendix C: Test Vectors
-Test vectors for basic block filters on five testnet blocks, including the filters and filter headers, can be found [[bip-0158/testnet-19.json|here]]. The code to generate them can be found [[bip-0158/gentestvectors.go|here]].
+Test vectors for basic block filters on five testnet blocks, including the filters and filter headers, can be found [here](https://github.com/bitcoin/bips/blob/master/bip-0158/testnet-19.json). The code to generate them can be found [here](https://github.com/bitcoin/bips/blob/master/bip-0158/gentestvectors.go).
-== References ==
+## References
-<references/>
+(inlined above)
-== Copyright ==
+## Copyright
 This document is licensed under the  Creative Commons CC0 1.0 Universal license.