diff --git a/home.md b/home.md index 4d8cdc3..87e8a33 100644 --- a/home.md +++ b/home.md @@ -56,7 +56,7 @@ Secp256k1 — Public Key — Private Key — ECDSA Signatures — Schnorr Signat [sendcmpct](/protocol/network/messages/sendcmpct) — [get_xthin](/protocol/network/messages/get_xthin) — [xthinblock](/protocol/network/messages/xthinblock) — [thinblock](/protocol/network/messages/thinblock) — [get_xblocktx](/protocol/network/messages/get_xblocktx) — [xblocktx](/protocol/network/messages/xblocktx) — [xupdate](/protocol/network/messages/xupdate) — [xversion](/protocol/network/messages/xversion) — [xverack](/protocol/network/messages/xverack) ### Simple Payment Verification (SPV) -[SPV](/protocol/spv) - [Bloom Filters](/objects/bloom__filter) +[SPV](/protocol/spv) - [Bloom Filters](/protocol/spv/bloom-filter) ### Simple Ledger Protocol ### Miscellaneous [Endian](/protocol/misc/endian) diff --git a/protocol/network/messages/filterload.md b/protocol/network/messages/filterload.md index cf9b77a..5560226 100644 --- a/protocol/network/messages/filterload.md +++ b/protocol/network/messages/filterload.md @@ -5,20 +5,18 @@ # Request: Filter Load (“filterload”) -Inserts a transaction and merkle block filter into the peer, overwriting any existing filter. +Inserts a transaction and merkle block filter into the receiving peer, overwriting any existing filter the peer has stored for the sender. ### Effect on Transactions -This message installs a bloom filter into the peer. Subsequent [inv](/protocol/network/messages/inv) and [merkleblock](/protocol/network/messages/merkleblock) messages only provide transactions that in match this bloom filter in some manner. The following items in a transaction are checked against the bloom filter: +This message installs a [bloom filter](/protocol/spv/bloom-filter) into the peer. Subsequent [inv](/protocol/network/messages/inv) and [merkleblock](/protocol/network/messages/merkleblock) messages only provide transactions that in match this bloom filter in some manner. The following items in a transaction are checked against the bloom filter: - The transaction hash - - Each data field in every [output script](/glossary/output__script.md) in the transaction - - Most importantly, this allows public keys and public key hashes (essentially bitcoin addresses) to be added to the bloom filter, allowing a wallet to detect an incoming transfer. - - Each [previous output](/glossary/previous__output.md) in the transaction + - Each data field in every [locking script](/protocol/blockchain/transaction/locking-script) in the transaction + - Most importantly, this allows public keys and public key hashes (essentially Bitcoin addresses) to be added to the bloom filter, allowing a wallet to detect an incoming transfer. + - Each [previous output](/protocol/blockchain/transaction#transaction-input) for the inputs of the transaction (see [Outpoints](/protocol/spv/bloom-filter#outpoints)) - This allows a wallet to detect that a different wallet has spent funds that are co-controlled - - Each data field in every [input script](/glossary/input__script.md) in the transaction. - -See [CBloomFilter::MatchAndInsertOutputs](https://github.com/BitcoinUnlimited/BitcoinUnlimited/blob/bucash1.7.0.0/src/bloom.cpp#L186), and [CBloomFilter::MatchInputs](https://github.com/BitcoinUnlimited/BitcoinUnlimited/blob/eb264e627e231f7219e60eef41b4e37cc52d6d9d/src/bloom.cpp#L234) + - Each data field in every [unlocking script](/protocol/blockchain/transaction/unlocking-script) in the transaction. ### Effect on Merkle Blocks @@ -31,16 +29,15 @@ If a [filtered block](https://github.com/BitcoinUnlimited/BitcoinUnlimited/blob/ | bloom filter size | variable | [variable length integer](/protocol/formats/variable-length-integer) | The size of the bloom filter to be used. | | bloom filter data | variable | `bloom_filter_size` bytes | Raw bloom filter data (up to 36,000 bytes). | | number of hash functions | 4 bytes | unsigned integer[(LE)](/protocol/misc/endian/little) | The number of hash functions used when generating this filter (max 50). | -| tweak | 4 bytes | unsigned integer[(LE)](/protocol/misc/endian/little) | Arbitrary number that helps randomize this bloom filter. | +| tweak | 4 bytes | unsigned integer[(LE)](/protocol/misc/endian/little) | Arbitrary number (nonce) that helps randomize this bloom filter. | | flags | 1 byte | bit field | See [bloom filter flags](#bloom-filter-flags). | ### Bloom Filter Flags The lowest two bits of flags control how the installed bloom filter is automatically updated. The remaining bits are reserved and **should** be set to 0. -| value | description | -|--|--| -| 0 | Do not update -| 1 | update all -| 2 | update pay-to-pubkey, multi-sig, or check-lock-time-verify scripts only - +| Value | Name | Description | +|--|--|--| +| 0 | BLOOM_UPDATE_NONE | Do not update the bloom filter +| 1 | BLOOM_UPDATE_ALL | Add outpoints to the bloom filter when any of the locking script data elements match the bloom filter +| 2 | BLOOM_UPDATE_P2PUBKEY_ONLY | Add outpoints to the bloom filter when any of the locking script data elements match the bloom filter, only when the locking script is a pay-to-public-key or multi-sig script diff --git a/protocol/spv.md b/protocol/spv.md index ba3343c..493ea84 100644 --- a/protocol/spv.md +++ b/protocol/spv.md @@ -13,10 +13,10 @@ The clients can further increase their confidence for the validity of the said t ## Design -The support for SPV was proposed in [BIP-0037](/protocol/forks/bip-0037). +The support for SPV was proposed in [BIP-37](/protocol/forks/bip-0037). Clients that wish to operate in SPV mode should utilize the [`filterload`](/protocol/network/messages/filterload) and [`filteradd`](/protocol/network/messages/filteradd) messages to request the remote peers to install and add content to the bloom filter on its connection with the client. A Bloom filter is a probabilistic data structure which allows for testing set membership - they can have false positives but not false negatives. -Upon receipt of these messages, the remote peers are expected to only announce transactions selected by the filter, with the matching algorithm proposed in [BIP-0037](/protocol/forks/bip-0037). +Upon receipt of these messages, the remote peers are expected to only announce transactions selected by the filter, with the matching algorithm proposed in [BIP-37](/protocol/forks/bip-0037). Nodes that accept a filterload message are expected to only send [inventory messages](/protocol/network/messages/inv) containing transactions that match the client's bloomfilter, or blocks that contain transactions that match the client's Bloom filter. When blocks are requested by the client, they are sent not as [`block`](/protocol/network/messages/block) messages, but as [`merkle block`](/protocol/network/messages/merkleblock) messages, which contain the partial [merkle trees](/protocol/blockchain/block/merkle-tree) of the blocks as the proof of the inclusion for the transactions announced. The transactions can be transmitted to the clients normally, via [`tx`](/protocol/network/messages/tx) messages. diff --git a/protocol/spv/bloom-filter.md b/protocol/spv/bloom-filter.md new file mode 100644 index 0000000..ae2eaf6 --- /dev/null +++ b/protocol/spv/bloom-filter.md @@ -0,0 +1,32 @@ +# Bloom Filter + +A bloom filter is an imperfect but efficient set membership test. The filter will never incorrectly report that member is NOT in the set. However, it may report that a member is in a set when it is actually not. See [wikipedia](https://en.wikipedia.org/wiki/Bloom_filter), or the [original paper](https://dl.acm.org/citation.cfm?id=362686.362692) for details. + +Bloom filters are used in the Bitcoin peer-to-peer protocol to filter the transactions and blocks that a client is sent from another node. The client first "installs" a bloom filter that contains information about the objects it is interested in by sending a [filterload](/protocol/network/messages/filterload) message to a peer. The peer will subsequently only provide objects that match the installed filter. + +At a minimum, an [SPV](/protocol/spv) client must insert addresses and outpoints (identifiers of previous outputs, see [Outpoints](#outpoints)) into the bloom filter. The addresses will cause every transaction that pays INTO the wallet to be reported, and the outpoints will cause every transaction that pays OUT OF the wallet to be reported. + +Bitcoin Bloom filters have an additional feature. Based on the setting of a flag byte, it is possible to direct the peer to automatically insert new data into the bloom filter based on transactions that match. In particular, outpoints will be automatically inserted that correspond to the outputs of addresses that match the filter (see [Bloom Filter Flags](/protocol/network/messages/filterload#bloom-filter-flags)). This facility is extremely important because if a receive and spend occur within the same transaction, the client does not have the opportunity to update the filter. Bloom filters should therefore be sized with the expectation that new elements will be inserted. + +However, due to the bloom filter's tendency to have false positives, these insertions mean that unwanted data will be inserted into the filter, creating even more false positives in a positive feedback loop. Eventually the node will start sending a lot of unnecessary traffic to the client. Therefore, it is essential for clients to periodically refresh the bloom filter (using [filterload](/protocol/network/messages/filterload)) to remove these unnecessary entries. + +## Using Bloom Filters + +At its most basic level, a bloom filter is a large bit-string. Data to be added is hashed with a series of hash operations, whose outputs are interpreted as unsigned integers (modulo the number of bits in the bloom filter) are treated as indices for bits to be set. To test if a value was previously added to the bloom filter, the same hash functions are run and those bit indices are checked. If any of the checked bits are not set, then the value was definitely not previously added to the bloom filter. If all of the bits are set, it is assumed the value was previously added, though the likelihood that it was depends heavily on the size of the bloom filter and the number of values that have been added. + +Since the raw bloom filter data is transmitted between nodes as a part of the filterload message, it is important that all of the parameters are well-defined for a given bloom filter. In Bitcoin, the hash function used is always [32-bit Murmur Hash version 3](https://en.wikipedia.org/wiki/MurmurHash). The seed value used to initialize the Murmur hash is dependent on a tweak (or nonce), chosen at random by the creator of the bloom filter, and the number of hashes being performed (max of 50). For each hash of the input data, the seed is calculated as: `hash_number * 0xFBA4C795 + tweak`. So the seed for the first hash operation is `0 * 0xFBA4C795 + tweak`, the second is `1 * 0xFBA4C795 + tweak`, and so on, up to the number of hashes selected when the bloom filter was created. + +In practice, the size of the bloom filter and the number of hash operations are calculated based on a target false-positive rate and an expected number of values to store. Given a target false-positive probability, `P`, and number of items to add, `N`, the corresponding bloom filter size, `S`, is calculated as `(-1 / pow(log(2), 2) * N * log(P)) / 8`. The number of hash operations is then calculated as `S * 8 / N * log(2)`. + +For more detail, see [BIP-37](/protocol/forks/bip-0037). + +## Outpoints + +Outpoints are a form of serialized identifier for transaction outputs. The format allows for the addition of specific outputs to a bloom filter, particularly for the purposes of matching a future transaction that spends that output. + +### Format + +| Field | Length | Format | Description | +|--|--|--|--| +| transaction hash | 32 bytes | [transaction hash](/protocol/blockchain/hash)[(LE)](/protocol/misc/endian/little) | The hash of the transaction containing the referenced output. | +| output index | 4 bytes | unsigned integer[(LE)](/protocol/misc/endian/little) | This index of the output in the transaction. | \ No newline at end of file