The Extension Protocol in BitTorrent is a mechanism used to extend and enhance the functionality of the BitTorrent download protocol. It allows clients to communicate by sending and responding to messages that are different from the standard protocol. These messages can implement new features such as Distributed Hash Table (DHT) exchange, Peer Exchange (PEX), and support for higher-level encryption.
The Extension Protocol plays a crucial role in the BitTorrent protocol as it allows clients to engage in more efficient and reliable data exchange, while providing better privacy and security. For example, DHT supports finding other peers connected to the network without a tracker, while PEX allows clients to communicate directly with other clients, making data exchange more stable and reliable.
Protocol Overview
The purpose of this protocol is to provide a simple and lightweight transmission method for extending the BitTorrent protocol. Supporting this protocol makes it easy to add new extension features without disrupting the standard BitTorrent protocol or clients that do not support the desired extension features. In other words, this protocol provides a flexible and scalable mechanism that allows developers to personalize and customize the BitTorrent protocol without affecting other users.
In the BitTorrent protocol, one byte from the reserved bytes is used to propagate oneself to other supporting clients. The byte used for the extension protocol is the 20th byte from the right (counting from 0). Therefore, the expression (reserved_byte[5] & 0x10) can be used to check if a client supports extension message transmission. This expression means taking the 5th byte from the reserved bytes and performing a bitwise AND operation with 0x10. If the result is not 0, it means the client supports extension messages.
After establishing support for the protocol, the client should support one new message:
| name | id |
|---|---|
| extended | 20 |
This message is sent like other BitTorrent messages, with a 4-byte length prefix and a single-byte identifier (in this case, 20). At the beginning of the message payload, there is a single-byte message identifier. This identifier can refer to different extension messages, but only one ID is specified, which is 0. If the ID is 0, it means the message is a handshake message, which will be described below. The general layout of extension messages is as follows (including the message header used by the BitTorrent protocol):
| size | description |
|---|---|
| uint32_t | Length prefix specifying the number of bytes in the entire message (big-endian 1) |
| uint8_t | BitTorrent message ID, = 20 |
| uint8_t | Extension message ID. 0 = handshake, >0 = handshake-specific extension message. |
Handshake Message
The payload of the handshake message is a bencoded dictionary. All items in the dictionary are optional. The client should ignore any unknown names in the dictionary. All parts of the dictionary are case-sensitive. The following items are defined in the dictionary:
| name | description |
|---|---|
| m | A dictionary of supported extension messages mapping each extension message's name to an extension message ID. The only requirement for these IDs is that no two extension messages can share the same ID. Setting the extension number to zero means that the extension is not supported/disabled. The client should ignore any extension names it does not recognize. The extension message ID is used to send the extension message to the peer sending this handshake. In other words, these IDs are local IDs specific to this particular peer. |
Here are some items that implementations may choose to support:
| name | description |
|---|---|
| p | Local TCP listening port. Allows each party to know the TCP port number of the other party. Note that the receiving party does not need to send this extension message as its port number is known. |
| v | Client name and version (as a UTF-8 string). This is a more reliable way to identify the client than relying on peer IDs. |
| yourip | A string containing the compact representation of an IP address to represent the IP address that this node sees for you. In other words, it is the external IP address of the receiving party (excluding the port). It can be an IPv4 (4 bytes) or IPv6 (16 bytes) address. |
| ipv6 | If this node has an IPv6 interface, this is the compact representation of that address (16 bytes). The client may prefer to connect via IPv6 address. |
| ipv4 | If this node has an IPv4 interface, this is the compact representation of that address (4 bytes). The client may prefer to connect via IPv4 address. |
| reqq | An integer representing the number of outstanding request messages this client supports without dropping any messages. In libtorrent, the default value is 250. |
The handshake dictionary can also include extended handshake information, such as support for encrypted headers or any other possible content.
Here is an example of the payload of a handshake message:
| Dictionary | |
|---|---|
| m | Dictionary / LT_metadata······1 / ut_pex··················2 |
| p | 6881 |
| v | "µTorrent 1.2" |
And in encoded form:
d1:md11:LT_metadatai1e6:µT_PEXi2ee1:pi6881e1:v13:\xc2\xb5Torrent 1.2e
To avoid accidental conflicts with extension names, a one or two-character code should be added before the extension name to identify the client introducing the extension. This applies to both the names of extension messages and any other information placed in the top-level dictionary. Unless defined by this specification, all one-byte and two-byte identifiers are invalid.
This message should be sent immediately to any node that supports this extension protocol after the standard BitTorrent handshake. Handshake messages can be sent multiple times throughout the connection, and the sender should not disconnect. Implementations may choose to ignore subsequent handshake messages (or parts of them).
Subsequent handshake messages can be used to enable/disable extensions without restarting the connection. If a node wants to disable support for LT_metadata at runtime without affecting any other extensions, it should send this message: d11. As mentioned above, the value 0 is used to disable the extension.
The extension ID must be stored for each node, as each node may have a different ID for the same extension.
This specification intentionally does not define any extensions such as node exchange or metadata exchange. The protocol is only a transmission method for actual extensions of the BitTorrent protocol, and the extensions named in the example above (such as p) are just examples of possible extensions.
Reasonable Basis
The reason the ID of extension messages is defined during the handshake process instead of using a global message ID registry is to ensure the uniqueness of extension message names, which is easier than using a global registry. The convention is to use a two-letter prefix on the extension message name, which identifies the client that first implemented the extension message. For example, LT_metadata is implemented by libtorrent, so it has the LT prefix.
If the client supporting the extension can decide which numbers its received messages will have, then those numbers are constants within that client. In other words, they can be used in switch statements. For the other end, it is very easy to store an array containing the IDs expected for each message and use that array for lookup when sending extension messages.
The reason for using a dictionary instead of an array (which would implicitly allocate index numbers for extensions) is that if a client wants to disable certain extensions, the ID numbers would change, and it would not be possible to use constants (thus, they cannot be used in switch statements). If the message ID directly maps to a BitTorrent message ID, the extensions in the handshake can also be mapped to existing extensions with fixed message IDs.
The reason for using a single-byte extension message identifier is to follow the BitTorrent specification's use of single-byte message identifiers. This is considered sufficient. It does not limit the total number of extensions, only the number of extensions used simultaneously.
The reason for using a single-byte identifier as the standardized handshake identifier is: 1) the mainline DHT uses a single-byte identifier; 2) bandwidth savings. The only advantage of longer messages is to make the protocol more human-readable, but the BT protocol is not designed to be a human-readable protocol, so why bother.
Overall, the Extension Protocol is an important part of the BitTorrent protocol, providing users with more features and a better user experience.
Reference Links