IMFile

BitTorrent is a protocol used for file distribution. It identifies content through URLs and is designed to seamlessly integrate with the web. Its advantage over simple HTTP downloads is that when multiple users are downloading the same file simultaneously, these users can upload file blocks to each other, allowing multiple users to participate in file transfer instead of relying on a single file server. This allows the file source to support a large number of users downloading simultaneously with minimal impact on its own load.

BitTorrent file distribution consists of the following entities:

• A regular web server
• A static metainfo file
• A BitTorrent tracker
• An original downloader
• A web browser
• A BT download client
• A file that can be shared by many end users for download in the most ideal situation.

To start the service, the host must perform the following steps:

1. Start a tracker (or already have one running).
2. Start a regular web server, such as Apache (or already have one running).
3. Associate the .torrent extension with the MIME type application/x-bittorrent on their web server (or already have done so).
4. Generate a metainfo (.torrent) file using the complete file to be provided and the URL of the tracker.
5. Place the metainfo file on the web server.
6. Link to the metainfo (.torrent) file from other web pages.
7. Start a downloader that already has the complete source file.

To start downloading, the user performs the following actions:

1. Install a BitTorrent client.
2. Surf the internet.
3. Open the link to the .torrent file.
4. Choose the location to save the file locally or select which parts to continue downloading.
5. Wait for the download to complete.
6. Exit the download client (it will continue uploading until it receives the exit command).

Bencode Encoding:
Bencode encoding is used to serialize (also known as "serialize") values of any data type into byte streams for transmission over the network or storage on disk. Bencode encoding supports four basic data types: strings, integers, lists, and dictionaries.

• For string types, they are represented using a length prefix followed by a colon, for example, "spam" is encoded as 4.
• For integer types, they start with the character "i", followed by the decimal representation of the integer, and the character "e", for example, 3 is encoded as i3e, -3 is encoded as i-3e.
• For list types, they start with the character "l", followed by the Bencode encoding of each element in the list, and end with the character "e", for example, ['spam', 'eggs'] is encoded as l4:spam4.
• For dictionary types, they start with the character "d", followed by the Bencode encoding of key-value pairs sorted by the ASCII value of the keys, and end with the character "e", for example, {'cow': 'moo', 'spam': 'eggs'} is encoded as d3:cow3:moo4:spam4. It is important to note that all keys of dictionary types must be strings and sorted according to the original string.

Bencode encoding provides a simple, compact, and readable data serialization scheme that is suitable for serialization and deserialization operations of various data types.

Metainfo File:
A Metainfo file (also known as a .torrent file) is a dictionary encoded using Bencoding, which includes the following keys:

announce
The URL address of the tracker.
info
This key maps to a dictionary that contains the keys described below.
All strings in the .torrent file containing text must be encoded using UTF-8.

info dictionary#

Metadata information used when downloading a BitTorrent file. It contains multiple key-value pairs, where each key represents a specific attribute, such as the suggested file name to save, the size of file blocks, and hash values, etc. These attributes help the client download and assemble the file correctly.

Some important attributes include:

• name: The suggested file name to save, purely for suggestion and not mandatory.
• piece length: The file is divided into fixed-size blocks, and this attribute specifies the size of each block (usually a power of 2).
• pieces: The hash values corresponding to each block, used to verify the integrity of the file.
• length: The length of the file (only applicable in the case of a single file) or the total length after concatenating all files.
  If downloading a single file, only the name and length attributes are needed. If downloading multiple files, the files list is used to represent the directory structure of the files, where each file is represented by a dictionary containing the length and path attributes.

Trackers:
Tracker GET requests include the following keywords:

info_hash: The 20-byte SHA1 hash value of the info value in the metainfo file, in its bencoded form. This value almost certainly needs to be escaped. Note that this is a substring of the metainfo file. The info-hash must be the bencoded hash value as it appears in the .torrent file, which is exactly the same as decoding the metainfo file, extracting the info dictionary, and encoding it, if the bencoded encoding fully validates the input (e.g. key sorting, missing leading zeros), which it must. Conversely, this means that clients must reject invalid metainfo files or extract the substring directly. Decoding-encoding round-trips on invalid data are not allowed.

peer_id: A 20-byte string used as the ID for this downloader. Each downloader generates its own ID randomly when starting a new download. This value almost certainly needs to be escaped as well.

ip: An optional parameter that specifies the IP address (or DNS name) of this peer. It is typically used for the source address, and if the source is on the same computer as the tracker, this parameter is used.

port: The port number that this peer is listening on. Common behavior is for the downloader to try to listen on port 6881, and if that port is taken, try 6882, 6883, etc. and give up after 6889.

uploaded: The total amount uploaded so far, in ASCII decimal.

downloaded: The total amount downloaded so far, in ASCII decimal.

left: The number of bytes this peer still has to download, in ASCII decimal. Note that this cannot be computed from downloaded and the file length since it might be a resume, and there's a chance that some downloaded data failed an integrity check and had to be re-downloaded.

event: This is an optional key that maps to started, completed, or stopped (or empty, which is the same as not being present). If not present, it's one of the periodic announcements. When downloading first starts, the client sends a started announcement; when the download completes, it sends a completed announcement. If the file was complete when started, the completed event should not be sent. The stopped event is sent when the downloader is shutting down gracefully.

The tracker response is a bencoded dictionary. If the tracker response has a failure reason key, it maps to a human-readable string explaining the failure and no other keys are required. Otherwise, it must have two keys: interval, which maps to the number of seconds the downloader should wait between regular re-requests, and peers, which maps to a dictionary of peers, each of which contains the keys peer id, ip, and port, representing the self-selected ID of the peer, the IP address or DNS name, and the port number, respectively. Note that if a compact representation of the peer list is returned by the tracker, see Tracker Returns Compact Peer Lists.

If you want to make any extensions to the metainfo file or tracker queries, please coordinate with Bram Cohen to ensure that all extensions are compatible.

UDP tracker protocol can also be used for announcements.

Peer Protocol:
BitTorrent's peer protocol operates over TCP or the uTorrent transport protocol. In BitTorrent, downloaders and uploaders are peers, and data can flow between them. Data is divided into many small pieces and downloaded from multiple sources simultaneously, improving download speed and transfer reliability. Whenever a downloader completes downloading a piece of the file and verifies that the hash value matches, it broadcasts the information about that piece to all peers.

There are two types of connection states: choked or unchoked; interested or not interested. Choked means no data will be sent until unchoked. Data transfer only occurs when one side is interested and the other side is unchoked. To achieve correct data transfer, the downloader must maintain an interested state and update it promptly, even when choked. When transferring data, the downloader should keep a queue of requests for multiple blocks to achieve good TCP performance, and requests that cannot be immediately written to the TCP buffer should be queued in memory to be discarded in case of choking.

The peer protocol consists of a handshake process and endless length-prefixed messages. The handshake begins with a 19 (decimal) character followed by the string "BitTorrent protocol". Before connecting two nodes, they need to perform a handshake to exchange necessary information. At the start of the handshake, the sending party sends a fixed handshake message to the receiving party, which includes the protocol version, reserved bytes, metadata hash value, and node ID. The receiving party verifies this information and checks if they match the expected values. If there is any mismatch, the connection is terminated. After a successful handshake, both parties start exchanging data. Data is transmitted through a series of length-prefixed messages. The length prefix indicates the length of the message, followed by the actual message content.

Peer Messages:
All non-handshake messages start with a single byte, and below are their types.

Possible values are:

• 0 - choke
• 1 - unchoke
• 2 - interested
• 3 - not interested
• 4 - have
• 5 - bitfield
• 6 - request
• 7 - piece
• 8 - cancel
  'choke', 'unchoke', 'interested', and 'not interested' have no payload.

bitfield message: This message is only sent as the first message. Its payload is a bitfield where each index sent by the downloader is set to 1 and the rest of the indices are set to 0. Downloaders with no data can skip this 'bitfield' message. The first byte of the bitfield corresponds to indices 0-7 (from high bit to low bit), followed by 8-15, and so on. Any extra bits at the end are set to 0.

have message: The payload of this message is a single number representing the index that the downloader has completed and verified the hash value for.

request message: This message contains an index, offset, and length. The last parameter is usually a power of 2 unless it reaches the end of the file. All current implementations use 2^14 (16 kiB) and discard requests that exceed this limit.

cancel message: It has the same payload as the request message. They are typically only sent at the end of a download during the so-called "endgame mode". When a download is nearly complete, the last few pieces tend to be downloaded from a single modem line, which takes a long time. To ensure that the last few pieces enter quickly, a downloader will send requests for the entire content to everyone who is downloading it, once all parts that haven't been received by a given downloader are currently in the pending state. To prevent this from becoming very inefficient, a cancel message is sent to everyone else whenever a piece arrives.

piece message: It contains an index, a start, and a segment. Note that they are implicitly associated with request messages. Unexpected segments can occur if choke and unchoke messages are sent rapidly and/or the transfer rate is very slow.

First, downloaders typically download different parts of the file randomly to avoid situations where one downloader has a subset or superset of another downloader. Secondly, to ensure that each downloader can achieve a consistent download speed, operations to limit upload speed (also known as choking) are needed. This algorithm allows downloaders to use a "tit-for-tat" strategy among themselves to ensure they can achieve a stable download speed. Finally, the currently used choking algorithm is emphasized, and the importance of new algorithms performing well in different network environments.

First, to ensure good TCP performance, the algorithm needs to limit the number of connections that can upload simultaneously. Secondly, to avoid frequent choking and unchoking, the algorithm needs to select upload targets stably. Additionally, it should somewhat reward peers that allow themselves to be downloaded. Finally, the algorithm should attempt to utilize unused connections to improve download speed, which is called optimistic unchoking.

The currently deployed choking algorithm takes some measures to achieve these goals. To avoid frequent changes, the algorithm only changes which peers are choked every 10 seconds. To reward others and limit the number of upload connections, the algorithm selects the top four peers with the best download rates and interest to unchoke. If there are other peers with better upload rates but no interest, they may also be unchoked if they suddenly show interest, and the peer with the worst download rate will be choked. If a downloader has completed the file, it will use the upload rate instead of the download rate to decide which peers to unchoke.

As for optimistic unchoking, the algorithm selects one peer to unchoke every 30 seconds, regardless of its upload rate (as long as it is a downloader with interest). In the rotation, new connections are three times more likely to be placed in the current optimistic unchoke position, giving them enough opportunities to upload complete data blocks.

Reference links:
http://bittorrent.org/beps/bep_0003.html