Abstract

Since its initial inception, the UDP Tracker Protocol has seen widespread adoption by the BitTorrent ecosystem. It is by far the dominant tracker protocol in use, dwarfing the earlier HTTP-based protocol. However, the current version of the UDP protocol used to today does not include any method of adding extensions or additional features. Worse, previous versions of the protocol have described contradictory, incompatible methods. This BEP defines a new extension mechanism that allows new features to be easily added without breaking the existing protocol.

Overview

The original UDP Tracker specification [1] was written by Olaf van der Spek in 2004, for use with his xbtt-tracker. An additional specification was written by Arvid Norberg for rasterbar.com [2] in 2005, referencing Olaf's earlier work. These two specifications are almost identical, differing only in how user authentication and future extendibility is handled. Unfortunately, these differences are significant enough to make the two specifications incompatible with one another when either of these optional features is used.

In 2008, the BitTorrent Enhancement Proposal (BEP) process was put in place to codify many of the improvements that had been made to BitTorrent since it was first released. The same year, Olaf authored BEP-15 [3] to once again document the UDP Tracker Protocol. BEP-15 is largely based on his earlier specification, but differs in that it does not include any form of authentication, or define any extension mechanism, stating only: "Extension bits or a version field are not included. Clients and trackers should not assume packets to be of a certain size. This way, additional fields can be added without breaking compatibility."

As a result, BEP-15 can be thought of as describing the least common denominator of the two specifications that came before it, and is the most interoperable version that has ever been documented. More importantly, there do not appear to be any existing trackers or clients in use today that implement any of the user authentication methods or extension mechanisms that were described in the earlier specifications. It is therefore possible for us to safely define a new extension mechanism, so long as care is taken not to conflict with the existing protocol.

This document defines such an extension mechanism. This mechanism is completely optional, and is backwards compatible with the existing protocol. It has been widely deployed in uTorrent since version 3.2.2 (released in October, 2012) and is not known to cause any problems communicating with existing trackers.

Motivation

Prior to the extensions defined in this document, UDP tracker URLs that contained PATH and QUERY components were completely valid, but those components were invisible to the tracker. For example, from the tracker's perspective, the URL:

udp://tracker.example.com:80/dir?a=b&c=d

and the URL:

udp://tracker.example.com:80

were completely indistinguishable. This differs from the HTTP-based Tracker Protocol, and places restrictions on how a UDP tracker URLs may be used in practice.

Our primary motivation was to develop an extension to the UDP protocol to allow the PATH and QUERY components to be sent to UDP trackers. Rather than implementing a protocol change that only addressed this one issue, we chose to define an extension mechanism to allow future extensions to be easily added without risk of breaking the existing protocol.

Extension Format

The extension mechanism we chose is modeled after the one used in IP and TCP packets [4]. Extensions are defined by a series of options, which can define new behavior.

Option parsing starts immediately at the end of a valid UDP Announce Request packet as defined by BEP-15 (at byte offset 98), and continues until either the end of the UDP packet is reached, or an EndOfOptions option is encountered, whichever happens first. The EndOfOptions option allows the protocol to be further extended in the future, should this extension mechanism not prove to be as flexible as we hope.

Each option made up of either one, two, or three fields:

• 1 byte that identifies the option-type
• 1 byte that specifies the length of the associated data that follows
• A variable-length data field.

Most options have all three fields. For option-types that include a length, the length field does not include the 2 bytes consumed by the option-type field and the length field itself in its total. A length field of zero indicates that the data field has been omitted, and that the very next byte is either the option-byte for the next option, or the end of the packet.

There are only two special-case options that use only the first field: NOP, and EndOfOptions. Any options added in the future are required to include a length field. This ensures that an extension parser written today can safely skip over any unknown option-types that may be defined in subsequent BEPs.

This document defines one option which uses all three fields: URLData

This extension mechanism only allows options to be sent by the client, and only in the Announce Request packet.

Option Types

The complete list of currently defined options is as follows:

EndOfOptions: <Option-Type 0x0>

A special case option that has a fixed-length of one byte. It is not followed by a length field, or associated data. This option indicates that option parsing should stop. Use of this option is optional; option parsing continues until either the end of the packet is reached, or an EndOfOptions option is encountered, whichever happens first.

NOP: <Option-Type 0x1>

A special case option that has a fixed-length of one byte. It is not followed by a length field, or associated data. A NOP has no affect on option parsing. It is used only if optional padding is necessary in the future.

URLData: <Option-Type 0x2>, <Length Byte>, <Variable-Length URL Data>

A variable-length option, followed by a length byte and variable-length data. The data field contains the concatenated PATH and QUERY portion of the UDP tracker URL. If this option appears more than once, the data fields are concatenated. This allows clients to send PATH and QUERY strings that are longer than 255 bytes, chunked into blocks of no larger than 255 bytes.

To reiterate, it is critical that option parsers know which option-types are followed by a length byte, and which are not. Option-types less than or equal to 0x1 are never followed by a length byte. Option-types greater than or equal to 0x2 are always followed by a length byte. Additionally, when a length-byte is present, option parsers must always be prepared for the length field to be zero, indicating no associated option data accompanies this option.

Examples

A client communicating with a tracker at the URL:

udp://tracker.example.com:80/dir?a=b&c=d

might include the following extension bytes in its UDP Announce Request packet, at byte offset 98, describing an URLData option to inform the tracker the full URL the client is using:

0x2, 0xC, '/', 'd', 'i', 'r', '?', 'a', '=', 'b', '&', 'c', '=', 'd'

The following extensions bytes would be equally valid, and evaluate identically. Here, the URLData option is followed by two NOPs, to position the next option on a 32-bit word-boundary. The final option is an EndOfOptions:

0x2, 0xC, '/', 'd', 'i', 'r', '?', 'a', '=', 'b', '&', 'c', '=', 'd', 0x1, 0x1, 0x0

This is admittedly a convoluted example, constructed to demonstrate the use of the NOP and EndOfOptions.

As a final example, a client communicating with a tracker at the URL:

udp://tracker.example.com:80

might include the following extension bytes in its UDP Announce Request packet, containing the URLData option with a length-byte of zero:

0x2, 0x0

A client may wish to send an empty URLData option to indicate that it supports the URLData extension, but that no PATH or QUERY components were present in the tracker URL it used.

Use Cases

The initial problem we were attempting solve was how to run a high-performance tracker while restricting the infohashes it would serve. Using a whitelist file was quickly ruled out, due to the large number of infohashes to be supported, and a requirement to support adding new infohashes very frequently and quickly.

The mechanism we chose used a public-key signature system to cryptographically sign each valid infohash. This was first developed using an HTTP-based tracker, where the infohash signature could easily be included in the tracker URL. This had the advantage of allowing us to deploy the system with complete interoperability with existing clients. But, we also wanted to support UDP trackers, for better performance, which led to the creation of this extension mechanism.

As an example, a tracker URL might be:

udp://tracker.example.com:80/?auth=0x00000000000000000000

Where the string "0x00000000000000000000" is the public-key signature of the infohash. A tracker which receives this QUERY string can authenticate the signature using the public-key. New valid tracker URLs can be created offline using the secret-key.

Implementations

uTorrent versions since 3.2.2 (released in October, 2012) have supported this extension mechanism, including support for the URLData option.

BitTorrent, Inc has patched OpenTracker to support this extension mechanism, including support for the URLData option, and uses the QUERY string to authenticate valid infohashes as described in the previous section. The patch uses Ed25519 [5] as the public-key signature system. We intend to release this patch as an Open Source contribution to OpenTracker in the near future.

References

Copyright

This document has been placed in the public domain.