Warp - Segmented Live Media Transport

1.1. Terms and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

Commonly used terms in this document are described below.¶

Congestion:

Packet loss and queuing caused by degraded or overloaded networks.¶

Frame:

An video image or group of audio samples to be rendered at a specific point in time.¶

I-frame:

A frame that does not depend on the contents of other frames.¶

Group of pictures (GOP):

A I-frame followed by a sequential series of dependent frames.¶

Group of samples:

A sequential series of audio samples starting at a given timestamp.¶

Segment:

A sequence of video frames and/or audio samples serialized into a container.¶

Media player:

A component responsible for presenting frames to a viewer based on the presentation timestamp.¶

Media encoder:

A component responsible for creating a compressed media stream.¶

Media producer:

A QUIC endpoint sending media over the network. This could be the media encoder or middleware.¶

Media consumer:

A QUIC endpoint receiving media over the network. This could be the media player or middleware.¶

Middleware:

A media consumer that forwards streams to one or more downstream media consumers.¶

2.1. Establishment

A connection is established using WebTransport over HTTP/3 [WebTransport]. This involves establishing a HTTP/3 connection, issuing a CONNECT request to establish the session, and exposing the underlying QUIC stream API while the session is active.¶

The application is responsible for authentication based on the CONNECT request.¶

The application is responsible for determining if an endpoint is a media producer, consumer, or both.¶

2.2. Streams

Endpoints communicate over unidirectional QUIC streams. The application MAY use bidirectional QUIC streams for other purposes.¶

Both endpoints can create a new stream at any time. Each stream consists of byte data with an eventual final size. A stream is reliably delivered in order unless canceled early with an error code.¶

The delivery of each stream is independent. The sender MAY prioritize their delivery (Section 2.2.2); intentionally starving streams in favor of more important streams.¶

2.3. Middleware

Media may go through multiple hops and processing steps on the path from the broadcaster to player. The full effectiveness of warp as an end-to-end protocol depends on middleware support.¶

• Middleware SHOULD maintain stream idependence to avoid introducing head-of-line blocking.¶
• Middleware SHOULD maintain stream prioritization when traversing networks susceptible to congestion.¶
• Middleware MUST forward the priority message (Section 4.3) to downstream servers.¶

2.4. Termination

The QUIC connection can be terminated at any point with an error code.¶

The media producer MAY terminate the QUIC connection with an error code of 0 to indicate the end of the media stream. Either endpoint MAY use any other error code to indicate a fatal error.¶

3.1. Initialization

Initialization segments contain track metadata but no sample data.¶

Initialization segments MUST consist of a File Type Box (ftyp) followed by a Movie Box (moov). This Movie Box consists of Movie Header Boxes (mvhd), Track Header Boxes (tkhd), Track Boxes (trak), followed by a final Movie Extends Box (mvex). These boxes MUST NOT contain any samples and MUST have a duration of zero.¶

Note that a Common Media Application Format Header [CMAF] meets all these requirements.¶

3.2. Media

Media segments contain media samples for a single track.¶

Media segments MUST consist of a Segment Type Box (styp) followed by at least one media fragment. Each media fragment consists of a Movie Fragment Box (moof) followed by a Media Data Box (mdat). The Media Fragment Box MUST contain a Movie Fragment Header Box (mfhd) and Track Box (trak) with a Track ID (track_ID) matching a Track Box in the initialization segment.¶

Note that a Common Media Application Format Segment [CMAF] meets all these requirements.¶

4.1. init

The init message indicates that the remainder of the stream contains an initialization segment.¶

{
  init: {
    id: int
  }
}

id:

Incremented by 1 for each unique initialization segment.¶

4.2. media

The segment message contains metadata about the next media segment in the stream.¶

{
  segment: {
    init: int,
    timestamp: int,
    timescale: int, (optional)
  }
}

init:

The id of the cooresponding initialization segment. A decoder MUST block until the cooresponding initialization segment has been fully processed.¶

timestamp:

The presentation timestamp in timescale units for the first frame/sample in the next segment. This timestamp takes precedence over the timestamp in media container to support stream stitching.¶

timescale (optional):

The number of units in second. This defaults to 1000 to signify milliseconds.¶

4.3. priority

The priority message informs middleware about the intended priority of the current stream. Middleware MUST foward this message but it is OPTIONAL to obey it.¶

{
  priority: {
    precedence: int,
  }
}

precedence:

An integer value, indicating that any available bandwidth SHOULD be allocated to streams in descending order.¶

4.4. Extensions

Custom messages MUST start with x-. Unicode LATIN SMALL LETTER X (U+0078) followed by HYPHEN-MINUS (U+002D).¶

Custom messages could control playback. For example: x-pause could halt the transfer of segments until followed by a x-play.¶

Custom messages SHOULD use a unique prefix to reduce collisions. For example: x-twitch-load would contain identification required to start playback of a Twitch stream.¶

5.1. Playback Buffer

It is RECOMMENDED that a media player use a playback buffer to ensure smooth playback at the cost of higher latency. The buffer SHOULD be at last large enough to synchronize audio/video and to account for network/encoding jitter.¶

The size of the playback buffer MAY be increased by temporarily pausing playback or reducing playback speed. The playback buffer MAY be fragmented such that unreceived media can be skipped.¶

A larger playback buffer gives the application more time to recover from starvation without user impact. A media player MAY increase the size of the playback buffer when future starvation events are anticipated.¶

Middleware SHOULD NOT use a buffer, as it will increase latency for each hop.¶

5.2. Congestion Control

Warp uses the underlying QUIC congestion control [QUIC-RECOVERY]. The default congestion control algorithm [NewReno] will work in many situations but can be improved.¶

This section outlines how a live media congestion control algorithm should perform, but does not recommend a specific algorithm.¶

5.3. Prioritization

Media segments might be delivered out of order during starvation.¶

The media player determines how long to wait for a given segment (buffer size) before skipping ahead. The media consumer MAY cancel a skipped segment to save bandwidth, or leave it downloading in the background (ex. to support rewind).¶

Prioritization allows a single media producer to support multiple media consumers with different latency targets. For example, one consumer could have a 1s buffer to minimize latency, while another consumer could have a 5s buffer to improve quality, while a yet another consumer could have a 30s buffer to receive all media (ex. VOD recorder).¶

  if is_audio:
    precedence = timestamp + 3s
  else:
    precedence = timestamp

  precedence = -timestamp

5.4. Bitrate Selection

Live media is encoded in real-time and the bitrate can be adjusted on the fly. This is common in 1:1 media delivery.¶

A media producer MAY reduce the media bitrate in response to starvation. This can be detected via the estimated bitrate as reported by the congestion control algorithm. A less accurate indication of starvation is when the QUIC sender is actively prioritizing streams, as it means the congestion control window is full.¶

5.5. Rendition Selection

Live media is can be encoded into multiple renditions, such that media consumers could receive different renditions based on network conditions. This is common in 1:n media delivery.¶

A media producer MAY switch between renditions at segment boundaries. Renditions SHOULD be fragmented at the same timestamps to avoid introducing gaps or redundant media.¶

5.6. Fragmentation

Segments are encoded as fragmented MP4. Each fragment is a moof and mdat pair containing data for a number of samples. Using more fragments introduces more container overhead (higher bitrate), so it's up to the application to determine the fragment frequency.¶

For the highest latency: one fragment per segment. This means the entire segment must be received before any of the samples can be processed. This is optimal for content that is not intended to be decoded in real-time.¶

For the lowest latency: one fragment per frame. This means that each frame can be decoded when fully received. This is optimal for real-time decoding, however it introduces the largest overhead.¶

Fragments can be created with variable durations. However, the fragment duration SHOULD be relatively consistent to avoid introducing additional playback starvation. Likewise audio and video SHOULD be encoded using similar fragment durations.¶

5.7. Encoding

Warp is primarily a network protocol and does enforce any encoding requirements. However, encoding has a significant impact on the user experience and should be taken into account.¶

B-frames MAY be used to improve compression efficiency, but they introduce jitter. This necessitates a larger playback buffer, increasing latency.¶

Audio and video MAY be encoded and transmitted independently. However, audio can be encoded without delay unlike video. Media players SHOULD be prepared to receive audio before video even without congestion.¶

6.1. Resource Exhaustion

Live media requires significant bandwidth and resources. Failure to set limits will quickly cause resource exhaustion.¶

Warp uses QUIC flow control to impose resource limits at the network layer. Endpoints SHOULD set flow control limits based on the anticipated media bitrate.¶

The media producer prioritizes and transmits streams out of order. Streams might be starved indefinitely during congestion and SHOULD be cancelled (Section 2.2.3) after hitting some timeout or resource limit.¶

The media consumer might receive streams out of order. If stream data is buffered, for example to decode segments in order, then the media consumer SHOULD cancel a stream (Section 2.2.3) after hitting some timeout or resource limit.¶