sfu

Overview ¶

Design of Prober

Probing is to used to check for existence of excess channel capacity. This is especially useful in the downstream direction of SFU. SFU forwards audio/video streams from one or more publishers to all the subscribers. But, the downstream channel of a subscriber may not be big enough to carry all the streams. It is also a time varying quantity.

When there is not enough capacity, some streams will be paused. To resume a stream, SFU would need to know that the channel has enough capacity. That's where probing comes in. When conditions are favorable, SFU can send probe packets so that the bandwidth estimator has more data to estimate available channel capacity better. NOTE: What defines `favorable conditions` is implementation dependent.

There are two options for probing

• Use padding only RTP packets: This one is preferable as probe rate can be controlled more tightly.
• Resume a paused stream or forward a higher spatial layer: Have to find a stream at probing rate. Also, a stream could get a key frame unexpectedly boosting rate in the probing window.

The strategy used depends on stream allocator implementation. This module can be used if the stream allocator decides to use padding only RTP packets for probing purposes.

Implementation: There are a couple of options

• Check prober in the forwarding path (pull from prober). This is preferred for scalability reasons. But, this suffers from not being able to probe when all streams are paused (could be due to downstream bandwidth constraints or the corresponding upstream tracks may have paused due to upstream bandwidth constraints). Another issue is not being to have tight control on probing window boundary as the packet forwarding path may not have a packet to forward. But, it should not be a major concern as long as some stream(s) is/are forwarded as there should be a packet atleast every 60 ms or so (forwarding only one stream at 15 fps). Usually, it will be serviced much more frequently when there are multiple streams getting forwarded.
• Run it a go routine. But, that would have to wake up very often to prevent bunching up of probe packets. So, a scalability concern as there is one prober per subscriber peer connection. But, probe windows should be very short (of the order of 100s of ms). So, this approach might be fine.

The implementation here follows the second approach of using a go routine.

Pacing: ------ Ideally, the subscriber peer connection should have a pacer which trickles data out at the estimated channel capacity rate (and estimated channel capacity + probing rate when actively probing).

But, there a few significant challenges

1. Pacer will require buffering of forwarded packets. That means more memory, more CPU (have to make copy of packets) and more latency in the media stream.
2. Scalability concern as SFU may be handling hundreds of subscriber peer connections and each one processing the pacing loop at 5ms interval will add up.

So, this module assumes that pacing is inherently provided by the publishers for media streams. That is a reasonable assumption given that publishing clients will run their own pacer and pacing data out at a steady rate.

A further assumption is that if there are multiple publishers for a subscriber peer connection, all the publishers are not pacing in sync, i. e. each publisher's pacer is completely independent and SFU will be receiving the media packets with a good spread and not clumped together.

Given those assumptions, this module monitors media send rate and adjusts probing packet sends accordingly. Although the probing may have a high enough wake up frequency, it is for short windows. For example, probing at 5 Mbps for 1/2 second and sending 1000 byte probe per iteration will wake up every 1.6 ms. That is very high, but should last for 1/2 second or so.

5 Mbps over 1/2 second = 2.5 Mbps
2.5 Mbps = 312500 bytes = 313 probes at 1000 byte probes
313 probes over 1/2 second = 1.6 ms between probes

A few things to note

1. When a probe cluster is added, the expected media rate is provided. So, the wake up interval takes that into account. For example, if probing at 5 Mbps for 1/2 second and if 4 Mbps of it is expected to be provided by media traffic, the wake up interval becomes 8 ms.
2. The amount of probing should actually be capped at some value to avoid too much self-induced congestion. It maybe something like 500 kbps. That will increase the wake up interval to 16 ms in the above example.
3. In practice, the probing interval may also be shorter. Typically, it can be run for 2 - 3 RTTs to get a good measurement. For the longest hauls, RTT could be 250 ms or so leading to the probing window being long(ish). But, RTT should be much shorter especially if the subscriber peer connection of the client is able to connect to the nearest data center.

Design of StreamAllocator ¶

Each participant uses one peer connection for all downstream traffic. It is possible that the downstream peer connection gets congested. In such an event, the SFU (sender on that peer connection) should take measures to mitigate the media loss and latency that would result from such a congestion.

This module is supposed to aggregate down stream tracks and drive bandwidth allocation with the goals of

• Try and send highest quality media
• React as quickly as possible to mitigate congestion

Setup: ------ The following should be done to set up a stream allocator

• There will be one of these per subscriber peer connection. Created in livekit-sever/transport.go for subscriber type peer connections.
• In `AddSubscribedTrack` of livekit-server/participant.go, the created downTrack is added to the stream allocator.
• In `RemoveSubscribedTrack` of livekit-server/participant.go, the downTrack is removed from the stream allocator.
• Both video and audio tracks are added to this module. Although the stream allocator does not act on audio track forwarding, audio track information like loss rate may be used to adjust available bandwidth.

Callbacks: ---------- StreamAllocator registers the following callbacks on all registered down tracks

• OnREMB: called when down track receives RTCP REMB. Note that REMB is a peer connection level aggregate metric. But, it contains all the SSRCs used in the calculation of that REMB. So, there could be multiple callbacks per RTCP REMB received (one each from down track pertaining to the contained SSRCs) with the same estimated channel capacity.
• AddReceiverReportListener: called when down track received RTCP RR (Receiver Report).
• OnAvailableLayersChanged: called when the feeding track changes its layers. This could happen due to publisher throttling layers due to upstream congestion in its path.
• OnSubscriptionChanged: called when a down track settings are changed resulting from client side requests (muting/pausing a video or limiting maximum layer).
• OnPacketSent: called when a media packet is forwarded by the down track. As this happens once per forwarded packet, processing in this callback should be kept to a minimum.

The following may be needed depending on the StreamAllocator algorithm

• OnBitrateUpdate: called periodically to update the bit rate at which a down track is forwarding. This can be used to measure any overshoot and adjust allocations accordingly. This may have granular information like primary bitrate, retransmitted bitrate and padding bitrate.

State machine: -------------- The most critical component. It should monitor current state of channel and take actions to provide the best user experience by striving to achieve the goals outlined earlier

States:
------
- State_PRE_COMMIT: Before the first estimate is committed.
                      Estimated channel capacity is initialized to some
                      arbitrarily high value to start streaming immediately.
                      Serves two purposes
                      1. Gives the bandwidth estimation algorithms data
                      2. Start streaming as soon as a user joins. Imagine
                         a user joining a room with 10 participants already
                         in it. That user should start receiving streams
                         from everybody as soon as possible.
- State_STABLE: When all streams are forwarded at their optimal requested layers.
- State_DEFICIENT: When at least one stream is not able to forward optimal requested layers.
- State_GRATUITOUS_PROBING: When all streams are forwarded at their optimal requested layers,
                            but probing for extra capacity to be prepared for cases like
                            new participant joining and streaming OR an existing participant
                            starting a new stream like enabling camera or screen share.

Signals:
-------
Each state should take action based on these signals and advance the state machine based
on the result of the action.
- Signal_ADD_TRACK: A new track has been added.
- Signal_REMOVE_TRACK: An existing track has been removed.
- Signal_ESTIMATE_INCREASE: Estimated channel capacity is increasing.
- Signal_ESTIMATE_DECREASE: Estimated channel capacity is decreasing. Note that when
                            channel gets congested, it is possible to get several of these
                            in a very short time window.
- Signal_RECEIVER_REPORT: An RTCP Receiver Report received from some down track.
- Signal_AVAILABLE_LAYERS_ADD: Available layers of publisher changed, new layer(s) available.
- Signal_AVAILABLE_LAYERS_REMOVE: Available layers of publisher changed, some previously
                                  available layer(s) not available anymore.
- Signal_SUBSCRIPTION_CHANGE: Subscription changed (mute/requested layers changed).
- Signal_PERIODIC_PING: Periodic ping

There are several interesting challenges which are documented in relevant code below.