stunner

STUNner: A Kubernetes ingress gateway for WebRTC

Ever wondered how to deploy your WebRTC infrastructure into the cloud? Frightened away by the complexities of Kubernetes container networking, and the surprising ways in which it may interact with your UDP/RTP media? Tried to read through the endless stream of Stack Overflow questions asking how to scale WebRTC services with Kubernetes, just to get (mostly) insufficient answers? Want to safely connect your users behind a NAT, without relying on expensive third-party TURN services?

Worry no more! STUNner allows you to deploy any WebRTC service into Kubernetes, smoothly integrating it into the cloud-native ecosystem. STUNner exposes a standards-compliant STUN/TURN gateway for clients to access your virtualized WebRTC infrastructure running in Kubernetes, maintaining full browser compatibility and requiring minimal or no modification to your existing WebRTC codebase. STUNner implements the standard Kubernetes Gateway API so you can configure it in the familiar YAML-engineering style via Kubernetes manifests.

Table of Contents

1. Description
2. Features
3. Getting started
4. Tutorials
5. Documentation
6. Caveats
7. Milestones

Description

Currently WebRTC lacks a virtualization story: there is no easy way to deploy a WebRTC media service into Kubernetes to benefit from the resiliency, scalability, and high availability features we have come to expect from modern network services. Worse yet, the entire industry relies on a handful of public STUN servers and hosted TURN services to connect clients behind a NAT/firewall, which may create a useless dependency on externally operated services, introduce a bottleneck, raise security concerns, and come with a non-trivial price tag.

The main goal of STUNner is to allow anyone to deploy their own WebRTC infrastructure into Kubernetes, without relying on any external service other than the cloud-provider's standard hosted Kubernetes offering. This is achieved by STUNner acting as a gateway for ingesting WebRTC media traffic into the Kubernetes cluster, exposing a public-facing STUN/TURN server that WebRTC clients can connect to.

In the headless deployment model STUNner acts as a simple scalable STUN/TURN server that WebRTC clients can use as a NAT traversal facility for establishing a media connection. This is not that much different from a standard public STUN/TURN server setup, but in this case the STUN/TURN servers are deployed into Kubernetes, which makes lifecycle management, scaling and cost optimization infinitely simpler.

In the fully fledged media-plane deployment model STUNner implements a STUN/TURN ingress gateway service that WebRTC clients can use to open a transport relay connection to the media servers running inside the Kubernetes cluster. This makes it possible to deploy WebRTC application servers and media servers into ordinary Kubernetes pods, taking advantage of Kubernetes's excellent tooling to manage, scale, monitor and troubleshoot the WebRTC infrastructure like any other cloud-bound workload.

Don't worry about the performance implications of processing all your media through a TURN server: STUNner is written in Go so it is extremely fast, it is co-located with your media server pool so you don't pay the round-trip time to a far-away public STUN/TURN server, and STUNner can be easily scaled up if needed, just like any other "normal" Kubernetes service.

Features

Kubernetes has been designed and optimized for the typical HTTP/TCP Web workload, which makes streaming workloads, and especially UDP/RTP based WebRTC media, feel like a foreign citizen. STUNner aims to change this state-of-the-art, by exposing a single public STUN/TURN server port for ingesting all media traffic into a Kubernetes cluster in a controlled and standards-compliant way.

• Seamless integration with Kubernetes. STUNner can be deployed into any Kubernetes cluster, even into restricted ones like GKE Autopilot, using a single command. Manage your HTTP/HTTPS application servers with your favorite service mesh, and STUNner takes care of all UDP/RTP media. STUNner implements the Kubernetes Gateway API so you configure it in exactly the same way as the rest of your workload through easy-to-use YAML manifests.

• Expose a WebRTC media server on a single external UDP port. Get rid of the Kubernetes hacks, like privileged pods and hostNetwork/hostPort services, typically recommended as a prerequisite to containerizing your WebRTC media plane. Using STUNner a WebRTC deployment needs only two public-facing ports, one HTTPS port for the application server and a single UDP port for all your media.

• No reliance on external services for NAT traversal. Can't afford a decent hosted TURN service for client-side NAT traversal? Can't get a decent audio/video quality because the third-party TURN service poses a bottleneck? STUNner can be deployed into the same cluster as the rest of your WebRTC infrastructure, and any WebRTC client can connect to it directly without the use of any external STUN/TURN service whatsoever, apart from STUNner itself.

• Easily scale your WebRTC infrastructure. Tired of manually provisioning your WebRTC media servers? STUNner lets you deploy the entire WebRTC infrastructure into ordinary Kubernetes pods, thus scaling the media plane is as easy as issuing a kubectl scale command. STUNner itself can be scaled with similar ease, completely separately from the media servers.

• Secure perimeter defense. No need to open thousands of UDP/TCP ports on your media server for potentially malicious access; with STUNner all media is received through a single ingress port that you can tightly monitor and control.

• Simple code and extremely small size. Written in pure Go using the battle-tested pion/webrtc framework, STUNner is just a couple of hundred lines of fully open-source code. The server is extremely lightweight: the typical STUNner container image size is only about 5 Mbytes.

Getting Started

STUNner comes with a Helm chart to fire up a fully functional STUNner-based WebRTC media gateway in minutes. Note that the default installation does not contain an application server and a media server: STUNner in itself is not a WebRTC service, it is merely an enabler for you to deploy your own WebRTC infrastructure into Kubernetes. Once installed, STUNner makes sure that your media servers are readily reachable to WebRTC clients, despite running with a private IP address inside a Kubernetes pod.

With a minimal understanding of WebRTC and Kubernetes, deploying STUNner should take less than 5 minutes.

• Customize STUNner and deploy it into your Kubernetes cluster.
• Optionally deploy a WebRTC media server.
• Set STUNner as the ICE server in your WebRTC clients.
• ...
• Profit!!

Installation

The simplest way to deploy STUNner is through Helm. STUNner configuration parameters are available for customization as Helm Values. We recommend deploying STUNner into a separate namespace and we usually name this namespace as stunner, so as to isolate it from the rest of the workload.

helm repo add stunner https://l7mp.io/stunner
helm repo update
helm install stunner stunner/stunner --set stunner.namespace=stunner
helm install stunner-gateway-operator stunner/stunner-gateway-operator

Configuration

The standard way to interact with STUNner is via Kubernetes via the standard Gateway API version v1alpha2. This is akin to the way you configure all Kubernetes workloads: specify your intents in YAML files and issue a kubectl apply, and the STUNner gateway operator will automatically reconcile the STUNner dataplane for the new configuration.

1. Given a fresh STUNner install, the first step is to register STUNner with the Kubernetes Gateway API. This amounts to creating a GatewayClass, which serves as the root level configuration for your STUNner deployment.

   Each GatewayClass must specify a controller that will manage the Gateway objects created under the class hierarchy. In our case this must be set to stunner.l7mp.io/gateway-operator for STUNner to pick up the GatewayClass. In addition, a GatewayClass can refer to further implementation-specific configuration via a parametersRef; in our case, this will be a GatewayConfig object to be specified next.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1alpha2
   kind: GatewayClass
   metadata:
     name: stunner-gatewayclass
   spec:
     controllerName: "stunner.l7mp.io/gateway-operator"
     parametersRef:
       group: "stunner.l7mp.io"
       kind: GatewayConfig
       name: stunner-gatewayconfig
       namespace: stunner
     description: "STUNner is a WebRTC ingress gateway for Kubernetes"
   EOF
   

2. The next step is to set some general configuration for STUNner, most importantly the STUN/TURN authentication credentials. This requires loading a GatewayConfig custom resource into Kubernetes.

   Below we set the plaintext authentication mechanism for STUNner, using the username/password pair user-1/pass-1, and the authentication realm stunner.l7mp.io. See the package docs for further configuration options available via GatewayConfigs.

   kubectl apply -f - <<EOF
   apiVersion: stunner.l7mp.io/v1alpha1
   kind: GatewayConfig
   metadata:
     name: stunner-gatewayconfig
     namespace: stunner
   spec:
     realm: stunner.l7mp.io
     authType: plaintext
     userName: "user-1"
     password: "pass-1"
   EOF
   

   Note that these two steps are required only once per STUNner installation.

3. At this point, we are ready to expose STUNner to clients! This occurs by loading a Gateway resource into Kubernetes.

   In the below example, we open a STUN/TURN listener service on the UDP listener port 3478. STUnner will automatically expose this listener on a public IP address and port (by creating a LoadBalancer service for each Gateway), await clients to connect to this listener and, once authenticated, forward client connections to an arbitrary service backend inside the cluster. Note that we set the gatewayClassName to the name of the above GatewayClass; this is the way STUNner will know which class hierarchy the Gateway belongs so that it pick up the STUN/TURN credentials.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1alpha2
   kind: Gateway
   metadata:
     name: udp-gateway
     namespace: stunner
   spec:
     gatewayClassName: stunner-gatewayclass
     listeners:
       - name: udp-listener
         port: 3478
         protocol: UDP
   EOF
   

4. The final step is to tell STUNner what to do with the client connections received on the Gateway. In the media-plane deployment model we will want to route client connections to a WebRTC media server, but we may also let connections to loop back to STUNner itself in the headless deployment model. You can route connections received on a Gateway to any Kubernetes service by attaching a UDPRoute resource to it and specifying the target service in the backendRef. A UDPRoute can be attached to any Gateway by setting the parentRef to the Gateway's name, there is just one rule: the Gateway and the UDPRoute must both live in the same Kubernetes namespace.

   The below UDPRoute will configure STUNner to route client connections received on the Gateway called udp-gateway to the WebRTC media server pool identified by the Kubernetes service media-plane in the default namespace.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1alpha2
   kind: UDPRoute
   metadata:
     name: media-plane
     namespace: stunner
   spec:
     parentRefs:
       - name: udp-gateway
     rules:
       - backendRefs:
           - name: media-plane
             namespace: default
   EOF
   

And that's all: once configured, STUNner will make all this happen automatically, and you don't need to worry about client-side NAT traversal and request routing because STUNner has you covered! Even better, every time you change a Gateway API resource in Kubernetes, say, you update the GatewayConfig to reset your STUN/TURN credentials or change the protocol or port in one of your Gateways, the STUNner gateway operator will automatically pick up your modifications and update the underlying dataplane in a matter of milliseconds. Kubernetes is beautiful, isn't it?

Check your config

The current STUNner dataplane configuration is always made available in a convenient ConfigMap called stunnerd-config (you can choose the name in the GatewayConfig). There is one rule: the ConfigMap always lives in the same namespace as the GatewayConfig that belongs to the corresponding gatewayclass hierarchy. Actually, the STUNner dataplane pods themselves will use the same very ConfigMap to reconcile their internal state so you can consider its content to be the ground truth.

STUNner comes with a small utility to dump the running configuration in human readable format (you must have jq installed in your PATH to be able to use it). Chdir into the main STUNner directory and issue:

cmd/stunnerctl/stunnerctl running-config stunner/stunnerd-config
STUN/TURN authentication type:	plaintext
STUN/TURN username:		user-1
STUN/TURN password:		pass-1
Listener:	udp-listener
Protocol:	UDP
Public address:	34.118.36.108
Public port:	3478

As it turns out, STUNner has successfully assigned a public IP and port to our Gateway and set the STUN/TURN credentials based on the GatewayConfig. You can use the below to dump the entire running configuration; jq is there just to pretty-print JSON:

kubectl get cm -n stunner stunnerd-config -o jsonpath="{.data.stunnerd\.conf}" | jq .

Testing

We have successfully configured STUNner to route client connections to the media-plane service but at the moment there is no backend that would respond. Below we will use a simplistic UDP greeter service for testing: every time you send some input, the greeter service will respond with a heartwarming welcome message.

1. Fire up the UDP greeter service.

   The below manifest spawns the service in the default namespace and wraps it in a Kubernetes service called media-plane. Note that the type of the media-plane service is ClusterIP, which means that Kubernetes will not expose the service to the Internet: the only way for clients to obtain a response is via STUNner.

   kubectl apply -f deploy/manifests/udp-greeter.yaml
   

2. We also need a STUN/TURN client to actually initiate a connection. STUNner comes with a handy STUN/TURN client called turncat for this purpose.

   Build turncat first.

   cd stunner
   go build -o turncat cmd/turncat/main.go
   

3. We also need the ClusterIP assigned by Kubernetes to the media-plane service (recall, this service is the backend service we specified in the UDPRoute).

   export PEER_IP=$(kubectl get svc media-plane -o jsonpath='{.spec.clusterIP}')
   

4. Now we can start turncat at last! Send any input: pressing Enter you should see a nice greeting from your cluster!

   ./turncat - k8s://stunner/stunnerd-config:udp-listener udp://${PEER_IP}:9001
   Hello STUNner
   Greetings from STUNner!
   

Observe how we haven't specified the STUNner public IP address and port for turncat. The turncat utility is clever enough to read the running configuration from Kubernetes directly; just specify the special STUNner URI k8s://stunner/stunnerd-config:udp-listener, identifying the namespace and the name for the STUNner ConfigMap and the name of the listener to connect to, and turncat will do the heavy lifting.

Note that your actual WebRTC clients will not need to use turncat to reach the cluster: all modern Web browsers and WebRTC clients come with a STUN/TURN client included. Here, turncat is used only to simulate what a real WebRTC client would do when trying to reach STUNner. For more info, see the turncat documentation.

Reconcile

Any time you see fit, you can update the STUNner configuration through the Gateway API: STUNner will automatically reconcile the underlying dataplane for the new configuration in a matter of millisecs.

For instance, you may decide to open up your WebRTC infrastructure on TCP as well; say, because an enterprise NAT on the client network path has gone berserk and started to actively filter UDP/TURN traffic. The below steps will do just that: open another gateway on STUNner, this time on the TCP port 3478, and reattach the UDPRoute to both Gateways so that no matter which protocol a client connection was received on it will be routed to the media-plane service (i.e., the UDP greeter) by STUNner.

1. Add the new TCP Gateway.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1alpha2
   kind: Gateway
   metadata:
     name: tcp-gateway
     namespace: stunner
   spec:
     gatewayClassName: stunner-gatewayclass
     listeners:
       - name: tcp-listener
         port: 3478
         protocol: TCP
   EOF
   

2. Update the UDPRoute so that it attaches to both Gateways.

   kubectl apply -f - <<EOF
   apiVersion: gateway.networking.k8s.io/v1alpha2
   kind: UDPRoute
   metadata:
     name: media-plane
     namespace: stunner
   spec:
     parentRefs:
       - name: udp-gateway
       - name: tcp-gateway
     rules:
       - backendRefs:
           - name: media-plane
             namespace: default
   EOF
   

3. Fire up turncat again, but this time let it connect through TCP. This is achieved by specifying the name of the TCP listener (tcp-listener) in the STUNner URI.

   ./turncat -l all:INFO - k8s://stunner/stunnerd-config:tcp-listener udp://${PEER_IP}:9001
   [...] turncat INFO: Turncat client listening on -, TURN server: TCP://34.118.18.210:3478, peer: udp://10.120.0.127:9001
   [...]
   Hello STUNner
   Greetings from STUNner!
   

We have set the turncat loglevel to INFO to learn that this time turncat has connected via the TURN server TCP://34.118.18.210:3478. And that's it: STUNner automatically routes the incoming TCP connection to the UDP greeter service, silently converting from TCP to UDP in the background and back again on return.

Configuring WebRTC clients

Real WebRTC clients will need a valid ICE server configuration to use STUNner as the TURN server. STUNner is compatible with all client-side TURN auto-discovery mechanisms. When no auto-discovery mechanism is available, clients will need to be manually configured to stream audio/video media over STUNner.

The below JavaScript snippet will direct a WebRTC client to use STUNner as the TURN server. Make sure to substitute the placeholders (like <STUNNER_PUBLIC_ADDR>) with the correct configuration from the running STUNner config; don't forget that stunnerctl is always there for you to help.

var ICE_config = {
  iceServers: [
    {
      url: 'turn:<STUNNER_PUBLIC_ADDR>:<STUNNER_PUBLIC_PORT>?transport=udp',
      username: <STUNNER_USERNAME>,
      credential: <STUNNER_PASSWORD>,
    },
  ],
};
var pc = new RTCPeerConnection(ICE_config);

Note that STUNner comes with a small Node.js library that simplifies generating ICE configurations and STUNner credentials in the application server.

Tutorials

STUNner comes with several tutorials that show how to use it to deploy different WebRTC applications into Kubernetes.

• Opening a UDP tunnel via STUNner: This introductory tutorial shows how to tunnel an external connection via STUNner to a UDP service deployed into Kubernetes. The demo can be used to quickly check a STUNner installation.
• Headless deployment: Direct one to one video call via STUNner: This tutorial showcases the headless deployment model, that is, when WebRTC clients connect to each other directly via STUNner using it as a TURN server, but without the mediation of a WebRTC media server.
• Media-plane mode: One to one video call with Kurento via STUNner: This tutorial extends the previous demo to showcase the media-plane deployment model, that is, when WebRTC clients connect to each other via a media server deployed into Kubernetes. This time, the media server is provided by Kurento, but you can easily substitute your favorite media server instead. STUNner will ingest WebRTC media into the cluster and route it to Kurento, and all this happens without modifying the media server code in any way, just by adding 5-10 lines of straightforward JavaScript to configure clients to use STUNner as the TURN server.
• Media-plane mode: Magic mirror via STUNner: This tutorial has been adopted from the Kurento magic mirror demo. The demo shows a basic WebRTC loopback server with some media processing added: the application uses computer vision and augmented reality techniques to add a funny hat on top of faces. The computer vision functionality is again provided by the Kurento media server, being exposed to the clients via a STUNner gateway.
• Media-plane mode: Cloud-gaming with STUNner: If this was still not enough from the fun, this tutorial lets you play Super Mario or Street Fighter in your browser, courtesy of the amazing CloudRetro cloud-gaming project and of course STUNner. The tutorial shows how to deploy CloudRetro into Kubernetes, expose the media port via STUnner, and have endless retro-gaming fun!

Documentation

See further documentation here.

Caveats

STUNner is a work-in-progress. Some features are missing, others may not work as expected. The notable limitations at this point are as follows.

• STUNner is not intended to be used as a public STUN/TURN server. The intended use of STUNner is as a Kubernetes ingress gateway for WebRTC. Being deployed into a Kubernetes service, STUNner will not be able to identify the public IP address of a client sending a STUN binding request to it (without special hacks), and the TURN transport relay connection opened by a WebRTC client via STUNner is reachable only to clients configured to use the same STUNner service (again, without further hacks). This is intended: STUNner is a Kubernetes ingress gateway which happens to expose a STUN/TURN compatible service to WebRTC clients, and not a public TURN service.
• STUNner targets only a partial implementation of the Kubernetes Gateway API. In particular, only GatewayClass, Gateway and UDPRoute resources are supported. This is intended: STUNner deliberately ignores some unnecessary complexity in the Gateway API and deviates from the prescribed behavior in some case, all this to simplify the configuration process. The STUNner Kubernetes gateway operator docs contain a detailed list on the differences.
• Certain Kubernetes control plane operations will trigger a STUN/TURN server restart in STUNner, which leads to dropping all active client connections. In particular, adding, removing or modifying Gateways currently require a full server restart. Modifying the GatewayConfig or UDPRoutes, however, is reconciled seamlessly. We plan to remove this restriction in a later release; for now it is best to refrain from intrusive changes on live STUNner deployments.
• STUNner supports arbitrary scale-up without dropping active calls, but scale-down might disconnect calls established through the STUNner pods and/or media server replicas being removed from the load-balancing pool. Note that this problem is universal in WebRTC, but we plan to do something about it in a later STUNner release so stay tuned.
• SCTP DataChannels are not supported at the moment.

Milestones

• v0.9: First public release: STUNner basic UDP/TURN connectivity + helm chart + tutorials
• v0.10: Long-term STUN/TURN credentials and STUN/TURN over TCP/TLS/DTLS.
• v0.11: Day-2 operations: STUNner Kubernetes gateway operator and dataplane reconciliation.
• v0.12: Security: Hide plain text credentials in the Gateway API and expose TLS/DTLSS.
• v0.13: Observability: Prometheus + Grafana dashboard.

• v0.14: Performance: eBPF acceleration
• v1.0: GA (this fall)
• v2.0: Service mesh: adaptive scaling & resiliency

Help

STUNner development is coordinated in Discord, send us an email to ask an invitation.

License

Copyright 2021-2022 by its authors. Some rights reserved. See AUTHORS.

MIT License - see LICENSE for full text.

Acknowledgments

Initial code adopted from pion/stun and pion/turn.