System Design: Video Calls (Zoom / Google Meet: SFUs, Simulcast, Bad Networks)
Goal: Design our own video call service, like Zoom or Google Meet. About 10 million people are in calls at peak. Voices must arrive with no awkward delay, even on trains and behind office firewalls.
A call can be two people or an all-hands with thousands watching.
Reading guide: You don't need to read this in order.
- Sections 1 to 5. The problem, requirements, API, high-level design and big choices.
- Sections 6 to 11. Deep dives.
- Sections 12 to 21. Features, failures, operations, the interview and sources.
- In an interview, sections 2 to 11 are the core. Sections 15 and 16 are the usual follow-ups.
1. What We Are Building
🔒 Premium section
2. Requirements and Scale
2.1 What the System Must Do
For people in a meeting:
- Create and join meetings. A host creates a meeting and gets an ID and a link. Others join with the ID and a passcode, or with the link.
- Join from anywhere. From a browser with nothing to install, or from desktop and phone apps.
- Audio and video. Everyone can send their microphone and camera, and see and hear the others.
- Screen share. Anyone allowed can share a screen or one window. Text on it must stay sharp.
- Mute and host controls. People mute themselves. The host can mute others, remove someone, lock the meeting and hand over the host role.
- Chat. Text messages to everyone or to one person, kept for the length of the meeting.
- Waiting room. Joiners can wait until the host lets them in.
Larger features:
- Recording. Save the meeting as a video file the host can download.
- Live captions. Show spoken words as text while people talk.
- Large meetings and webinars. Up to about 1,000 people who can all talk, and a view-only audience of tens of thousands.
- Phone dial-in as a follow-up. A person on a plain phone line can join with audio only.
2.2 How Well It Must Do It
One-way delay under 150 ms as the target, and 400 ms as the planning limit. One-way delay is the time from Emily's mouth to Jake's ear.
Under 150 ms, talk feels natural. At about 300 ms, people start to talk over each other. Past 400 ms, a conversation gets hard.
These numbers come from the voice call guide of the ITU (the UN body that sets telephone standards).
Where does the time go? One way, a voice passes through five stages:
- capture and encode,
- the network to the SFU and on to the receiver,
- SFU forwarding,
- the jitter buffer, which holds packets briefly so they play smoothly,
- decode and play.
Forwarding is tiny, because the SFU never decodes. Across an ocean, the network is the biggest part. On a bad network, the jitter buffer is the part that grows.
We control two of these. We put SFUs near users to keep the network part short, and we size the jitter buffer to the network (section 9).
And lost packets never hold up the stream. Lost audio is never resent. Lost video may be resent once, only if it can still arrive in time (section 9).
Join in a few seconds. A person clicks "Join" and should see and hear others within a few seconds on a good network. Setting up a call takes about 6 round trips today (section 6.5), so each extra round trip shows.
Survive the loss of a media server. If an SFU dies, its people rejoin on another one, with media back in about 5 seconds.
The meeting itself must not end. So the list of people (the roster), the host role and chat must live somewhere other than the SFU.
Work on bad networks and behind firewalls. The call must keep going through packet loss, slow links and network switches.
It must also connect where only web traffic is allowed. When the network gets worse, quality drops step by step. Audio is the last thing we give up.
Available 99.9% of the time or better. Zoom sets the same target. A failure in one part should stop new joins at worst. Calls already running should keep going (section 15).
Privacy. All media is encrypted on the network. A host can also turn on end-to-end encryption, where even our servers can't see the media (section 13).
2.3 The Numbers
These inputs are round numbers that we chose for "a large Zoom-like service". For a sense of size, Zoom reported 300 million daily participants in April 2020. We plan for about a third of that.
Count participants, not users. One person in 5 meetings counts 5 times.
- Daily participants. 100 million, each in a meeting for 45 minutes on average.
- Average at any moment. Spread those minutes over a whole day, and about 3 million people are in calls at once.
- Peak. Work hours bunch up, so we plan for about 3 times the average. That's 10 million people at once.
- Meetings. With 4 people in an average meeting, that's 2.5 million meetings at peak.
Now the traffic. Google publishes average numbers from large companies on Meet. Each person uploads about 1 Mbps and downloads about 1.3 Mbps.
Download is higher because each person receives several others. This average is lower than one camera's full upload. A camera sends three sizes of video at once (called simulcast, section 4). But many people turn video off, and sizes nobody watches are paused.
- Into our servers at peak. 10 million people times 1 Mbps is about 10 Tbps (terabits per second).
- Out of our servers at peak. 10 million times 1.3 Mbps is 13 Tbps. Servers in different regions also relay to each other (section 10). We add about 10% for that, so about 14 Tbps.
- Out per day. At the average of 3 million people, that's about 45 petabytes a day.
Now count packets. Video packets hold about 1,200 bytes, and audio packets are small but frequent. One person downloads about 300 packets a second.
Why 300 and not fewer? Audio adds many small packets, one every 20 ms per voice. So the average packet is only about 540 bytes, and 1.3 Mbps / 540 bytes is about 300 packets a second. At peak, our servers send out about 3 billion packets a second.
A media server's work is per packet, not per bit, so packets per second sets the server count (section 7).
How many servers? We plan for about 1,000 people per 16-vCPU server (16 virtual CPUs), kept at about half its CPU. So 10 million people need about 10,000 media servers worldwide. Section 7 shows where that planning number comes from.
2.4 Out of Scope
- Phone network details. We show where a phone gateway plugs in (section 12), not how phone calls are routed.
- AI features beyond captions, such as summaries and meeting notes.
- Billing and accounts. Plans, payments and sign-up work like any web app.
3. The API and Signaling
🔒 Premium section
4. From One Call to a Global Service: The High-Level Design
🔒 Premium section
5. Key Design Choices
🔒 Premium section
6. Deep Dive: Getting Connected
🔒 Premium section
7. Deep Dive: Inside One SFU
🔒 Premium section
8. Deep Dive: Fitting Video to Each Receiver
🔒 Premium section
9. Deep Dive: Surviving Bad Networks
🔒 Premium section
10. Deep Dive: Placing Meetings and Spanning Regions
🔒 Premium section
11. Deep Dive: Large Meetings and Webinars
🔒 Premium section
12. Recording, Captions, Chat and Other Features
🔒 Premium section
13. Security and Privacy
🔒 Premium section
14. Capacity
🔒 Premium section
15. What Happens When Each Part Goes Down
🔒 Premium section
16. Bottlenecks
🔒 Premium section
17. Running It: Rollouts, Quality Monitoring and Cost
🔒 Premium section
18. Trade-offs
🔒 Premium section
19. In the Interview
🔒 Premium section
20. How We Measured
🔒 Premium section
21. Further Reading
🔒 Premium section
Explore the Technologies
🔒 Premium section