System Design: News Feed (500M DAU, Hybrid Fan-out, Ranking)
1. The Problem in One Page
A news feed sounds like one query that finds the accounts I follow, takes their recent posts, sorts them and shows twenty. At 500 million daily users it fails in four ways.
One post has to reach many people. For most users that's cheap. But some accounts have 50 million followers, and one post from them is that many pieces of work. People call this the celebrity problem.
Reads far outnumber writes. People read about a hundred times more often than they post, so reads must be cheap. That pushes work to the moment a post is written, and most of this design is about where that work goes.
The feed is ranked, not sorted by time. The order depends on the reader, the post and how others reacted, and some of that is only known at read time.
Small mistakes are very visible. A missing own post, a repeat, an unfollowed account still showing. None is an outage, but each feels broken.
The numbers we design for (inputs, not measurements from a real company):
- 1 billion accounts, 500 million active on a normal day
- 5 billion feed requests and 50 million new posts a day. Posting is burstier than reading, so its peak is three times its average, where reading's is two. Section 2.2 has the peak rates.
- An account follows 200 others on average, and at most 5,000
- A new post reaches followers in under 5 seconds
Mistakes this design avoids:
- Pushing every post to every follower. One huge account can keep the whole write pipeline busy.
- Building every feed at read time. At peak that's 23 million author lookups a second.
- Fetching full posts for every candidate. A feed has hundreds of candidates and shows 20. Fetch only what you rank closely.
- Saving the post and sending the event as two steps. A crash between them loses the post for every follower.
- Paging by offset. New posts arrive while the user scrolls, and a post shows up twice.
- Pushing to people who never open the app. Their lists use memory nobody reads.
- Treating the feed store as a database. It's a cache that can be rebuilt. The posts and the follow graph are the real data.
Words used everywhere here:
- A pod is one running copy of a service.
- A shard is one piece of a cluster, holding part of the data.
- A primary is the shard node that takes writes. A replica, or standby, copies it and can take over.
- A zone is one data centre. A region has three, and the design should survive losing one.
- Kafka is a durable queue. Services write messages to a named topic, and others read them later, in order.
2. Requirements
2.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Show a ranked feed of posts from the accounts a user follows | P0 |
| FR-02 | Create a post (text, with links to media) | P0 |
| FR-03 | Scroll through the feed with no repeated and no skipped posts | P0 |
| FR-04 | Follow and unfollow an account | P0 |
| FR-05 | A user always sees their own new post | P0 |
| FR-06 | Posts from an unfollowed, muted or blocked account stop showing | P0 |
| FR-07 | A deleted post stops showing | P0 |
| FR-08 | Pull to refresh shows new posts | P1 |
| FR-09 | Don't show a post the user has already seen | P1 |
| FR-10 | Hide a post, and use that as a signal for ranking | P1 |
2.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Feed latency at the server, median | under 50 ms |
| NFR-02 | Feed latency at the server, p99 | under 200 ms |
| NFR-03 | Feed requests at peak | 116,000 a second |
| NFR-04 | New posts at peak | 1,750 a second |
| NFR-05 | Time for a post to reach followers, p99 | under 5 seconds for normal accounts |
| NFR-06 | A feed is returned | 99.99% of requests. It may be a simpler feed. |
| NFR-07 | A saved post is never lost | Yes, inside a region. Section 12.1 has the one exception. |
| NFR-08 | Feed lists and sessions | May be lost. They can be rebuilt. |
"p99" is the time all but the slowest 1% of requests beat. Latency is measured at the server, without the user's network or images.
Read NFR-06 and NFR-08 together. The feed must always load, yet the feed store may lose data, and the fallbacks in section 10 make both of those true.
2.3 Out of Scope
- Media. Object storage behind a CDN. A post holds links.
- Likes, comments and shares. Their own services. We only read counts.
- Search, trending and ads. Separate systems.
- Recommending posts from accounts the user doesn't follow. A slot is left (section 6.5) and filled with popular posts.
- Spam and safety checks. They run before a post becomes visible.
- Training the ranking model. Section 6 covers using and rolling it out.
3. Push, Pull, or Both?
🔒 Premium section
4. Architecture
🔒 Premium section
5. The Write Path
🔒 Premium section
6. The Read Path
🔒 Premium section
7. The Social Graph
🔒 Premium section
8. Capacity
🔒 Premium section
9. Correctness
🔒 Premium section
10. What Happens When Each Part Goes Down
🔒 Premium section
11. Bottlenecks and How to Fix Them
🔒 Premium section
12. Regions and Scaling
🔒 Premium section
13. Operations and Cost
🔒 Premium section
14. Trade-offs, and What Is Standard
🔒 Premium section
15. Questions You'll Get Asked
🔒 Premium section
Explore the Technologies
🔒 Premium section