System Design: News Aggregator (100K Sources, Story Clustering, Breaking News)
1. The Problem in One Page
A news aggregator sounds like a loop. You read every news site, sort the articles by time and show them to everyone in the same order. At this size that loop fails in four ways.
Nobody tells you when a site publishes. Most sites offer an RSS or Atom feed, a small file listing their newest articles. You fetch it and look for items you haven't seen. Big sites publish hundreds of articles a day, small blogs a few a week.
Check every feed every 5 minutes and you make about 330 requests a second (100,000 feeds every 300 seconds), and nearly all of them find nothing. Check rarely and you miss the news.
The same event arrives 200 times. When an earthquake hits, hundreds of sites write about it. Some copy a wire story word for word, others use their own words. A user wants it once, with a link to every version.
Spotting copies is cheap. Spotting one event in different words is hard, and it's the core of this design.
A story must be ranked before anyone has read it. A ten-minute-old story has almost no clicks, so the first ranking has to come from how many trusted sources cover it and how fast that number grows.
Breaking news must be fast, and a mistake is costly. A push sent by mistake reaches millions of phones. It can't be taken back.
The numbers we design for (inputs for this post, not measurements):
- 100,000 sources, on about 40,000 websites
- 2 million new articles a day, and 10 times that rate for up to an hour during a big event
- 50 editions, one per language and country pair, like Hindi for India
- 50 million daily users, about 10,000 feed requests a second at peak
- about 70 million monthly users
Mistakes this design avoids:
- Checking every source at the same rate. It wastes requests and still misses news from busy sites.
- Keeping the crawl schedule only in a cache. If the cache loses it, crawling stops.
- Treating "same event" as "same text". Letter similarity can't see that two different headlines describe one event.
- Counting breaking news per category. A busy category hides one story growing fast.
- Ranking single articles. Twenty copies of one story would fill the feed.
- Reading the same few cache keys on every request. They become the bottleneck.
2. Requirements
2.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Read 100,000 RSS and Atom feeds and find their new articles | P0 |
| FR-02 | Group articles about the same event into one story | P0 |
| FR-03 | Show a ranked list of stories for an edition and a section (top stories, world, sports and so on) | P0 |
| FR-04 | Show every source that covers a story ("full coverage") | P0 |
| FR-05 | Find breaking stories and move them to the top | P0 |
| FR-06 | Mix in each user's interests, like sections they read, topics they follow and sources they muted | P1 |
| FR-07 | Don't show a story the user has already seen, unless it changed a lot | P1 |
| FR-08 | Send push notifications for breaking stories | P1 |
| FR-09 | Search articles by keyword | P1 |
| FR-10 | Let editors pin, block or demote a story, and block a source | P1 |
| FR-11 | Remove an article quickly after a legal takedown | P1 |
2.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Feed latency at the server, median | under 30 ms |
| NFR-02 | Feed latency at the server, p99 | under 150 ms |
| NFR-03 | Feed requests | 10,000 a second at peak |
| NFR-04 | Time from publishing to the feed, for the 10,000 busiest sources | under 10 minutes, for 9 in 10 articles |
| NFR-05 | The same, for every other source | under 1 hour, for 9 in 10 articles |
| NFR-06 | A breaking story reaches the top | within 2 minutes of meeting the rule |
| NFR-07 | Requests to one website | at most 1 a second, and robots.txt (a file where a website says which pages crawlers may fetch) is followed |
| NFR-08 | A feed is returned | 99.99% of requests, perhaps less personal or a little older |
| NFR-09 | An article that joined a story is lost | never |
"p99" means 99 in 100 requests are faster. Latency is low because feeds are built from memory (section 8). Freshness depends on how often each source is checked (section 4).
2.3 Out of Scope
- The full article. A story links to the publisher. We keep only the title, a short summary and an image link, which is also safer legally.
- Comments, likes, sharing, ads and paid plans.
- Fact checking. Editors can block a story or source (FR-10). Deciding what is true is another system.
- Training the models. We cover how an embedding model and a small classifier are used, not trained.
3. Architecture
🔒 Premium section
4. Crawling
🔒 Premium section
5. From Article to Story
🔒 Premium section
6. Ranking Stories
🔒 Premium section
7. Breaking News
🔒 Premium section
8. Serving the Feed
🔒 Premium section
9. Capacity
🔒 Premium section
10. Correctness
🔒 Premium section
11. What Happens When Each Part Goes Down
🔒 Premium section
12. Bottlenecks and How to Fix Them
🔒 Premium section
13. Regions and Scaling
🔒 Premium section
14. Operations and Cost
🔒 Premium section
15. Trade-offs, and What Is Standard
🔒 Premium section
16. Questions You'll Get Asked
🔒 Premium section
Explore the Technologies
🔒 Premium section