CrackingWalnuts
Design a news aggregator like Google News for 100,000 sources and 50 million daily users. A crawl schedule that survives crashes, four layers of duplicate detection from exact copies to the same event in different words, ranking stories with a key that never needs updating, breaking news per story, measured sizing, failures, bottlenecks and regions.