Design a Price Tracking Service

Each phase is what the interviewer expects you to do and say. Concrete steps, not topic hints. The diagrams are what you sketch on the board.

1. 1

   Clarify Requirements and Scope

   5 min

   GoalPin the scrape budget (50M/day, ~580/sec sustained), the alert latency SLO (<5s), and the fact that scraping is the bottleneck, not storage or alerting. Get the three scraping tiers on the board before drawing anything.

   Do & Say

   1. SAY·1SAY: “This looks like a database problem but it's actually a scraping problem.” “4,600 scrapes/sec is trivial for storage but hostile to retailers who will IP-ban you fast.” Write “SCRAPING IS THE BOTTLENECK” on the board.
   2. SAY·2Pin the scale: “10M products × 5 checks/day = 50M scrapes/day”, “~580 sustained, ~1,500 with headroom”. Storage at 1KB/event: “~50 GB/day raw”, “~1.8 TB/year”, “~120 GB/year compressed”.
   3. SAY·3Confirm alert latency SLO: “Sub-5 second end-to-end from scrape commit to push notification on the user's phone.” “This is what justifies Flink CEP over database polling.”
   4. SAY·4Park out of scope: “account management UI”, “the browser-extension auto-fill”, “A/B testing of alert message copy”. “Stay focused on the price pipeline.”
   5. SAY·5Establish the three scraping tiers up front: “Tier 1 API (Amazon PA-API, Walmart Open API) for ~20% of products at near-zero cost”, “Tier 2 HTTP scrape for server-rendered pages, ~30%”, “Tier 3 Playwright headless browser for JS-rendered pages, ~50%”. “The tier mix determines the proxy budget.”

   Interviewer is grading: You reframe the problem (scraping, not storage) within the first 2 minutes. You write the three scraping tiers down before they ask. You don't promise per-minute price freshness; you negotiate it via the priority score in the deep dive.

2. 2

   API and Data Model

   5 min

   GoalDefine the user-facing tracking and alert APIs, the time-series hypertable, the alert rule shape. Justify TimescaleDB over plain PostgreSQL with the index-doesn't-fit-RAM math.

   Do & Say

   1. WRITE·1Write the user APIs: “POST /products/track with retailer URL”, “GET /products/{id}/history with from/to/granularity”, “POST /alerts with type”. Five alert types: “price_below”, “percentage_drop”, “all_time_low”, “any_drop”, “back_in_stock”. Mention them explicitly so the interviewer can't blindside you with “what about back-in-stock?”
   2. DRAW·2Sketch the price_history hypertable: “(product_id, retailer_id, time, price_cents, currency, availability)”. “Primary key (product_id, retailer_id, time).” “Hypertable on time with 7-day chunks.”
   3. SAY·3Justify TimescaleDB out loud: “18B rows after a year.” “The B-tree index is ~200 GB, doesn't fit in RAM, every query hits disk.” “TimescaleDB partitions by time”, “compresses old chunks 15x”, “runs continuous aggregates for daily/weekly/monthly buckets”. “PostgreSQL alone would die at this scale.” Cross plain PostgreSQL off.
   4. WATCH·4Cross InfluxDB off too: “InfluxDB has weaker secondary indexing.” “We need to join price events with products and users for alert evaluation.” “TimescaleDB is still PostgreSQL underneath, so joins are first-class.”
   5. SAY·5Continuous aggregates: “price_daily”, “price_weekly”, “price_monthly” materialized views, auto-refreshed. “A 1-year chart reads from price_weekly (52 points), not from raw chunks (40K+ points).”
   6. SAY·6Alert rule shape (PostgreSQL): “(alert_id, user_id, product_id, retailer_id nullable, alert_type, target_price_cents, percentage_drop, snooze_until, is_active)”. Note: “retailer_id is nullable, so users can alert on the best price across all retailers, not just one.”

   Interviewer is grading: You volunteer the index-doesn't-fit-RAM math without prompting. You name continuous aggregates instead of saying 'we'll add a cache.' You distinguish price_history (time series) from alerts (relational) and put them in the right stores.

3. 3

   High-Level Design

   10 min

   GoalOne picture showing the four sub-pipelines: scheduling, scraping, processing, alerting. Label arrows with throughput and label Kafka topics by name.

   Draw on the board
   Do & Say

   1. DRAW·1Draw the four sub-pipelines left to right. Say: “these are not microservices for the sake of microservices”. “Scheduling, scraping, and alerting all have different failure modes and scaling axes.”
   2. SAY·2Scheduling: “Adaptive Scheduler computes a priority score from alert_count, watcher_count, volatility, last_price_change_at.” “Maps the score to a check interval: 30 min for hot products, 7 days for dormant ones.” “Enforces the 50M/day budget by scaling down low-priority intervals if the ideal sum exceeds budget.”
   3. SAY·3Scraping: “Three tiers in one router”: “Tier 1 retailer APIs, near zero cost”, “Tier 2 HTTP for server-rendered pages”, “Tier 3 Playwright pool for JS pages”. “Per-retailer session affinity: each residential IP stays with a retailer 5-15 min, then rotates.”
   4. SAY·4Per-retailer parser registry: “Each parser has a version”, “tries multiple extraction strategies in order (CSS selector, JSON-LD, regex fallback)”, “emits health metrics”. “Success rate drops below 90% triggers a page and stops scraping that retailer to preserve proxy quota.”
   5. SAY·5Anomaly validator: “Range check against the last known price (no 80% drops without corroboration)”, “format check (currency, decimal places)”. “Anomalous prices go to a review queue, not into TimescaleDB.” “This is what catches scraping bugs before they fire false alerts.”
   6. SAY·6Alert pipeline: “Flink CEP keyed by product_id holds last_price, all_time_low, last_availability per product in RocksDB state.” “Alert rules are a broadcast stream.” “Each price event evaluates against all active rules for that product.” “Triggers go to alert.triggers Kafka topic, dispatcher fans out to FCM/APNs/SES.”
   7. SAY·7Why Flink, not DB poll: “At 580 events/sec with 100M alerts, DB polling is 580 queries/sec at 50-200ms each.” “Flink with RocksDB state hits <10ms per event and handles patterns (percentage drop over window) without custom code.” “Exactly-once on failover.”

   Interviewer is grading: Your diagram has four sub-pipelines, not one blob. You label Kafka topics by name. You name Flink CEP and defend it with the DB-poll math, not just by saying 'streaming is better.'

4. 4

   Deep Dive: Anti-Bot Defense, Adaptive Frequency, Parser Resilience

   15 min

   GoalThe three places this design lives or dies in production. The interviewer will push on at least one of these and you should volunteer the others.

   Draw on the board
   Do & Say

   1. SAY·1Anti-bot defense first, this is what kills naive designs: “Residential IP pool (10K IPs, no datacenter)”, “per-retailer session affinity (5-15 min then rotate)”, “browser fingerprint with webdriver hidden, plugins spoofed, locale set”, “networkidle wait + 0.5-2s random delay”, “2captcha for CAPTCHA paths”.
   2. SAY·2Per-retailer session limits: “Amazon ~5 requests per session before its risk score climbs”, “Walmart 8”, “Target 6”, “less aggressive retailers 15”. “These are tuned per retailer based on block-rate observation, not guessed.”
   3. SAY·3Pivot to adaptive frequency. “We do not check every product every hour.” “That would be 240M scrapes/day, 4x our budget.” “The Adaptive Scheduler computes a priority score from four factors.” Draw the score: “alert_count 0-40”, “volatility 0-30”, “watchers 0-15”, “recency 0-15”.
   4. SAY·4Score to interval: “Score 80-100 → every 30 minutes (hot products with active alerts and volatility)”, “Score 0-19 → every 7 days (dormant, no alerts, stable price)”. “The mapping is deliberately coarse so the schedule is rehearsable and predictable.”
   5. SAY·5Budget enforcement: “If the sum of ideal_daily_checks exceeds 50M, sort by priority descending and scale down the low-priority tail until it fits.” “Headroom of ~45% of the budget is intentional for retries, Black Friday bursts, and growth.”
   6. SAY·6Pivot to parser resilience. “The most common production incident is a retailer redesigns their page.” “All Amazon parsers break at once.” Mitigation: “per-retailer parser modules (one redesign affects one parser, not all of them)”, “multiple extraction strategies per parser (CSS selector first, JSON-LD second, regex fallback)”.
   7. SAY·7Parser health: “Each parser tracks success_count, failure_count, last_success.” “If failure rate exceeds 10% over the last 100 attempts, page on-call”, “stop scraping that retailer (to preserve proxy quota)”, “queue affected products for re-scrape once the parser ships”. “The dashboard is the on-call's first stop during an incident.”
   8. SAY·8Alert-storm question: “Black Friday, 100 watchlist products all drop 30%, user gets 100 pushes and uninstalls.” Mitigation: “per-user rate limit (10 alerts/hour default)”, “aggregate into single 5 deals from your watchlist push”, “snooze controls per rule”.
   9. SAY·9Anomaly detection: “Validator compares new price to last known for that product-retailer pair.” “Drops >50% go to a review queue (usually wrong-currency parse or clearance glitch).” “Don't want iPhone for $20 alerts because someone parsed cents.”

   Interviewer is grading: You volunteer the proxy session-affinity detail without being asked (this is the giveaway that you've actually built a scraper). You don't promise uniform check frequencies for all products. You name parser health as the recurring production incident and have a runbook (page, stop, fix, backfill).

5. 5

   Failure Modes, Trade-offs, and Wrap-up

   5 min

   GoalDefend the freshness vs scrape-budget trade-off, name the cascading failure (retailer goes dark), close in one sentence.

   Do & Say

   1. SAY·1Major retailer goes dark: “Amazon Bot Manager flags us, success drops to ~10%.” “Parser health dashboard pages on-call.” Mitigation: “pause Amazon scraping 6h”, “rotate to a new residential proxy”, “refresh fingerprints”. “App shows last updated 4h ago badge.” “Amazon alerts don't fire, but we don't fire false ones either.”
   2. SAY·2TimescaleDB write outage: “Kafka buffers price.events with 3-day retention.” “Flink CEP keeps running on whatever's in Kafka, so alerts still fire.” “Once TimescaleDB recovers, the consumer catches up.” “Historical charts go stale, but the live alert path doesn't.”
   3. SAY·3Flink CEP outage: “Alerts stop firing.” “This is the worst-case for the product.” “RocksDB state is checkpointed every 60s so we restart from the last checkpoint.” “Worst-case alert latency during recovery: 1 to 2 minutes.” “We accept that because the alternative (DB polling) doesn't scale.”
   4. SAY·4Trade-off 1: scraping budget vs freshness. “We can buy more scrapes by raising the budget, but the marginal cost is dominated by Tier 3 Playwright + residential proxy cost.” “30-min freshness for the top 200K products is the sweet spot.”
   5. SAY·5Trade-off 2: false alerts vs missed alerts. “Strict anomaly validation means we sometimes miss real price drops (catastrophic-looking but actually correct).” “Loose validation means we sometimes fire false alerts.” “We prefer missed alerts because false alerts erode trust faster.”
   6. SAY·6Close in one breath: “Adaptive Scheduler distributes a 50M/day budget across 10M products by priority”, “three-tier scrape router (API + HTTP + Playwright) with residential proxy rotation”, “TimescaleDB for price history”, “Flink CEP for sub-5s alerts against 100M rules”, “Kafka as backbone”.
   7. SAY·7If there's time, offer: “the proxy session-affinity logic”, “the continuous-aggregate refresh policy”, “the Flink broadcast-state pattern for alert rule propagation”.

   Interviewer is grading: You frame 'scrape less' as a deliberate product decision, not a limitation. You name false-alert tolerance as the harder trade-off (it is). You don't pretend a retailer-blocks-us-entirely scenario is recoverable in minutes.

Questions an interviewer is likely to ask after your walkthrough. Rehearse the short answer.

1. Why TimescaleDB and not plain PostgreSQL or InfluxDB?
2. Why Flink CEP instead of just polling the alerts table on each price event?
3. What happens when Amazon redesigns their product page?
4. How do you handle anti-bot defenses without getting IP-banned?
5. Why not just check every product every hour?
6. What about the Black Friday alert storm?
7. What if TimescaleDB has a write outage?
8. How do you prevent iPhone-for-$20 false alerts?