Design a Tagging System

Each phase is what the interviewer expects you to do and say. Concrete steps, not topic hints. The diagrams are what you sketch on the board.

1. 1

   Clarify Requirements and Scale

   5 min

   GoalEstablish that this is a three-tier consistency problem, not a single-store problem. Get the interviewer to agree that tag CRUD, discovery, and analytics each have different SLAs.

   Do & Say

   1. ASK·1ASK: “How many tenants and what's the isolation requirement?” Land on “10K tenants”, “200 users each”, “2M total”. Write “10K tenants, 200K concurrent, 3 regions (US/EU/Asia)” on the board.
   2. SAY·2Now the load-bearing question: “Do tag operations, discovery queries, and view analytics need the same consistency?” Three tiers: “Tag CRUD needs strong consistency (no duplicate tags)”, “Discovery can tolerate ~30s staleness (caching is fine)”, “View counts are eventual, 2-minute global aggregation lag is acceptable”.
   3. SAY·3Lock the latency targets: “Tag creation ~100ms globally including the cross-region round-trip”, “Product discovery under 25ms p99”, “View-count ingest under 5ms (the click should never feel slow)”.
   4. SAY·4Pin the throughput: “35 tag creates/sec”, “700 item-tag mappings/sec”, “9,300 view events/sec”. “The analytics tier dominates.” Write all three on the board.
   5. SAY·5Park out of scope. In scope: “tag CRUD with global uniqueness”, “tag-to-content association”, “content discovery by tag”, “popular tags dashboard”. Out: “full-text search inside content”, “tag suggestions/autocomplete-ranking”, “recommendation models”. Wait for the nod.

   Interviewer is grading: You volunteer the three-consistency-tier framing in the first minute. You don't try to bolt strong consistency onto the click-counting path.

2. 2

   Data Model and Deduplication

   5 min

   GoalShow how deterministic UUIDs plus a UNIQUE constraint eliminate the cross-region race without a global lock. Sketch the schema with tenant_id everywhere.

   Do & Say

   1. DRAW·1Sketch the tags table columns: “tag_id (STRING, deterministic UUID from hash(tenant_id + normalized_name))”, “tenant_id”, “normalized_name (lowercase, trimmed)”, “display_name”, “created_region”, “status”, “timestamps”. Constraint: “UNIQUE(tenant_id, normalized_name)”.
   2. SAY·2Justify the deterministic UUID. “Two users in Tokyo and London create the tag marketing at the same millisecond.” “Both compute the same UUID from hash(tenant_id + marketing).” “The first INSERT wins, the second hits the UNIQUE constraint, fetches the existing row, returns the same tag_id.” “No coordination protocol needed.”
   3. DRAW·3Sketch item_tags columns: “id”, “tenant_id”, “tag_id (FK)”, “item_id (FK)”, “created_region”, “UNIQUE(tenant_id, item_id, tag_id)”. Composite indexes: “(tenant_id, tag_id) for product-discovery”, “(tenant_id, item_id) for tag-listing”.
   4. DRAW·4Sketch the analytics tables: “tag_increment_events for regional buffer flushes”, “tag_lifetime_stats keyed by (tenant_id, tag_id) with total_lifetime_views plus a regional_breakdown JSON blob”.
   5. SAY·5Justify tenant_id everywhere. “Every row, every cache key, every Kafka message carries tenant_id.” “Sharding is by tenant_id so a single tenant's data is co-located.” “Cross-tenant queries are physically impossible from the API layer.”

   Interviewer is grading: You name the deterministic UUID trick by what it accomplishes (eliminates global coordination), not just buzzword it. You mention the UNIQUE constraint as the final safety net, not the primary mechanism.

3. 3

   High-Level Design (three consistency tiers, three data paths)

   10 min

   GoalOne diagram with the three tiers visible: Spanner strong for tag CRUD, Valkey plus Elasticsearch for discovery, Redis-buffered Kafka pipeline for analytics. Label each path with its latency.

   Draw on the board
   Do & Say

   1. DRAW·1DRAW Tier 1, Tokyo tag creation. Flow: “API computes deterministic UUID”, “checks regional Redis (2ms)”, “INSERTs Spanner with UNIQUE (90ms)”, “on conflict, fetches existing tag”, “updates Redis, publishes Kafka for ES”. Total: “~100ms globally via Spanner TrueTime, no coordination protocol”.
   2. SAY·2Label the alternative kill. “Why Spanner not PostgreSQL?” “PostgreSQL is single-master, multi-region writes require manual sharding or active-passive failover.” “CockroachDB would work as an open-source alternative.” “DynamoDB Global Tables are eventually consistent, which fails the no-duplicate-tags requirement.”
   3. DRAW·3Draw Tier 2, London discovery. “API hits regional Redis tag_items cache (~2ms).” “95% of queries hit, return in 15ms.” “Miss falls to regional Elasticsearch (~18ms).” “ES unavailable falls back to Spanner direct (~40ms).” “Three-layer read fallback, three-layer latency budget.”
   4. SAY·4Justify Elasticsearch. “Pre-aggregated for popular tag-item combinations.” “Indexed asynchronously from Kafka events, up to 30s lag.” “Tenant_id is a hard filter on every query, no cross-tenant exposure.” “Cross-region replication every 2-3 minutes, which is fine because discovery tolerates that staleness.”
   5. DRAW·5Draw Tier 3, Sydney click ingest. Flow: “API does Redis INCR on live:tenant123:marketing:asia (~2ms), returns 200 immediately”, “Async publish to regional Kafka”, “Every 30s, batch processor does GETSET to atomically read-and-reset the counter”, “Writes a backup key”, “Flushes to Spanner tag_increment_events”, “Deletes backup only on successful commit”.
   6. SAY·6Explain the global aggregator. “Every 2 minutes, the US region (designated global aggregator) sums unprocessed regional events grouped by (tenant_id, tag_id), UPDATEs tag_lifetime_stats, marks events processed.” “No cross-region cache invalidation because each region rebuilds popular-tag cache on its own 5-minute TTL from Spanner.”

   Interviewer is grading: You separate the three tiers visually so the interviewer sees that strong consistency only touches Spanner, never the click path. You volunteer that CockroachDB is a viable open-source alternative without being asked about cloud lock-in.

4. 4

   Deep Dive: Zero Data Loss on Clicks, and the Race on Tag Creation

   15 min

   GoalTwo sub-dives. First, why the GETSET-with-backup pattern survives processor crashes. Second, why deterministic UUIDs plus UNIQUE constraint actually resolve the cross-region race, with numbers.

   Draw on the board
   Do & Say

   1. SAY·1Pivot to click counting. “The naive design loses clicks on processor crash.” Sequence: “INCR returns to the user”, “processor reads counter”, “crashes before writing to DB”, “restart reads 0, counts are gone”.
   2. SAY·2GETSET-with-backup. “Step 1 GETSET counter 0, atomic read+reset.” “Step 2 SET backup:t:tag:region count, durable in Redis.” “Step 3 INSERT Spanner.” “Step 4 DEL backup only on commit OK.” “Crash anywhere between 1-4, backup survives.” “Next tick INCRBYs it back and retries.”
   3. SAY·3Redis loss. “Redis down, we lose live counter and in-flight backups.” “Circuit breaker activates, click API writes directly to Kafka (~10ms instead of 2ms).” “Counter rebuilds from Kafka replay when Redis recovers.” “Clicks never drop, worst case is degraded latency.”
   4. SAY·4Pivot to the create race. “Two users on opposite continents type marketing into the same tenant at the same millisecond.” “Without coordination, you get two tags with two different IDs and the second one breaks every existing item_tag row that uses the first.”
   5. SAY·5Deterministic UUID resolution. “Both regions compute hash(tenant_id + normalized_name).” “Same input, same UUID anywhere.” “Tokyo INSERT wins in ~90ms via Spanner TrueTime.” “London arrives ~10ms later, hits UNIQUE, exception handler fetches existing tag by the same UUID.” “London pays ~85ms.” “No lock, no 2PC, no coordination.”
   6. WATCH·6Be ready for the cache-coherence question. “Tokyo's cache now has the new tag.” “London's cache doesn't, but the next London cache miss queries Spanner directly, finds the tag (Spanner replicates synchronously), populates London's local cache.” “No cross-region invalidation broadcasts needed.” “Spanner is the coordination layer, not Kafka.”
   7. SAY·7Stampede protection. “If a regional Redis dies, every tag lookup falls through to Spanner simultaneously.” “Circuit breaker caps concurrent Spanner queries per pod at 50.” “Pod-local in-memory LRU of 10K recently-seen tags serves as a fourth fallback layer.”
   8. SAY·8Hot tag contention. “Viral tag gets thousands of clicks/sec per region.” “Redis INCR on one key is single-threaded but handles ~100K ops/sec, well above any realistic hot-tag rate.” “30s batch flush means global totals lag the regional view, fine for a dashboard that refreshes every 5 min.”

   Interviewer is grading: You walk the GETSET-backup-DEL sequence step by step rather than waving at 'atomic'. You explicitly say Spanner, not Kafka, is the coordination layer for tag creation. You volunteer the 100K ops/sec Redis ceiling as why hot-tag contention isn't a problem here.

5. 5

   Trade-offs, Risks, and Wrap-up

   5 min

   GoalName the three consistency-tier trade-offs explicitly, the Spanner lock-in trade-off, and close with one sentence that ties the design to the requirement.

   Do & Say

   1. SAY·1Tier trade-offs. “Strong CRUD costs 100ms, accepted because creation is rare (35/sec) and duplicates break everything downstream.” “Eventual analytics buys 2ms click latency, dashboard tolerates 2-min lag.” “Cached discovery 15ms with 30s staleness, fine since new tags don't need millisecond visibility.”
   2. SAY·2Spanner lock-in. “Spanner gives us multi-region strong-consistent writes with no operational overhead, but it's GCP-only.” “CockroachDB is the open-source escape hatch.” “If we run on AWS, we'd accept the operational cost of CockroachDB or fall back to single-region Postgres plus active-passive with a longer failover window.”
   3. SAY·3ES indexing lag. “New tag-item associations take up to 30s to appear in search.” “Users who tag and immediately search may miss results.” Mitigation: “fall back to Spanner direct when ES results look suspiciously empty (under 5 when Spanner shows more)”. “Slower path, but never stale.”
   4. SAY·4Tenant isolation. “Every row, cache key, Kafka message, and ES document carries tenant_id.” “Per-tenant rate limits prevent noisy neighbors.” “mTLS plus JWT (15-min TTL) on all internal calls.” “Cross-tenant access can't happen because tenant_id is baked into the constraint and the index.”
   5. SAY·5Close in one breath: “Three consistency tiers matched to three workloads”, “Deterministic UUIDs + Spanner UNIQUE eliminate the cross-region race with no coordination”, “Redis-to-ES-to-Spanner cascade keeps discovery at 15ms while durable”, “GETSET-with-backup gives zero click loss without taxing click latency”. Biggest risk: “ES indexing lag, mitigated by Spanner fallback on empty results”.
   6. SAY·6If there's time, offer follow-ups: “the global aggregator implementation”, “Spanner interleaved tables for tag-content co-location”, or “the per-tenant rate limit design”.

   Interviewer is grading: You match each consistency tier to its specific user-facing requirement, not just buzzword them. You name CockroachDB as the lock-in escape hatch unprompted.

Questions an interviewer is likely to ask after your walkthrough. Rehearse the short answer.

1. Why deterministic UUIDs instead of a distributed lock?
2. Why Spanner over PostgreSQL or DynamoDB?
3. How do you not lose clicks on a processor crash?
4. What happens if Redis dies entirely?
5. How is cross-region cache coherence handled on tag creation?
6. Why is Elasticsearch in the mix at all?
7. How does tenant isolation prevent cross-tenant leakage?
8. Why is hot-tag click contention a non-issue?