System Design: Payment System (Checkout, Two Gateways, Webhooks, Marketplace Payouts)
Goal: Design the payment system of Trailhead, an outdoor gear marketplace in the US and India with about 2 million orders a day. Customers pay through two gateways, and sellers are paid after delivery. Nobody pays twice without an automatic refund, and nothing unpaid ships.
1. What We Are Building
🔒 Premium section
2. Requirements and Scale
2.1 What the System Must Do
- FR-01 Checkout with stock held. Stock is held while the customer pays (section 7).
- FR-02 Card, UPI and wallets. Cards in both countries, UPI in India (section 8).
- FR-03 One payment, many sellers. Split to each seller (section 11).
- FR-04 Confirm only paid orders. Never ship anything unpaid (section 6).
- FR-05 Cancel and refund per item (section 13).
- FR-06 Two gateways. Route each payment and fail over safely (section 10).
- FR-07 Seller payouts. After the return window, with statements (section 11).
- FR-08 Chargebacks. Send evidence; recover from the seller at fault (section 13).
- FR-09 Taxes. US sales tax as the marketplace; Indian taxes withheld from sellers (section 11).
- FR-10 Daily reconciliation. Against gateways and the bank (section 14).
- FR-11 Saved cards. One-click repeat purchase, P1 (after launch) (section 10).
- FR-12 Membership subscription. Trailhead+, P1 (section 15).
2.2 How Well It Must Do It
- NFR-01 Never charge twice, never ship unpaid. Any second charge is refunded automatically.
- NFR-02 The ledger balances. Every seller payable traces back to orders (section 12).
- NFR-03 Checkout is up 99.95% of the time. That's about 22 minutes in a 30-day month. The gateways bound it, and two gateways help.
- NFR-04 Under 300 ms added at p99. p99 means 99 of 100 calls are faster. Gateway time is extra.
- NFR-05 No oversell in a flash sale, at 3,000 checkout starts a second (section 7).
- NFR-06 Card data never touches our servers. PCI DSS is the card industry's security standard. Its shortest self-assessment, SAQ A, fits shops that hand all card entry to the gateway (section 8.1).
- NFR-07 Every payment is final within 5 days. Gateways resolve delayed bank results within that time (section 8.4).
- NFR-08 Sellers are paid on time and exactly (section 11).
- NFR-09 Every gateway transaction and payout is reconciled daily (section 14).
2.3 The Numbers
An attempt is one try to pay with one method, so a declined card and a second card make two. A seller part is the share of an order that belongs to one seller. Each part gets its own capture and ledger lines.
| Item | Value | How we got it |
|---|---|---|
| Orders a day | 2 million | US 0.8 million, India 1.2 million |
| Average rate | about 23 orders a second | 2 million / 86,400 seconds |
| Normal daily peak | about 100 orders a second | evening hour, about 4 times the average |
| Flash sale | 3,000 checkout starts a second; at most 1,000 paying checkouts a second | a waiting room, a queue page that lets buyers into payment at a fixed rate |
| Payment attempts | about 2.6 million a day, 130 a second at normal peak | about 1.3 attempts per order |
| Seller parts | about 3 million a day | 1.5 sellers per order |
| India mix | UPI 70%, cards 20%, net banking (paying from a bank's own website) and wallets 10% | about 840,000 UPI orders a day |
| US mix | cards and wallets 95%, other 5% | |
| Active sellers | 200,000, paid daily | |
| Webhooks in | 8 million a day (about 93 a second on average, 400 at the evening peak); we plan for 1,000, and up to 12,000 after a flash sale | about 4 events per order |
Is 3,000 a second realistic? Shopify load-tested 80,000+ checkouts a minute in 2025, about 1,300 a second, across all its stores. Only the front takes all 3,000 a second. Payment gets only what the waiting room (section 16) lets in.
Apart from sales, 100 orders a second is small for a database. The hard part is being right when calls repeat, results arrive late and servers crash halfway.
2.4 Out of Scope
The gateway stores cards, checks seller identity and scores fraud, so we don't. We also leave out cash on delivery, buy now pay later, crypto and pricing.
Shipping only sends us "ready to ship", "delivered" and "returned" events. A tax service gives us each tax amount.
3. Who Is Involved
🔒 Premium section
4. The Checkout and Payment API
🔒 Premium section
5. High-Level Design
🔒 Premium section
6. Deep Dive: Checkout as a Workflow
🔒 Premium section
7. Deep Dive: Stock Holds and Flash Sales
🔒 Premium section
8. Deep Dive: Talking to Gateways Safely
🔒 Premium section
9. Deep Dive: Gateway Webhooks
🔒 Premium section
10. Deep Dive: Two Gateways
🔒 Premium section
11. Deep Dive: Marketplace Money
🔒 Premium section
12. Deep Dive: The Marketplace Ledger
🔒 Premium section
13. Refunds, Cancellations and Chargebacks
🔒 Premium section
14. Reconciliation
🔒 Premium section
15. Subscriptions
🔒 Premium section
16. Capacity
🔒 Premium section
17. What Happens When Each Part Goes Down
🔒 Premium section
18. Bottlenecks
🔒 Premium section
19. Regions and Data
🔒 Premium section
20. Security and Privacy
🔒 Premium section
21. Running It
🔒 Premium section
22. Trade-offs
🔒 Premium section
23. In the Interview
🔒 Premium section
24. How We Measured
🔒 Premium section
25. Further Reading
🔒 Premium section
Explore the Technologies
🔒 Premium section