Card Tokenization System — 50K TPS

What is card tokenization, and why does it exist?

Imagine Raj buys a coffee at a Starbucks app. He types his 16-digit Visa card number — 4111 2222 3333 4444 — into the checkout screen. That string of digits is called the PAN (Primary Account Number — the number printed on the front of every credit card). For Starbucks to charge Raj, the PAN has to travel from his phone to a payment processor like Stripe, then to Visa, then to Raj's bank. So far so good — that's just a payment.

Now imagine Raj uses Starbucks every morning for two years. Starbucks would love to never ask him for his card again — they want to remember it. The naive solution is "store the PAN in the database." That instantly creates three nightmares:

Tokenization is the way out. It says: "We'll keep the PAN in exactly one vault, locked behind hardware encryption. Everywhere else — your application DB, your analytics, your logs, your CRM — we'll store a meaningless lookup string called a token that looks like a card number (tok_8392_1947_2055_8821) but maps to the real PAN only inside the vault." Now if your application DB leaks, the attacker walks away with tokens that are worthless without access to the vault. Your PCI scope shrinks from "everything" to "just the vault."

This page designs a tokenization service that handles 50,000 tokenize/detokenize calls per second — roughly the volume of every major Indian payment gateway combined. The bulk of the page is the high-level design (capacity, architecture, network topology, deployment, multi-region, observability). The low-level Java code lives at the bottom as an appendix.

Clarify requirements — functional & non-functional

Before drawing boxes, pin down what the system must do, what it must guarantee, and — just as important — what it will not do. This list is what you'd hand the interviewer in the first 5 minutes.

Back-of-the-envelope — does 50K TPS even fit?

Numbers ground every design decision. Before choosing Redis or Cassandra or Postgres, work out how much data we're storing, how much we're moving across the network, and how many machines we'll need. The interviewer is watching for whether you can do this without a calculator.

Traffic

Storage

Each row in the vault is roughly: token (24 B) + encrypted PAN (32 B) + merchant_id (16 B) + BIN (6 B) + last4 (4 B) + expiry (4 B) + metadata (50 B) + timestamps (16 B) ≈ ~200 bytes per record (with indices and overhead, budget 500 B).

Network & CPU

Actors & use cases — who calls this thing, and why?

The vault is an internal service; humans never touch it directly. The "actors" are other services inside our own payments platform. Naming them in a flowchart prevents the common mistake of designing the vault in isolation from the apps it serves.

Notice three things from this diagram. First, the Merchant App never sees the PAN after the initial tokenize call — it only holds tokens. Second, the Payment Gateway is the only caller allowed to fetch the full PAN (and only when it's about to send the transaction to Visa). Third, Analytics can read masked fields (BIN, last4) without unlocking the vault — this is where most "useful queries" actually happen and we want them to be cheap and safe.

API design — five endpoints, no more

A good vault has a deliberately small API. Every extra endpoint is another attack surface. Five endpoints cover 100% of the use cases above.

Why no PATCH /tokens/{token}? Cards are immutable from the vault's perspective — if the customer's card changes, the merchant tokenizes the new PAN and gets a new token. Mutability would create audit-trail nightmares.

High-Level Architecture — built up in three passes

Rather than dumping the final diagram and labelling boxes, let's discover the architecture the way you'd reason through it on a whiteboard. Three passes: naive → mental model → production. At each step a real failure forces the next piece into existence.

Pass 1 — The naive design (and exactly where it breaks)

The simplest thing that could work: one Postgres database with a table (token, pan, merchant_id, ...). Stripe → Postgres → done. Two endpoints, one table.

Now list what breaks. This is the most important sentence of any architecture interview: "This naive design fails because…"

Each of these three failures will drive one specific component into the production design — a Vault Service with HSM for security, sharded storage for throughput, and a read-through cache for latency.

Pass 2 — The mental model: split the "answer" from the "secret"

Here's the central architectural idea, and it's the one most candidates miss: the PAN and the token are two completely different objects with completely different access patterns.

Once you internalize this split, every other choice falls out naturally: encrypted PAN goes in a separate datastore from the metadata, the cache holds only metadata (never PAN), the HSM sits in front of the PAN-store and nothing else. The architecture is just the careful drawing of this fence.

Pass 3 — The production shape (numbered)

Now we draw the real architecture. Every numbered box is a component we added to solve a specific failure mode from Pass 1 or to honor the split from Pass 2. After the diagram, each number gets its own card with "what it does" and "what problem it solves."

Component-by-component — what each numbered box does

Use the numbers in the diagram above to find the matching card below. Every card has the same shape: what it does in plain language, then what would break if we removed it.

Concrete walkthrough — Raj's first Starbucks purchase

Let's trace one real request through every numbered component. Watch how the data-plane / control-plane split from Pass 2 keeps the PAN in one place.

1. Raj types 4111 2222 3333 4444 into Starbucks checkout. The Merchant SDK ① running in his browser collects the PAN and POSTs it to our API Gateway ② over TLS 1.3, presenting Starbucks' client cert.
2. Gateway validates the cert, forwards to Auth Service ③, which checks the scope is token:create only (no pan:read), then routes the request to Tokenize Service ④.
3. Tokenize Service checks Redis for the idempotency key — miss. It calls HSM ⑦ to encrypt the PAN under the current DEK, getting back ciphertext + DEK-wrapped-by-KEK. It generates a UUIDv7 token tok_018f....
4. Tokenize Service writes the ciphertext row to Vault DB ⑧ and the metadata row (BIN 411122, last4 4444) to Metadata DB ⑨ in a two-phase commit.
5. Tokenize Service publishes an audit event to Kafka ⑩ with acks=all, then returns the token to Starbucks. Total latency: ~7 ms.
6. Five months later, Raj clicks "Buy" on his daily latte. Starbucks' backend sends { token: tok_018f..., amount: 250 } to its payment gateway.
7. The payment gateway calls Detokenize Service ⑥ with scope=pan:read. Detok service hits Cache ⑤ first for the BIN routing (95% hit), then fetches ciphertext from Vault DB ⑧, asks HSM ⑦ to decrypt, gets the PAN back for 3 ms, and immediately submits to Visa.
8. Both calls are written to Kafka → ClickHouse ⑪ in real time. 90 days later they tier down to Glacier ⑫.

Latency Budget — every millisecond accounted for

"p99 ≤ 25 ms" sounds easy until you add up where the milliseconds actually go. The 25 ms budget has to cover network round-trips, mTLS handshakes (mostly cached), auth, the HSM RPC, two DB writes, a Kafka acks=all, and the response serialization. If any one of these blows its share, the budget is gone. Let's pull each flow apart by component so you can defend it on a whiteboard.

Tokenize (write path) — target p99 = 18 ms

Tokenize is the most expensive call because it hits the HSM and writes to two databases. Here's the millisecond-by-millisecond breakdown a senior engineer would walk through:

Sum p50 ≈ 10 ms · p99 ≈ 18 ms (HSM tail dominates)

Detokenize (PAN-read) — target p99 = 15 ms

Detokenize skips the metadata write but still touches the HSM. The hot read is from Vault DB (which is small enough to fit fully in page cache) rather than disk.

Sum p50 ≈ 8 ms · p99 ≈ 15 ms

Metadata read (the hot 95%) — target p99 = 5 ms

The cheap path. No HSM, no Vault DB. Just gateway + auth + Redis. This is where most of the 50K TPS goes.

Sum p50 ≈ 1.6 ms · p99 ≈ 5 ms

Where the tail (p99) actually comes from

Tokenization strategies — how do you actually generate the token?

"Generate a token" sounds trivial — give it a UUID and call it a day. In practice there are three families of approaches, each with very different consequences for the database, the cache, and the security posture.

Token format we emit: tok_<base32(uuidv7)> — e.g. tok_018f7a2b9c4d6e8f9a1b2c3d. Prefix lets logs/dashboards tell tokens apart from other IDs at a glance. Base32 (not base64) avoids URL-encoding issues.

Data model — ER diagram

We deliberately keep the PAN-bearing table in a different schema (and on a different DB) from the metadata. The ER diagram below shows both schemas as if they were one logical model — but in production they live behind different network boundaries.

Three things to note:

• TOKEN and METADATA share the same PK (token) but live in different physical databases. The join happens only inside the Vault Service — the Metadata DB never knows the ciphertext exists, and the Vault DB never holds BIN/last4 (so a vault leak doesn't reveal which cards belong to whom).
• KEK has versions. When we rotate the master key (quarterly per PCI), we don't decrypt and re-encrypt every row immediately — we mark kek_id_v2 active for new tokens, and rotate old rows in a background job. The token PK never changes.
• AUDIT_EVENT is append-only. It lives in Kafka + ClickHouse, not in the relational vault. Never updated, never deleted before the 7-year retention window expires.

Sequence diagrams — the three hot flows

Tokenize (write path)

Detokenize (PAN-read path) — separate service, stricter auth

Metadata-only read (hot path, 95% of traffic)

Token lifecycle — state diagram

Every token goes through one of these states. Transitions are the only writes that mutate an existing row.

Network Topology & PCI Zones — drawing the fence

PCI scope is defined by network reachability, not by what your code does. If a machine can open a TCP connection to a machine that holds PANs, the auditor considers it in-scope. So the network diagram is literally how you keep the auditor away from 95% of your fleet. Think of it as four concentric rings, with firewall rules that only allow traffic going inward, never sideways or outward.

Why the segmentation matters — the "blast radius" view

If an attacker compromises a pod in Zone 2 (say, a vulnerable image in the Metadata API), here's what they can and cannot reach:

PCI-DSS & security — the rules you cannot violate

PCI-DSS (Payment Card Industry Data Security Standard) is the rulebook every system touching PANs must follow. It has 12 top-level requirements; the ones that actively shape this design are the ones below. Treat them as hard constraints, not nice-to-haves — failing any of them means the auditor revokes your right to process cards.

Defense-in-depth — what we add beyond PCI minimums

• Token format reveals nothing. A token tok_018f... contains no embedded BIN, no merchant_id, no expiry — leaking a token gives the attacker an opaque blob.
• Detokenize rate-limit per actor. A compromised payment-gateway credential should not be able to bulk-exfiltrate PANs. We rate-limit detokenize to ~50 req/s per actor in non-charge contexts; bulk reads require an out-of-band approval workflow.
• Anomaly alerts. ClickHouse runs continuous queries: "actor X did 5× their 30-day average detokenize rate in the last 5 min" → page on-call.
• Per-merchant pepper. Even in deterministic mode, the HMAC key is per-merchant. Compromise of one merchant's pepper doesn't allow guessing tokens across merchants.
• No PAN in logs, ever. Logger formatters strip anything that looks like a 13–19 digit number passing the Luhn check. We unit-test this; CI fails if a print statement contains pan.

Deployment Topology — Kubernetes, HSM cluster, and the autoscaler

The architecture diagram tells you what services exist. The deployment topology tells you where they run — which K8s namespace, how many replicas, how they autoscale, and how the HSM (which is not a Pod — it's literally a physical box) plugs in. This is where interviewers test whether you've actually run a production system, or just drawn boxes.

Pod sizing — back of the envelope

Autoscaling — what the HPA actually watches

Most teams default to CPU-based HPA. For a tokenization service that's wrong because the bottleneck is the HSM, not CPU. We use custom metrics via Prometheus Adapter:

HSM cluster — outside Kubernetes

HSMs are not Pods. They're physical (or AWS-managed) appliances reached over a VPC endpoint. We run 4 HSMs per region (2 per AZ for HA). The app reaches them via an internal HSM proxy that:

• Pools connections — the HSM SDK has a limited connection budget; we share connections across pods via a sidecar/sidecar-less proxy (HashiCorp Boundary or custom).
• Hides the vendor SDK — application code talks gRPC; vendor swap (Thales → CloudHSM) only changes the proxy.
• Load-balances across HSMs — round-robin with health checks. A dead HSM is taken out of rotation in < 5 s.
• Implements circuit breaker — if 50% of HSM calls fail within 30 s, the breaker opens and tokenize returns 503 instead of queuing forever.

Multi-Region & Disaster Recovery — how to survive a region outage

One region is enough for capacity. Two regions are required for survival. The hard question is which kind of two regions: active-active (both serve writes) or active-passive (one serves, one waits). For a payment vault, the answer is active-passive with regional pinning, and we'll walk through why active-active is a trap here.

The active-active trap

Active-active feels obviously better — more throughput, faster regional failover, lower latency for users. But for tokenization, it creates a correctness problem you can't paper over:

The chosen design — active-passive with merchant pinning

What replicates, and how

Failover playbook

1. Detection (≤ 30 s). Route 53 healthchecks fail on us-east gateway. PagerDuty fires.
2. Decision (≤ 2 min). On-call confirms primary-region failure (not just a flapping check). Pages incident commander.
3. Promote (≤ 3 min). Promote us-west Vault DB replica to primary. Aurora handles this with aws rds failover-db-cluster. Repoint app config via consul.
4. Scale up (≤ 5 min). West region was running at 25% capacity (cost saving). HPA + cluster autoscaler bring it to 100% within minutes.
5. Cut DNS (≤ 1 min). Route 53 weighted-routing shifts all traffic to us-west. TTL is 30 s, so worldwide propagation completes in ~1 min.
6. Verify audit trail (continuous). Check that MirrorMaker 2 has caught up; if Kafka audit lag > 60 s, alert security on a possible compliance gap.

Why a separate EU region

EU GDPR + PSD2 require EU cardholder data to stay in the EU. EU merchants are pinned to eu-west-1 at the Route 53 layer; their PANs never touch us-east. The EU region is its own isolated tokenization vault with its own HSM keys — a US-region breach doesn't expose EU cards. The cost is operating two parallel vaults, but compliance leaves no choice.

Scaling to 50K TPS — where the bottlenecks actually live

You almost never scale "the database" first. The three real bottlenecks in a tokenization vault, in order, are: HSM throughput, audit-log durability, and cache miss storms. Solving them is most of the scaling work.

Sharding the Vault DB

500M tokens × 500 B = 250 GB. Fits one node, but we shard for write throughput and blast radius:

• Shard key: hash(token) with consistent hashing across 16 shards. Why not merchant_id? Because one huge merchant (Amazon) would be a hot shard. Hashing the token guarantees uniform distribution.
• Per-shard: primary + 2 replicas (sync to one, async to the other for cross-region DR). Postgres logical replication on AWS RDS Multi-AZ, or Aurora Global if going cross-region.
• Routing: the Vault Service holds the shard map in memory, refreshed from a coordination service (etcd) every 30 s.

Service tier scaling

Both Tokenize Service and Detokenize Service are stateless — scale by adding pods behind an L4 load balancer. Per-pod budget: ~500 req/s comfortably (Go) or ~300 req/s (Java with JIT warm). For 50K TPS plan ~150 pods total across regions. CPU scales linearly; memory is mostly Redis client pools and connection pools.

Observability — SLOs, golden signals, and the alerts you can't sleep through

A payment vault that's "down" but not alerted is worse than no vault at all — every payment in flight times out, merchants disable cards, customer support floods. So observability isn't optional plumbing; it's part of the product. Three layers: SLOs set the contract, metrics + traces tell you whether you're meeting it, and alerts wake the on-call when you're not.

Service Level Objectives (SLOs)

Once the error budget is burned, all new feature deploys are frozen until the budget resets. This is the SRE contract — it forces the team to invest in reliability, not just feature velocity.

Golden signals — what every dashboard shows

Distributed tracing

Every request gets an OpenTelemetry trace from gateway → service → HSM → DB → Kafka. Sampled at 1% in steady state, 100% for any request flagged as anomalous (e.g., 5xx, p99 outlier). Traces let an engineer answer "where did this one slow request spend its time?" without log-grepping.

Security-specific telemetry — the alerts that matter for tokenization

Failure modes — what breaks, and what we do about it

For each failure, the question is the same: does the system corrupt data, return wrong data, or just fail loudly? Tokenization can never afford the first two — better to fail loud than to silently mint a token whose PAN cannot be retrieved.

LLD appendix — Class diagram

The HLD is the system view. The LLD here zooms into one box from the architecture diagram — the Tokenize Service (④) — and shows the classes inside it. Reserved for the last 15 minutes of an interview if asked "now show me the code."

Design patterns at work

LLD appendix — Java implementation

Java 17+, no frameworks. The interesting parts are the orchestration in TokenizationService.tokenize and the envelope-encryption sequence in HsmEncryptionService.encrypt. Everything else is plumbing.

Enums & records

public enum TokenStrategy { RANDOM, DETERMINISTIC }
public enum TokenStatus   { ACTIVE, REVOKED, EXPIRED, ROTATING }
public enum CardNetwork   { VISA, MASTERCARD, AMEX, RUPAY, DISCOVER }

public record Ciphertext(byte[] cipher, byte[] wrappedDek, String kekId) {}
public record VaultRecord(String token, String merchantId, Ciphertext payload,
                          TokenStatus status, Instant createdAt, Instant revokedAt) {}
public record MetadataRecord(String token, String bin, String last4, CardNetwork network,
                             String expiryYYYYMM, String merchantId, TokenStatus status) {}
public record AuditEvent(String eventId, Instant ts, String actor, String action,
                         String token, String merchantId, String ip, String scopeUsed) {}
public record TokenizeResponse(String token, String bin, String last4, CardNetwork network) {}
public record DetokenizeResponse(String pan, String expiryYYYYMM) {}

Request with Builder

public final class TokenizeRequest {
  private final String pan, expiryYYYYMM, merchantId, idempotencyKey, ip;
  private final TokenStrategy strategy;

  private TokenizeRequest(Builder b) {
    this.pan = b.pan; this.expiryYYYYMM = b.expiry;
    this.merchantId = b.merchantId; this.idempotencyKey = b.idem;
    this.ip = b.ip; this.strategy = b.strategy;
  }
  public String pan(){ return pan; }
  public String expiry(){ return expiryYYYYMM; }
  public String merchantId(){ return merchantId; }
  public String idempotencyKey(){ return idempotencyKey; }
  public String ip(){ return ip; }
  public TokenStrategy strategy(){ return strategy; }

  public static Builder builder(){ return new Builder(); }
  public static final class Builder {
    private String pan, expiry, merchantId, idem, ip;
    private TokenStrategy strategy = TokenStrategy.RANDOM;
    public Builder pan(String v){ this.pan = v; return this; }
    public Builder expiry(String v){ this.expiry = v; return this; }
    public Builder merchantId(String v){ this.merchantId = v; return this; }
    public Builder idempotencyKey(String v){ this.idem = v; return this; }
    public Builder ip(String v){ this.ip = v; return this; }
    public Builder strategy(TokenStrategy v){ this.strategy = v; return this; }
    public TokenizeRequest build(){ return new TokenizeRequest(this); }
  }
}

HSM envelope encryption (the security-critical bit)

public final class HsmEncryptionService implements EncryptionService {
  private final HsmClient hsm;            // vendor SDK, thread-safe
  private final String   currentKekId;
  private static final SecureRandom RNG = new SecureRandom();

  public Ciphertext encrypt(byte[] plaintext) {
    byte[] dek = new byte[32]; RNG.nextBytes(dek);     // fresh DEK per record
    byte[] iv  = new byte[12]; RNG.nextBytes(iv);
    byte[] cipher = AesGcm.encrypt(plaintext, dek, iv);  // local AES-256-GCM
    byte[] wrapped = hsm.wrap(currentKekId, dek);        // KEK never leaves HSM
    Arrays.fill(dek, (byte)0);                            // best-effort zeroize
    return new Ciphertext(concat(iv, cipher), wrapped, currentKekId);
  }

  public byte[] decrypt(Ciphertext c) {
    byte[] dek = hsm.unwrap(c.kekId(), c.wrappedDek());
    try {
      byte[] iv      = Arrays.copyOfRange(c.cipher(), 0, 12);
      byte[] payload = Arrays.copyOfRange(c.cipher(), 12, c.cipher().length);
      return AesGcm.decrypt(payload, dek, iv);
    } finally {
      Arrays.fill(dek, (byte)0);
    }
  }
}

The orchestrating service

public final class TokenizationService {
  private final Map<TokenStrategy, TokenGenerator> generators;
  private final EncryptionService encryption;
  private final VaultRepository    vault;
  private final MetadataRepository metadata;
  private final AuditPublisher     audit;
  private final IdempotencyStore   idem;
  private final Clock              clock;

  public TokenizeResponse tokenize(TokenizeRequest req) {
    // 1) idempotency
    Optional<String> existing = idem.get(req.idempotencyKey());
    if (existing.isPresent()) {
      MetadataRecord m = metadata.findByToken(existing.get()).orElseThrow();
      return new TokenizeResponse(m.token(), m.bin(), m.last4(), m.network());
    }
    // 2) mint
    String token = generators.get(req.strategy())
        .generate(req.pan(), req.merchantId(), req.strategy());
    // 3) encrypt
    Ciphertext payload = encryption.encrypt(req.pan().getBytes(StandardCharsets.UTF_8));
    // 4) two writes
    Instant now = clock.instant();
    vault.save(new VaultRecord(token, req.merchantId(), payload, TokenStatus.ACTIVE, now, null));
    String bin = req.pan().substring(0, 6);
    String last4 = req.pan().substring(req.pan().length() - 4);
    CardNetwork network = NetworkResolver.fromBin(bin);
    metadata.save(new MetadataRecord(token, bin, last4, network,
        req.expiryYYYYMM(), req.merchantId(), TokenStatus.ACTIVE));
    // 5) audit (sync)
    audit.publish(new AuditEvent(UUID.randomUUID().toString(), now,
        req.merchantId(), "TOKENIZE", token, req.merchantId(), req.ip(), "token:create"));
    // 6) memoize idempotency
    idem.putIfAbsent(req.idempotencyKey(), token, Duration.ofHours(24));
    return new TokenizeResponse(token, bin, last4, network);
  }

  public DetokenizeResponse detokenize(String token, String actor, String ip) {
    VaultRecord r = vault.findByToken(token)
        .orElseThrow(() -> new NotFoundException(token));
    if (r.status() != TokenStatus.ACTIVE) throw new GoneException("token revoked");
    byte[] pan = encryption.decrypt(r.payload());
    audit.publish(new AuditEvent(UUID.randomUUID().toString(), clock.instant(),
        actor, "DETOKENIZE", token, r.merchantId(), ip, "pan:read"));
    MetadataRecord m = metadata.findByToken(token).orElseThrow();
    try {
      return new DetokenizeResponse(new String(pan, StandardCharsets.UTF_8), m.expiryYYYYMM());
    } finally {
      Arrays.fill(pan, (byte)0);     // do not keep PAN bytes in heap
    }
  }
}

Thread-safety in one sentence

TokenizationService is stateless once constructed; all collaborators are themselves thread-safe (HSM client connection pool, Redis client, JDBC pool, Kafka producer). The only mutable secret on heap is the DEK byte array, which we zeroize in a finally block — best-effort, since GC may have moved it; the HSM is the real boundary.

Trade-offs & interview talking points

The "right answer" in an interview isn't a design — it's recognising what you traded away. Walk through these out loud and you signal seniority.

Extension points (Open/Closed)

• New encryption backend. Implement EncryptionService with AWS KMS, GCP CloudKMS, or HashiCorp Vault — no change in TokenizationService.
• New token strategy. Add a FpeTokenGenerator implementing TokenGenerator and register it in the strategy map. Existing code is untouched.
• New audit sink. Add another consumer to the Kafka topic — say, a SIEM (Splunk, Datadog) — without changing the producer side.
• Card-on-file network token rails. Visa and Mastercard now offer "network tokens" (issuer-side tokens). Plug them in as an alternate VaultRepository implementation that calls Visa Token Service instead of writing to Postgres.
• Multi-region read replicas. Wrap VaultRepository + MetadataRepository in routing decorators that pick the closest replica on reads.

Likely interviewer follow-ups