Explain the Design, Then Defend the Decisions

Guide 5 of 5 · Interview workbook and case studies

Series overview · Previous: Reliability, security, and operations

1. Treat the interview as a design discussion

An interviewer asks, "Design a library reservation system." Do not begin by listing databases and queues. First find out what the library needs.

A good answer connects decisions to evidence: "Reservations need a transaction because two students may request the same physical copy." That is stronger than naming a database without explaining its job.

There can be several good designs, but not every design is valid. A design that permits two confirmed reservations for one copy violates our requirement, even if its diagram looks impressive.

2. A flexible 45-minute structure

Time	Activity	Useful result
0–5 minutes	Clarify users, features, exclusions, and invariants	Agreed scope
5–9 minutes	Clarify scale, latency, availability, and consistency	Assumptions and useful estimates
9–15 minutes	Define entities and APIs	Concrete contracts
15–25 minutes	Draw a working design and trace read/write paths	End-to-end behavior
25–38 minutes	Explore two or three important risks	Defensible trade-offs
38–45 minutes	Cover operations and summarize limitations	Recovery, monitoring, and next steps

Adapt this to the interview. Ask which area deserves more depth. A calculation should support a decision, not consume time without changing the design.

A useful speaking pattern is: "Because we need X, I would choose Y. It gives us Z, but costs W. I would revisit it if condition V changes."

3. Worked case study: reserve a library copy

This is an original teaching design. All scale and service targets below are assumptions, not measurements of a real system.

Step A: agree on scope

Students search the catalog, reserve an available physical copy, view their reservations, and cancel before collection. Library staff mark a reservation ready and collected.

Exclude delivery, recommendation ranking, cross-school transfers, and complex waiting lists from this first version.

The key invariant is that one copy has at most one active reservation. Search availability may be slightly stale, but a confirmation must come from authoritative data.

Assume the workload from guide 1: 100,000 daily active students, about 231 peak searches/s under an assumed 10× peak factor, and 10,000 reservations/day. Propose p95 search latency below 300 ms and a 99.9% request success SLO, then confirm whether they fit the interviewer's needs.

Step B: define the data

Entity	Important fields	Why it exists
Book	book_id, title, author	Describes a title
Copy	copy_id, book_id, school_id, status	Tracks an actual physical item
Reservation	reservation_id, copy_id, student_id, status, expires_at	Tracks the business action
IdempotencyRecord	actor_id, operation, key, request_hash, saved_result	Resolves duplicate requests
OutboxEvent	event_id, aggregate_id, sequence, type, payload, publication_state	Records an event that must be published

Distinguish a title from a physical copy. Five copies of one title can be reserved by five students. The invariant belongs to the physical item.

erDiagram
    BOOK ||--o{ COPY : has
    COPY ||--o{ RESERVATION : has_history
    STUDENT ||--o{ RESERVATION : makes
    RESERVATION ||--o{ OUTBOX_EVENT : produces

The Copy-to-Reservation relationship includes historical reservations. A separate database constraint limits active ones; the diagram alone does not enforce that rule.

Step C: define the API

Endpoint	Purpose	Important rule
GET /books?query=clouds&cursor=...	Search with pagination	Bound page size
POST /reservations	Reserve a copy	Authenticate, authorize, and accept an idempotency key
GET /reservations/{id}	Read a reservation	Check ownership or staff permission
DELETE /reservations/{id}	Cancel under the API contract	Enforce valid lifecycle transition
POST /reservations/{id}/collection	Staff records collection	Require staff authorization

For POST /reservations, the body can contain copy_id. Derive the actor from validated authentication. Return 201 with the reservation ID after commit, 409 if the copy is no longer available, and an appropriate validation or permission error otherwise.

An idempotency key is scoped to the authenticated actor and operation. Save a hash of the meaningful request fields. Reusing the same key with different data is a conflict, not a second action.

Step D: draw the first useful architecture

flowchart TD
    U[Student app] --> G[HTTPS entry and load balancing]
    G --> A[Library app instances]
    A --> C[(Optional catalog cache)]
    A --> D[(Relational database and outbox)]
    R[Outbox relay] --> D
    R --> Q[Notification queue]
    Q --> W[Notification worker]
    W --> N[Notification provider]
    U --> E[CDN for public covers]
    E --> O[(Cover object storage)]

The app can begin as a modular monolith. The cache is optional at this scale. Redundancy of the database, queue, and entry layer is a deployment detail to specify during the reliability discussion; this drawing does not imply that one database machine is enough for the target.

Search reads can use the catalog cache or database. Reservation writes go to authoritative storage. Notifications happen after the reservation commits, so a slow notification provider does not undo a successful reservation.

Step E: protect the last copy

Imagine two students pressing Reserve at the same moment. Both seeing "available" on a screen is harmless. Both receiving a confirmed reservation is not.

Use one local transaction to claim the copy, create the reservation, save the idempotent result, and add its outbox event. For a PostgreSQL-style schema, a partial unique index on copy_id for active reservation statuses can provide a final constraint. The exact DDL depends on the lifecycle model.

Illustrative transaction outline, not production-ready code:

BEGIN
  Claim (actor_id, operation, idempotency_key) using a unique constraint.
  If another committed result owns that key:
      Check the request hash and return the saved result.
  If a concurrent transaction owns it:
      Wait within a deadline, then re-read or return retry guidance.

  Atomically change the requested copy from AVAILABLE to RESERVED.
  If no row changed:
      Finish with an unavailable result under the chosen key policy.

  Insert the active reservation.
  Insert ReservationCreated into the outbox.
  Store the success result for this idempotency key.
COMMIT
Return the committed result.

In a database that supports conditional row updates, the claim can take this form:

UPDATE copies
SET status = 'RESERVED'
WHERE copy_id = :copy_id AND status = 'AVAILABLE';

Check the affected-row count and keep this update in the same transaction as the reservation insert. Cancellation and expiration must also update reservation and copy state together. Handle serialization or deadlock retries where the chosen database requires them.

An atomic claim makes one contender win. The constraint is a second line of protection. A normal cache read followed by an unrelated write does not provide the same guarantee.

Step F: handle a lost response

sequenceDiagram
    participant S as Student app
    participant A as Library app
    participant D as Database
    S->>A: Reserve copy with key K
    A->>D: Commit reservation, result K, and outbox
    D-->>A: Committed
    Note over S,A: Success response is lost
    S->>A: Retry same request with key K
    A->>D: Read committed result for K
    D-->>A: Existing reservation ID
    A-->>S: Return saved success

The first transaction may have succeeded even when the client saw a timeout. The stable key lets the system resolve uncertainty without creating another reservation.

Choose an idempotency retention period that covers the expected retry window. If records expire, the client contract must explain what an old retry can do. The reservation invariant should remain enforced independently.

Step G: discuss failures and recovery

Failure	User-visible behavior	Recovery
One app crashes before commit	Request fails or times out	Transaction rolls back; retry safely
App crashes after commit	Client may be uncertain	Retrieve saved result by idempotency key
Cache unavailable	Search may be slower	Fall back with bounded database load
Notification broker unavailable	Reservation remains confirmed	Outbox relay publishes after recovery
Worker repeats a message	Duplicate delivery is possible	Deduplicate effects; use provider idempotency where available
Database leader unavailable	Writes pause during safe recovery	Fail over under stated replication guarantees
Catalog search is stale	Student may see an unavailable copy	Reject conflicting reservation against authoritative state
Reservation expiry worker is delayed	Copy may remain held longer	Monitor overdue holds and reconcile with atomic transitions

Measure confirmed reservation success, latency, outbox age, notification lag, and invariant violations. Use request tracing to connect an API request to its event and worker result.

Step H: explain how it grows

First measure. Optimize indexes and query plans. Add app instances when application capacity is the limit. Add a suitable search system when search needs exceed simple database queries. Add read replicas only for reads that tolerate their consistency behavior.

If one database eventually cannot handle the workload, a school-based shard key may keep most operations local. Test for large-school hotspots and cross-school requirements. Migration needs backfill, verification, and a controlled cutover; "add sharding" is not a complete plan.

A concise interview closing answer

"I chose a modular application and relational store because the central risk is concurrent reservation of one physical copy. A transaction and database constraint protect that rule. Catalog reads can tolerate limited staleness, but confirmation cannot. An outbox keeps notifications recoverable without placing the provider in the reservation's critical path. I would measure database pressure before adding sharding or more services."

4. Second case: asynchronous school reports

An administrator requests a large report. Returning 202 quickly is useful, but only if the work was durably accepted.

Store a job with QUEUED, RUNNING, SUCCEEDED, or FAILED state. Use an outbox or durable dispatcher to enqueue it. Workers claim jobs with a lease, generate the report, upload it to private object storage, and record the result.

stateDiagram-v2
    [*] --> Queued
    Queued --> Running: Worker claims lease
    Running --> Succeeded: Result stored and recorded
    Running --> Queued: Retryable failure or expired lease
    Running --> Failed: Attempts exhausted
    Failed --> Queued: Approved replay
    Succeeded --> [*]

Use attempt identifiers or fencing so an old worker cannot overwrite the result after its lease expires and a new worker takes over. A stable object key or recorded result identity helps avoid uncontrolled duplicate artifacts.

Define the report's data boundary. Does it represent data at request time, worker start time, or an explicitly selected snapshot? A report that mixes changing pages may not represent any real point in time.

Apply access control both when requesting the report and downloading it. Expire results according to the product's retention policy. Monitor the oldest queued job, not just the queue length.

Interview lesson: accepting work, performing work, and publishing the result are distinct steps. Explain the recovery between each pair.

5. Third case: a classroom update feed

A teacher posts an announcement and students see it live. Start with ordinary storage and a read API. Add SSE if server-to-browser delivery meets the interaction needs; add WebSocket if frequent two-way interaction is central.

Keep durable announcement IDs and a cursor for resume. Each connection is authorized for a classroom. On reconnect, replay permitted retained messages or return a current snapshot if history is no longer available.

Do not confuse presence with durable truth. A disconnected student may still be studying. "Online" is often a best-effort signal inferred from heartbeats and expiry.

Connection capacity and request rate are different. A service may hold many idle connections but struggle with a sudden announcement to every classroom. Bound per-client buffers and disconnect or resynchronize slow consumers instead of allowing unlimited memory growth.

Interview lesson: choose the transport after the communication pattern, then explain history, authorization, reconnects, ordering, and slow clients.

6. Learn from real architectures without copying the diagram

A company architecture article describes a particular time, workload, and set of constraints. It is not evidence that the company still uses the exact same design.

Use this reading worksheet:

Ask	What to record
What was the problem?	User impact, data size, traffic, or operational pain
What constraint shaped the design?	Latency, consistency, cost, team ownership, hardware
What changed?	The mechanism, not just the product name
What did it cost?	Complexity, failure modes, delayed data, migration effort
What evidence was reported?	Measured outcome and its conditions
What transfers to my problem?	A principle with a matching constraint
What does not transfer?	Scale, organization, or assumptions that differ

Primary reading 1: Dynamo

Amazon's 2007 Dynamo paper describes an available key-value store using techniques including consistent hashing, versioning, replica repair, and decentralized coordination. The useful lesson is how these mechanisms work together around availability and conflict handling. Dynamo in this paper is not interchangeable with every feature of today's DynamoDB service. Dynamo paper

Apply it: explain why our read-friendly catalog and exclusive reservation may need different policies. Do not copy an availability-oriented write policy into a workflow whose central rule it violates.

Primary reading 2: Spanner

Google's Spanner paper describes a distributed database that combines replication and time-related coordination to support externally consistent transactions. It illustrates that strong guarantees require specific mechanisms and costs, rather than simply labeling a database "consistent." Spanner research paper

Apply it: ask whether our library really needs cross-region synchronous coordination. A single-region write boundary with suitable recovery may fit better at the assumed scale.

Primary reading 3: service objectives

Google's SRE treatment of objectives connects measurable service behavior to user expectations. The practical reading goal is to define a useful success metric and a realistic target. Service level objectives

Apply it: decide whether search and reservation should share a target. Track user impact rather than reporting that servers were powered on.

Keep a small decision record

For each important choice, write its context, alternatives, decision, consequences, and revisit trigger. For example: "Use database search now; add a search index if relevance requirements or measured latency justify the extra system." This is an architecture decision record, often called an ADR.

7. Interview practice: questions 41–50

Q41. What would you ask before designing this system?

Who uses it, which actions matter, what is out of scope, how many users and requests to expect, and which outcomes must never happen. I would clarify whether a reservation targets a title or a specific copy.

Q42. Two requests see one available copy. How do you prevent two reservations?

Use an atomic database claim inside the transaction and a suitable uniqueness rule on active reservations. The losing request gets a clear conflict. A stale screen does not control the write.

Q43. Why distinguish Book and Copy?

Book describes the title. Copy is a physical item with its own status and location. Without that distinction, inventory and reservation rules become ambiguous.

Q44. The client times out. Should it submit a new reservation?

Retry with the same idempotency key or query the known operation. A timeout does not reveal whether the original transaction committed.

Q45. Why is the notification provider outside the main transaction?

Its delay should not prevent a valid reservation. The transaction records an outbox event, and a retrying worker delivers the notification later. Notification failure remains visible and recoverable.

Q46. When would you introduce a separate search index?

When search features, relevance, or measured performance justify it. I would define acceptable index lag and rebuild procedures, while keeping reservation checks on authoritative data.

Q47. What if a report worker loses its lease but keeps running?

A replacement may run concurrently. Use attempt ownership or fencing when committing the result so the stale worker cannot overwrite the accepted outcome.

Q48. How would you handle a slow live-update client?

Bound its buffer, monitor lag, and disconnect with a resumable cursor or require a state refresh. Do not let one client consume unlimited server memory.

Q49. What would make you choose microservices later?

Stable business boundaries plus concrete needs for independent ownership, release cadence, or scaling. I would include the team's ability to operate distributed failures in that decision.

Q50. How do you respond when the interviewer changes a requirement?

Identify which invariant, data boundary, or estimate changed. Revisit the affected decisions, explain the new trade-off, and keep the parts that still meet the requirements.

8. Scenario drills

Cover the suggested direction first and talk through each prompt aloud.

New condition	Suggested direction	What a strong answer adds
Search traffic grows 20×	Measure and scale the read path	Cache hit rate, database fallbacks, tail latency
Every student requests the same title	Identify a hot key and contention	Cheap reads plus an authoritative claim
One region cannot reach another	Define per-operation partition behavior	Which writes pause and how recovery works
Notification provider is down for an hour	Durable backlog and bounded retries	Oldest-message age, recovery rate, duplicate risk
A new consumer needs six months of history	Review event retention and replay source	Schema evolution and replay side effects
A schema must change without downtime	Expand, migrate, then contract	Compatibility across old and new instances

9. A two-week practice path

This is a suggested study sequence, not a guarantee of interview readiness.

Days	Work	Explain without looking
1–2	Guide 1 through estimates and storage	Requirements, latency, throughput, transaction rules
3–4	Guide 1 consistency and caching	CAP, replica lag, stale reads, cache failure
5–6	Guide 2	One web request and API/transport choices
7–8	Guide 3 messaging	Duplicate delivery, outbox, saga, backlog
9	Guide 3 object patterns	Strategy vs State; Adapter vs Facade
10–11	Guide 4	Failure containment, identity, recovery
12	Reservation design from a blank page	Write path and concurrent claims
13	Reports and live updates	Leases, history, backpressure
14	Timed mock and review	Decisions, alternatives, and limitations

After each mock, record one unclear explanation, one unsupported assumption, and one failure mode you missed. Rework those before adding more topics.

10. Review your own answer

Dimension	Weak answer	Stronger answer
Scope	Starts drawing immediately	Confirms features and exclusions
Correctness	Says "the database handles it"	Names the transaction and invariant
Scale	Says "millions of users"	Estimates relevant peak operations
Components	Lists products	Explains each component's purpose
Failure	Only describes the happy path	Resolves partial success and retries
Security	Says "add authentication"	Checks resource ownership and permissions
Operations	Says "add monitoring"	Names signals, recovery actions, and owners
Communication	Explains every detail equally	Selects the risks that matter most

Final revision card

Before you finish a design, answer these seven questions:

1. What can the user do?
2. What must never go wrong?
3. Where is authoritative data stored?
4. What happens on one read and one write?
5. What happens when a response is lost or a component fails?
6. What evidence would make us scale or change the design?
7. How will we know the system is serving users correctly?

If those answers are clear, your diagram has a purpose and your trade-offs can be evaluated.