A Good System Still Works on a Bad Day

Guide 4 of 5 · Reliability, security, and operations

Series overview · Previous: Architecture, messaging, and patterns · Next: Interview workbook and case studies

1. What if the library desk closes?

A second desk can keep the library open. But if both desks depend on one locked cabinet, adding desks does not solve the real problem.

Software needs the same end-to-end thinking. A duplicated application still depends on its database, network, credentials, configuration, and operators.

Term	Plain explanation
Availability	Can users get the required service?
Reliability	Does the system provide correct service over the relevant period?
Durability	Does accepted data survive the failures we plan for?
Resilience	Can the system absorb trouble and recover?
Fault tolerance	Can it continue despite specified failures?

Do not promise tolerance of every possible failure. State the fault model: one instance, one disk, one availability zone, or an entire region.

2. Define the promise and measure it

An SLI is a measurement, such as the fraction of valid reservation requests that succeed. An SLO is the target for that measurement. An SLA is an agreement that can specify consequences for missing agreed service levels. Teams can have internal SLOs without an external SLA. Google SRE: service level objectives

For a teaching example, an SLO might require 99.9% successful reservation requests over 30 days. Define which requests count, what success means, and which measurement point is used.

A time-based availability calculation is:

availability = uptime / (uptime + downtime)

Target	Allowed unavailability over 30 days
99%	7 hours 12 minutes
99.9%	43 minutes 12 seconds
99.99%	About 4 minutes 19 seconds
99.999%	About 26 seconds

These are time-based examples. A request-based SLO has an error budget in requests instead. For 1,000,000 eligible requests and a 99.9% success target, the budget is 1,000 failed requests.

A higher target usually requires more redundancy, testing, operational care, and cost. Choose it from user impact rather than automatically promising five nines.

3. Redundancy and failover

Active-passive means a standby is prepared to take over. Active-active means multiple instances serve traffic. Either approach needs health detection, capacity planning, and safe data ownership.

flowchart TD
    U[Students] --> E[Redundant traffic entry]
    E --> A[App in zone A]
    E --> B[App in zone B]
    A --> P[(Database leader)]
    B --> P
    P --> R[(Standby in another zone)]
    F[Failover controller] -.-> P
    F -.-> R

The replication mode, recovery time, and data-loss guarantees must be specified separately. Dotted lines represent control. They do not mean that adding a controller makes failover safe automatically.

A split brain occurs when two nodes act as authoritative writers without the intended coordination. Leader election and fencing help prevent an old leader from continuing to write after replacement.

If two independent required components each have availability 0.999, their series availability is 0.999 × 0.999 = 0.998001, or 99.8001%.

If either of two independent components can serve the entire workload and failover is perfect, their parallel availability is 1 - (0.001 × 0.001) = 99.9999%.

Those assumptions are strong. Shared power, a shared database, a bad deployment, or insufficient remaining capacity can make the real result much worse.

4. Timeouts, retries, circuit breakers, and bulkheads

A slow dependency can consume every worker while requests wait. Set timeouts and propagate a deadline: the total time the request is allowed to use.

Retry only when the failure may be temporary and repeating the operation is safe. Limit attempts, add backoff and jitter, and keep retries inside the deadline. Three attempts at each of three nested service layers can amplify one request into 27 downstream attempts.

A circuit breaker watches failures and temporarily stops calls to an unhealthy dependency. A bulkhead limits the resources one dependency can occupy, much like separate compartments contain a leak.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open: Failure threshold reached
    Open --> HalfOpen: Cooldown ends
    HalfOpen --> Closed: Limited probes succeed
    HalfOpen --> Open: Probe fails

Closed means calls are allowed. Open means calls are rejected quickly. Half-open allows limited probes. The breaker needs a defined sample window and thresholds; one ordinary business rejection should not necessarily count as dependency failure.

If recommendations fail, show the catalog without recommendations. If the reservation database fails, do not show a false confirmation. A fallback must preserve the business meaning of the result.

5. Rate limiting and overload protection

A library may let each student request a reasonable number of reports so one person does not occupy every printer. A rate limiter controls how often an operation may start.

Algorithm	Simple idea	Trade-off
Token bucket	Tokens refill; each request spends one	Allows controlled bursts
Leaky bucket	Work leaves a bounded queue at a controlled rate	Smooth output can add waiting
Fixed window	Count requests within fixed periods	Bursts can cross a window boundary
Sliding log	Track timestamps in the recent interval	Precise but uses more state
Sliding-window counter	Combine current and prior counts approximately	Lower cost with approximation

For example, a bucket with capacity 20 and refill rate 5 tokens/s permits an initial burst of 20 requests, then replenishes at five per second. It is not a strict "five requests in every possible one-second interval" limit.

If each of four app instances independently permits 100 requests/minute, a user reaching all four may get 400. Use shared atomic accounting or a deliberately partitioned allowance when a global limit matters.

A naive read-increment-write counter races under concurrency. The count update, time handling, and decision should be atomic for the chosen algorithm. Decide whether a limiter failure should reject traffic or allow a limited fallback; the right choice depends on the endpoint.

Rate limiting, concurrency limits, and queue bounds are different controls. Use 429 for quota or rate rejection where appropriate and provide retry guidance. Resource overload may call for 503. Edge protections can complement these controls, but no single limiter covers every failure mode.

6. Service discovery and service mesh

Services move as instances start and stop. Service discovery finds their current addresses.

With client-side discovery, the client chooses from a registry. With server-side discovery, an intermediary routes to a suitable instance. Registrations need expiry or health checks so dead instances do not remain forever.

A service mesh can provide shared traffic policy, service identity, and telemetry for service-to-service calls. It adds infrastructure and operational complexity. It does not implement reservation invariants or remove the need for application-level error handling.

7. Authentication is not authorization

Authentication asks, "Who are you?" Authorization asks, "May you do this?" A student card proves identity, but does not grant permission to edit another student's account.

Derive the caller's identity from validated authentication. Treat identifiers in the request as data to authorize, not proof of identity. A user ID is not automatically unsafe because it appears in a body; the danger is trusting it without checking ownership and permissions.

Use least privilege: each component gets only the access it needs. Keep secret values out of source code, public responses, and logs. Validate input and use parameterized database queries. Encrypt sensitive storage and control access to backups as carefully as live data.

8. OAuth, OIDC, SSO, and tokens

Term	School analogy	Actual role
OAuth 2.0	A limited permission slip	Delegated access to resources
OpenID Connect, or OIDC	A trusted identity statement	Authentication layer over OAuth
SSO	One school login works across approved apps	Sign-in experience across applications
SAML	Another agreed format for trusted identity statements	Federation using XML-based assertions

OIDC adds an ID token describing the authentication event. An access token is for resource access. An ID token is not a substitute for an API access token. OAuth access tokens may be opaque or structured; they are not all JWTs. OpenID Connect Core

For modern interactive login, use established libraries implementing authorization code flow with PKCE and the relevant validation. PKCE binds code redemption to the initiating client. Public clients cannot safely keep a client secret. Exact redirect checks and appropriate transaction binding matter; avoid obsolete implicit or password-based flows. OAuth security best current practice

sequenceDiagram
    participant B as Browser
    participant A as Library app
    participant I as Identity provider
    participant R as Library API
    B->>A: Start sign-in
    A-->>B: Redirect with transaction binding and PKCE challenge
    B->>I: Authenticate
    I-->>B: Redirect with authorization code
    B->>A: Deliver callback code
    A->>I: Exchange code with PKCE verifier
    I-->>A: Tokens
    A->>A: Validate identity result and establish session
    A->>R: Call with appropriate access token
    R->>R: Validate token and authorize operation
    R-->>A: Allowed result

This is a simplified server-side app flow. The browser receives a protected session cookie from the app rather than needing to hold the API token. Public-client flows have a different token-handling boundary. Real implementations must also validate issuer, audience, signature where applicable, expiry, and flow-specific state or nonce requirements.

A signed JWT is not necessarily encrypted. Anyone who has an ordinary signed JWT may be able to read its claims. Minimize what it contains and protect it in transit and storage.

SSO reduces repeated sign-in, but each application still authorizes actions. The identity provider becomes an important dependency. Existing application sessions and token validation strategies affect what happens during an identity-provider outage.

9. TLS and mTLS

TLS protects communication confidentiality and integrity and authenticates endpoints according to the negotiated setup. In common HTTPS use, the client validates the server. Mutual TLS also authenticates the client using certificates.

Think of TLS as a protected delivery channel. It does not prove that the receiver should be allowed to reserve a specific book. Application authorization remains necessary.

SSL is the older protocol family and should not be used as a modern protocol choice. The phrase "SSL certificate" remains common, but modern secure connections use TLS. Plan certificate renewal and trust management; expired certificates can cause outages.

10. Recovery and storage

Replication copies current changes. A backup preserves a recoverable state. If an operator deletes the wrong table, replication may rapidly copy the deletion. Recovery needs usable history and a tested restore process.

RTO is the target time to restore service. RPO is the target maximum loss window measured back from the incident. For example, RTO of 30 minutes and RPO of five minutes means aim to restore within 30 minutes with a recovery point no more than five minutes before the failure.

Approach	Useful property	Cost or limitation
Backup and restore	Lower steady running cost	Restore and provisioning take time
Pilot light	Essential recovery pieces already exist	Capacity must still be expanded
Warm standby	Reduced-capacity second environment	More cost, but faster activation
Active multi-site	Multiple sites serve work	Highest coordination and operating complexity

Test the whole recovery: data, application versions, secrets, network routes, and identity. A backup job reporting success is not proof of a successful restore.

Storage words you may hear

Term	Meaning
Block storage	Volumes expose addressable blocks, often used by filesystems and databases
File storage	Shared files and directories
Object storage	Objects addressed by keys, with content and metadata
NAS	Network-attached file storage
HDFS	A distributed filesystem designed for large data and streaming access
Volume	A logical storage unit that can be backed by one or several devices

RAID combines disks. RAID 0 stripes with no redundancy. A common two-disk RAID 1 mirrors data. RAID 5 tolerates one drive failure; RAID 6 tolerates two; RAID 10 combines mirrored pairs and striping, with survival depending on which drives fail. RAID protects against particular disk failures, not accidental deletion or site loss. It is not a backup.

11. Deploy, observe, and control cost

A VM has a guest operating system on virtualized hardware. A container packages an application while typically sharing the host kernel. Containers can improve deployment consistency, but do not create perfect isolation or unlimited portability. Runtime and architecture compatibility still matter.

A rolling deployment replaces instances gradually. A canary sends a small share of traffic to a new version before wider rollout. Blue-green keeps two environments and switches traffic. Every approach needs a rollback plan and backward-compatible data changes.

For a schema change, an expand-and-contract sequence can help: add the new field, deploy code that tolerates both versions, migrate data, and remove old use only after compatibility is no longer needed.

Observe	What it tells us	Library example
Metrics	How much and how often	Failure rate, p95 latency, queue age
Logs	Details about a particular event	Reservation rejected with a reason code
Traces	Where time went across components	API waits mainly on the database
Business checks	Whether outcomes remain correct	No copy has two active reservations

Propagate a correlation ID. Avoid logging tokens or sensitive student details. Alert on user impact and fast consumption of error budgets, not every isolated internal error.

Cost has several parts: compute, storage, network transfer, managed services, and engineering time. Measure useful units such as cost per completed report. More components may lower a hardware bill while increasing the team's operating burden.

12. Interview practice: questions 31–40

Q31. Why are two app instances not enough for high availability?

They may share a single database, traffic entry point, zone, or bad configuration. I would trace the full request path and test the failure of each required dependency.

Q32. Explain SLI, SLO, and SLA.

SLI is the measured behavior, SLO is its target, and SLA is the service agreement. I would define the measurement window and eligible requests before quoting a percentage.

Q33. Why can retries make an outage worse?

They add work to an overloaded dependency. I would limit attempts, use backoff with jitter, avoid retries at every layer, and ensure repeated effects are safe.

Q34. What does a circuit breaker do?

It temporarily stops calls after a defined failure threshold and later allows probes. It helps contain failure; it neither repairs the dependency nor replaces a timeout.

Q35. How would you enforce a rate limit across app instances?

I would use atomic shared accounting or allocate bounded local allowances with known approximation. I would explain behavior when the accounting store fails.

Q36. RTO versus RPO?

RTO concerns recovery duration. RPO concerns how far back the recoverable data may be. They drive different infrastructure and backup decisions.

Q37. Why is replication not a backup?

It can copy unwanted changes. A backup preserves a recoverable version and must be tested through restoration.

Q38. OAuth or OIDC for login?

OIDC provides the identity layer for login on top of OAuth. OAuth alone addresses delegated access, so I would not infer identity from an arbitrary access token.

Q39. Does mTLS authorize a student to see another student's reservation?

No. It authenticates endpoints at the transport layer. The application still checks identity, ownership, and permissions for the requested action.

Q40. What would you monitor first?

User-visible success and latency, followed by dependency saturation, queue age, and business invariants. Metrics locate the symptom; traces and focused logs help explain it.

Quick revision

State the failures you plan to tolerate. Set measurable targets. Bound waiting and retries. Make overload explicit. Separate identity from permissions. Test restores and failover. Measure the result the user cares about.