Designing Photo Storage: From Small Files to a Large Photo Service

400 wedding photos on a train

Sunita is on the overnight train home from her cousin's wedding. She uploads 400 photos to the family album on YaadeinPics as the Wi-Fi comes and goes. Some uploads retry three times. Next week she'll remove a relative from the album, and in five years one photo will be silently corrupted on a disk somewhere. (Sunita, YaadeinPics, Arvind and every number here are made up for teaching.)

Arvind designs YaadeinPics' storage. He starts with an old school problem. Imagine a school keeping a million photographs, each in its own envelope. Finding the right envelope could take more effort than looking at the picture. A catalogue that says "box 18, position 42" makes retrieval easy.

A photo service has two related jobs:

1. Store the image bytes.
2. Keep enough information to find and protect them.

Uploading a photo also creates thumbnails, records ownership, and eventually makes the photo visible to the people allowed to see it.

In 30 seconds, as an architect would frame it: "I keep metadata (owner, album, visibility, workflow state) in a relational store and bytes in a blob store, joined by stable IDs. Clients upload straight to blob storage with a scoped, signed link. The service verifies the object, and a worker builds deterministic variants from an outbox work item, so retries never duplicate. Reads go through a CDN with versioned URLs, and private photos get short-lived signed links after an access check. Upload, processing and deletion are tracked workflows. Packing small files into volumes, Haystack-style, is an optimisation I'd add only when measurements justify it."

Historical context. The topic draws on Facebook's published Haystack design. Haystack grouped many photos into larger storage files and kept a compact lookup index, to cut file-metadata work. Its architecture included directory, cache and store components. It is a historical design described in 2010, not a claim about Facebook's current infrastructure (original Haystack paper). The design below is an original interview design for the broader storage problem, not a reconstruction of Haystack.

1. The picture and its description live apart

A blob is a chunk of binary data treated as one object.

• Product metadata answers "May Sunita's aunt view this photo?"
• Storage lookup answers "Where are its bytes?"

Mixing the two makes both privacy changes and storage moves harder. To start, use an existing object store plus a relational metadata database. A custom storage engine is justified only when measured workload and cost show existing storage isn't enough.