The shop till and the head-office reports
Pooja is the first data engineer at FreshKart, a grocery chain with 300 shops and a delivery app. (Pooja, FreshKart and every number here are made up for teaching.)
Every shop has a till. The till must be quick and always correct for one sale at a time: scan, total, pay. Head office asks very different questions: which product sold most in each region last quarter? Answering that on the till would slow every customer in the queue. So the shop copies its sales into a separate reporting system built for big questions.
That split, OLTP (online transaction processing) for the till and OLAP (online analytical processing) for head office, plus the pipes between them, is what a data platform is. Pooja's first year is a series of complaints, and each one builds a piece of the platform:
| Complaint |
Idea it forces |
| "The quarterly report froze the tills" |
OLTP vs OLAP |
| "The CEO wants live Diwali sales, not yesterday's" |
Batch vs streaming; lambda and kappa |
| "Orders from the basement shop land in the wrong minute" |
Event time, windows, watermarks |
| "Search still shows products we deleted" |
Change data capture (CDC) |
| "Where do we put all this? It's becoming a swamp" |
Warehouse, lake, lakehouse |
| "I re-ran last night's job and revenue doubled" |
Idempotent, replayable pipelines |
This guide builds on Kafka and messaging and Inside a database. Here is where Pooja ends up. Each part is explained in the section that adds it.
flowchart LR
T@{ icon: "algomi:ext-device", label: "Shop tills and app", pos: "b", h: 52 }
DB@{ icon: "algomi:data-cylinder", label: "Orders database: OLTP", pos: "b", h: 52 }
T --> DB
CDC@{ icon: "algomi:svc-cube", label: "CDC connector", pos: "b", h: 52 }
DB -->|transaction log| CDC
K@{ icon: "algomi:async-pipe", label: "Kafka: one topic per table", pos: "b", h: 52 }
CDC --> K
SJ@{ icon: "algomi:async-pipe", label: "Stream job: clean, dedupe, window", pos: "b", h: 52 }
K --> SJ
LIVE@{ icon: "algomi:svc-cube", label: "Live Diwali dashboard", pos: "b", h: 52 }
SJ --> LIVE
B@{ icon: "algomi:data-cylinder", label: "Bronze: raw files", pos: "b", h: 52 }
K --> B
SV@{ icon: "algomi:data-cylinder", label: "Silver: cleaned tables", pos: "b", h: 52 }
B --> SV
G@{ icon: "algomi:data-cylinder", label: "Gold: business aggregates", pos: "b", h: 52 }
SV --> G
R@{ icon: "algomi:ext-doc", label: "Head-office reports and ML", pos: "b", h: 52 }
G --> R
You've reached the free preview of this post.
Subscribe to the Blogs plan (₹99/mo) — or use any Algomi Suite, Career, Investor, or Trader plan — to keep reading.
See plans
