Case study
Data lake house build
The situation
The client ran several operational systems, each owned by a different team, each with its own idea of what a customer record or a transaction looked like. Reporting was built the way reporting usually gets built under those conditions: whoever needed a number wrote a one-off extract against whichever source system seemed closest to the answer. Every new question meant a new pipeline, built against a schema that could change without notice, by someone who had to guess how another team's system worked.
That pattern is not a data problem so much as an organisational one. Two reports built against the same underlying reality can still disagree, because each was assembled by a different person, on a different day, making different reasonable-sounding judgement calls about which rows to include.
The approach
We built a lakehouse with a clear landing-to-curated structure: raw data lands exactly as the source produced it, then moves through a conformance layer that resolves the things source systems never agree on, date formats, currency, identifiers for the same entity across systems, before anything is allowed to reach a report.
Apache Airflow orchestrates the pipeline as a dependency graph rather than a set of jobs on independent clocks, so a downstream transform only runs once the data it depends on has actually landed and passed its checks, not just once its scheduled time has arrived. That distinction matters more than it sounds: a fixed-clock pipeline will happily read a half-written upstream table and produce a number that looks plausible and is wrong.
For sources where staleness itself was the problem, transaction feeds that needed to inform same-day decisions, we used Kafka to stream change events rather than waiting for the next batch window. Slower-moving reference data stayed on a batch cadence, because streaming everything would have added operational complexity the data did not need. PySpark handles the transform work at the volume involved: deduplication, slowly changing dimension logic, and the conformance rules that turn five systems' worth of assumptions into one.
What changed
Reports now read from one conformed layer instead of being assembled by stitching together fresh extracts from whichever source seemed relevant. A new reporting question gets answered against the existing curated model rather than by writing a new point-to-point pipeline into a specific operational system, which is both faster to build and structurally unable to disagree with an existing report the way two independently built extracts can. Power BI sits on top of the curated layer, so the definitions it displays are the ones the pipeline already enforced, not a fresh interpretation invented at the dashboard.