Building an AI-Ready Data Lakehouse: A Reference Architecture

If you’ve worked through the five signs that production data isn’t ready for AI and concluded that you do need a foundation, the next question is the practical one: what does it actually look like, and how much of it do you have to build?

The honest starting point is that a lakehouse is a pattern, not a product. It means keeping your data in cheap, open storage — the way a data lake does — while adding the transactional guarantees, enforced schemas and query performance you’d expect from a warehouse. Several platforms implement it well. Below we walk through one concrete architecture on Azure — not a whiteboard sketch, but a system we built and put into production — then map it onto the systems a manufacturer actually runs. Further down we set out where Microsoft Fabric fits the same pattern.

The layering: raw, reconciled, decision-ready

Every credible lakehouse organises data in tiers — conventionally bronze, silver and gold, though what matters is the discipline rather than the names.

Bronze

Raw, exactly as it arrivedLanded, never edited

ERP extracts, MES event files, quality records and historian tags, written down untouched — original columns, original timestamps, original mistakes.

You can only reprocess what you kept.

→

Silver

Reconciled and conformedOne part, one identity

Deduplicated, validated, and joined on a shared key — the layer where the ERP part number, the MES machine code and the supplier SKU finally become one part.

This is the layer most estates are missing.

→

Gold

Decision-readyDefined once, read by everything

Business tables and metrics — scrap rate, on-time delivery, cost per unit — defined in one place and consumed by dashboards, agents and downstream systems alike.

Every consumer reads the same definition.

The payoff is reprocessing. When Finance revises what counts as scrap, you re-run silver and gold from bronze and get a corrected history — rather than re-negotiating with four source systems that have long since moved on. That single property is what makes an AI answer auditable later.

The architecture, as delivered

The diagram below is generalized from a production deployment: batch and streaming ingestion side by side, a single governed landing zone, and parallel processing tracks feeding dashboards the business used daily. We don’t have the client’s permission to name them, so it is anonymized — sources are described by type rather than by product, and none of the figures behind them appear here. The shape, the component choices and the reasoning that follows are what was actually built.

One thing worth noting before the components: it ran both paths in production — scheduled batch alongside real-time events — which is the part teams most often assume has to be phased.

The column worth pausing on is storage. One copy of the data, in one open format, feeds every engine to its right — batch transformation, SQL querying, model training and the streaming path, side by side, none of them keeping its own copy. That is the argument for the pattern over a separate lake and warehouse: two systems holding two copies eventually hold two different numbers, and reconciling them becomes somebody’s permanent job.

The decisions behind the boxes

Any vendor can draw those boxes. The value is in why each one is there, and what goes wrong when it isn’t.

• Land raw before you transform anything. Bronze looks like wasted storage until the first time a supplier silently adds a column or renames one — the case schema evolution exists to absorb — or someone finds a bug in a transformation that has been running for eight months. You can reprocess what you kept; you cannot re-derive what you discarded, and storage is the cheapest part of the platform.
• Delta rather than plain Parquet files. This is the choice that makes reproducibility possible. Delta adds ACID transactions, schema enforcement and versioned history — time travel, in its own vocabulary — on top of the same Parquet files. That is what lets you answer “what did this table say in March?”, correct a record with a single merge rather than rebuilding a partition by hand, and stop a failed job from leaving half-written data for a dashboard to read.
• Storage separate from compute, because it’s a cost decision. Data sits in ADLS Gen2 and stays there; compute spins up for a job and shuts down after. A traditional warehouse charges for capacity whether or not anyone is querying — which, for two heavy reporting windows a month, is most of the bill.
• The streaming path is optional, and usually premature. Event Hubs is in the diagram because some use cases genuinely need it — but pulling the historian in batches every fifteen minutes answers nearly every question a mid-market manufacturer actually has. Build streaming when a use case demands it, not because the reference architecture has a box for it.
• The historian is the hard integration — plan for it. Machine telemetry is tag-based, high-volume, and keyed on time and asset rather than on a part or an order. Joining it to business data means deciding how a continuous signal maps to a discrete production run — a modelling decision with operations in the room, not a pipeline task. Teams routinely underestimate this one.
• Governance sits in the diagram, not after it. Retrofitting row- and column-level access to a lake that already has forty consumers is not a configuration change; it is a re-platform. Identity, catalog and lineage cost little with the first table and a great deal after the fortieth.

What this looks like in production

That deployment came from a different domain, and the useful thing about it is how little changes when you move it. The sources swap; almost nothing else does.

Same architecture, different inputs. The storage and governance layers are identical.

	In the deployment above	In a manufacturer
Sources	Operational system, analytics warehouse, clickstream events	ERP, MES, quality / LIMS, machine historian
Batch ingestion	Scheduled pipeline pulls from operational and warehouse systems	Unchanged — nightly ERP extract, shift-level MES export
Streaming	Event stream for tag-manager and clickstream events	Historian telemetry — and usually deferrable to phase two
Storage	Identical. One landing zone in open format — this is the layer that does not change.
Processing	SQL engine for reporting, Spark for transformation, ML and streaming	Same split, same reasons
Consumption	Power BI, plus CRM and marketing automation	Power BI, plus planning and CMMS — and an AI agent reading gold
The hard join	Continuous session activity → a discrete conversion	Continuous machine signal → a discrete production run

That last row carries the work. In both cases the difficulty isn’t the platform — it’s deciding how a continuous signal becomes a discrete business event you can count. Get that wrong and every number downstream is subtly off.

Databricks or Fabric?

We drew the architecture above on Databricks. The same pattern — open storage, Delta tables, medallion layering — runs just as legitimately on Microsoft Fabric, whose OneLake also stores data in Delta format. The choice is genuinely situational, so here are the criteria we actually use.

Both implement the same lakehouse pattern on Delta. The question is which fits your team, not which is better.

	Azure Databricks	Microsoft Fabric
Fits a team that	Has (or will hire) Spark, Python and data-engineering skills	Is built around Power BI, SQL and the Microsoft 365 estate
Power BI	Connects well via SQL endpoints	Native — Direct Lake reads Delta tables in OneLake without a separate import
ML & data science	Deep: notebooks, MLflow, mature model lifecycle tooling	Capable and improving; lighter than Databricks at the advanced end
Operating burden	More knobs — more control, and more to run	More managed — fewer decisions, less low-level control
Cost shape	Consumption-based per workload; scales to zero when idle	Reserved capacity (F-SKU) shared across workloads; predictable, sized up front
Portability	Runs across clouds; open Delta storage you control	Microsoft-centric by design; OneLake still open Delta underneath
Reach for it when	Heavy transformation, real ML ambitions, or a data team that wants control	Power BI is already your centre of gravity and IT is small

They also coexist: both sit on Delta in open storage, so plenty of estates run Databricks for engineering and ML while Fabric and Power BI serve the business layer. And you will have spotted Synapse in the architecture above — it was the right call for that build and it still runs. Worth saying plainly, though: Microsoft is now directing net-new investment to Fabric. Synapse remains supported and nothing is on fire if you run it, but for a greenfield build today we would weigh Fabric or Databricks SQL for that layer instead. That is a change in the landscape, not a verdict on the original decision.

What you actually need at your size

Reference architectures are drawn at full scale, which is exactly what makes them intimidating. Almost nobody should build the whole diagram at once. Here’s what we’d actually put in phase one at three different sizes.

The two things that decide whether it survives

Governance. Identity through Entra ID, a catalog recording what exists and where it came from, secrets in Key Vault rather than in notebooks. None of it is glamorous and all of it is cheap on day one. The value shows up the first time someone asks an agent a question it shouldn’t be able to answer — or an auditor asks where a figure came from.

Cost discipline. Runaway compute is the most common way these platforms lose their sponsor. The controls are unexciting: clusters that shut down when idle, someone reviewing spend monthly, and the habit of asking whether a job really needs to run hourly. Separating storage from compute gives you the ability to control cost; it doesn’t control it for you.

None of this requires a two-year program. The architecture is drawn at full scale because that is what a reference architecture is for — but the version that earns the next round of funding is the small one that answers a question someone is already asking.