Data Mesh Architecture

Why Centralized Data Teams Break Down

Every growing company hits the same wall with data. The business has fifteen teams that need analytics, ML features, and reporting. There is one data team of eight engineers. Requests queue up. Lead times stretch to six weeks. The data team becomes a bottleneck not because they are slow, but because they are structurally outnumbered.

The standard response is to hire more data engineers. This works for a while. But centralized data teams hit a knowledge ceiling around 15-20 people. Beyond that, no single team can hold enough domain context to build accurate data products for every part of the business. The payments data engineer does not understand the nuances of the logistics domain. The marketing analytics person cannot model the subscription billing edge cases. Quality degrades, and domain teams lose trust.

This is the actual problem data mesh was designed to solve. Not "we need a cooler architecture." Not "microservices but for data." The problem is that centralized data teams cannot scale their domain knowledge proportionally to the number of domains they serve.

The Four Principles in Practice

Zhamak Dehghani introduced data mesh in 2019 around four principles. They are well-documented elsewhere. What matters more is what each one looks like when it is working versus when it is not.

Domain ownership means the orders team does not file a ticket asking the data team to build an orders dashboard. The orders team owns the definition, quality, and publishing of order data. In practice, this looks like a data engineer embedded in the orders team (or an application engineer with data skills) who maintains the team's data outputs with the same rigor as their APIs. When it is not working, the orders team "owns" the data in name only, and the centralized data team still does the actual pipeline work.

Data as a product means domain data has documentation, an SLA on freshness (e.g., "order events available in the warehouse within 15 minutes of creation"), semantic versioning for schema changes, and monitoring for quality. Think of it as treating downstream data consumers with the same care you give API consumers. When it is not working, teams publish raw database replicas with no documentation and call them "data products."

Self-serve data infrastructure means domain teams can create a new data pipeline, register a schema, and publish to the data catalog without filing a ticket or waiting for platform team support. The platform team provides templates, CLI tools, and guardrails. When it is not working, publishing a data product requires 40 manual steps and three Jira tickets, which means nobody does it voluntarily.

Federated computational governance means standards for naming, schema evolution, access control, and data quality are defined globally but enforced through automation. Schema validation runs in CI. Quality checks execute on every pipeline run. Access policies are declared as code. When it is not working, each domain invents its own conventions, and six months later consumers face a fragmented landscape harder to navigate than the centralized warehouse it replaced.

When Data Mesh Makes Sense (and When It Does Not)

Data mesh works at scale, specifically organizations with more than 200 engineers, multiple distinct business domains, and a centralized data team that has become a delivery bottleneck.

If your data team can fulfill requests in under two weeks, you do not need data mesh. The organizational overhead of distributing data ownership is significant. Smaller companies are better served by a centralized team with good prioritization and clear SLAs.

If your engineering teams have no data engineering skills and no appetite to develop them, data mesh will fail regardless of how good your platform is. Domain ownership requires domain teams to actually do the work. If they treat data responsibilities as an unfunded mandate, quality will be worse than what the centralized team delivered.

Where Data Mesh Goes Wrong

The unfunded mandate failure. Leadership announces that domain teams now own their data. No additional headcount is approved. No training is provided. Domain teams, already stretched on feature work, treat data as a side project. Quality degrades. Consumers lose trust. Within a year, a shadow centralized team re-emerges because someone has to fix the mess.

The premature platform failure. A company starts distributing data ownership before the self-serve platform is mature. Each domain team builds its own pipeline infrastructure. You end up with four different orchestration tools, three schema formats, and no discoverability. This is more fragmented than the centralized approach, not less.

The governance gap failure. Governance gets punted to "phase two." By the time phase two arrives, twelve domains have published data products with incompatible naming conventions, different freshness guarantees, and no lineage tracking. Reconciling these after the fact is more expensive than building governance into the platform from day one.

The Zalando story, in full context. Zalando is frequently cited as a data mesh success, reporting a 40% reduction in time-to-insight for domain teams. What is less often mentioned: the migration took over 18 months, required significant investment in their self-serve platform (Datamesh Manager), and worked partly because Zalando already had strong engineering culture around domain ownership from their microservices architecture. They did not start from zero. Companies that cite Zalando's results without having Zalando's starting conditions consistently underestimate the effort.

Building the Self-Serve Platform

The platform is what makes data mesh practical. Without it, you are asking every domain team to become infrastructure experts.

A mature platform typically includes: a data catalog for discovery (Datahub, Amundsen, or OpenMetadata), a transformation layer (dbt), an orchestration engine (Airflow, Dagster, or Prefect), a schema registry (Confluent Schema Registry or AWS Glue), domain-organized storage (S3 or GCS with clear bucket conventions), and quality monitoring (Great Expectations, Soda, or Monte Carlo).

The critical design principle: publishing a new data product should take hours, not weeks. If the activation energy is too high, adoption will not happen voluntarily. Build golden path templates that let a domain team go from "we want to publish this dataset" to "it is in the catalog with documentation and quality checks" in a single afternoon. Intuit's data mesh platform achieves this through scaffolding CLI tools that generate pipeline code, register schemas, and configure monitoring from a single command.