The Good, Bad, and Ugly: Data Mesh
Imagine every team in your company treating data the way a chef treats their signature dish: they shop for the ingredients, cook the meal, and proudly serve it without waiting on a distant kitchen. Th
Disclaimer: As a software and data architect, I have successfully applied data mesh to a global Telco and video streaming companies, as well as a startup. Currently, I am applying it to another startup for an AI-proof data platform.
Meet Glow Telecom, in short, GlowTel. A fast-growing telecom that once kept all its data in a single, massive warehouse with dozens of hidden databases which are escaped from governance. The central “Data Kitchen” (data engineers with a central team) served the entire company for several use cases that they don’t even know what they mean like marketing wanted churn numbers in BSS, network ops wanted outage predictions in OSS, and Finance wanted revenue dashboards. Tickets piled up like orders on Saturday night. By the time a report arrived, the business had already moved on. The worst part is that there is no possibility to catch the data velocity! The data team and their environment are not built for Telco but as a commodity.
A brave staff engineer enters the scene with a bold plan: “What if every domain (marketing, network, finance) cooks their data dishes and publishes them on a shared menu?”
After many presentations and discussions, executives loved the agility; tech leads loved the modern architecture. So the mesh experiment has begun!
The early signals show that they are speedier from day one! The marketing team built a churn-risk dataset in two sprints and A/B-tested a retention offer the same month. No more ticket queue, forcing their requirements in quarter planning, begging for more reports…
It doesn’t end here! Everybody has experienced that local expertise gives a better flavor. Network engineers finally embedded signal-noise ratios and tower metadata. Operational analytics shine as never before!
What about scaling? Scaling is without strain! As GlowTel expanded into new regions, each region spun up its micro-domain without needing to request additional central capacity. The Data Kitchen stopped being the bottleneck but became like a shared flat: Everybody has their place in the fridge and uses the kitchen based on the rules.
I’d love for you to share fairy tale stories and provide numerous examples to show that Data Mesh is a silver bullet, an ultimate solution that everyone needs to follow. Unfortunately, it is not the case, and will never be.
The third quarter has started with messiness. The finance team called a customer “active” after one purchase; marketing required three. Dashboards began to contradict each other. There were endless discussions about what the “purchase” is. Is it a subscription? Is it postpaid (invoice)? Is it prepaid? What’s the difference between subscription and prepaid? Can we count both of them as recurring billing? If so, what are we going to call the two-year-committed post-paid contract’s payments? They decided to establish a Federated Governance Council, which would meet weekly.
The discussion between architects has not ended here! The IT inventory maintainer enterprise architect has begun to scream about tool chaos. One data product uses Apache Spark in Databricks, another one uses Snowflake, and another one is flirting with DuckDB without high availability. Should we have only one data platform or several? The platform team scrambled to lay down a “golden path” of approved tooling, but then project managers jumped into the scene: What will be the migration from one tool to another? Who will cover the cost?
The last brutal hit came from quality. As there are no shared tests, some data products have become outdated. Trust dipped until automated freshness checks were baked into every pipeline, but there is another problem. In which field will we measure freshness? The general governance tools verify the availability of files or data. However, acquiring data from over two hundred resources poses difficulties, and the data remains in the queue. They have been created but not ingested yet. When they are ingested, should we count them as fresh data?
Well, I would love you to tell here an incredible comeback, but not… The issues are not finished.
While everything was going great and everybody was excited, they missed the human part of the equation: The old warehouse team felt replaced, even though the people were still asking them for data because of the new system’s freshness issue. Architects thought that everybody should manage their data, but they forgot the teams like HR, not only as technological challenges but also redefining the roles with tied success metrics. Now, nobody is sure about their responsibility, how much they need to involve technical topics, and how the promotion cycle will be.
While our teams wanted to break the central team rules, everything has changed. Now, new silos have emerged. Domains were dealing with catching up with the latest changes, so they were not talking to each other as much as before. GlowTel became an enterprise that has many startups inside called domains.
After overcoming numerous technical challenges, the last voice heard by the CFO was: Tool licenses, re-architecture, and team upskilling… cost millions! CFO was delighted about churn reduction at the very beginning, but when the new year budget planning arrived… The face… puzzled like a thousand pieces.
I understand you, really! Many articles around the world popularized decentralized ownership of data. The main problem is that nobody tells you exactly what you need to do. There are articles, to-do lists, and books that tell you the theory, but who actually uses or applies it? What are the challenges? What are the best practices? What are the practical things?
I haven’t found, and experienced it the hard way. My first implementation was very challenging because Data Mesh is not only a technical challenge but also an organizational restructuring. When you begin to play with people’s comfort zones, it generally results in discomfort zones. People don’t want your LLM-generated to-do lists or theoretical articles.
There is good news! I have applied Data Mesh in companies of various sizes and decided to share my practical knowledge. This is not a deep dive article, but there will be. If you’re interested in learning more about Data Mesh and building AI-proof platforms, I invite you to follow me.