So, what do we mean when we say data mesh? There are four key principles:
- Domain Centric Ownership
- Federated Governance
- Data as a product
- Self-service platform
Whether you feel you need one more than the others or would like to apply all four, let’s explore each principle in a bit more depth.
Domain Centric Ownership
This principle encourages aligning the data architecture with the organisational structure by advocating for domain teams to fully own their data. Those with the closest understanding of the data's context should manage it. By only worrying about the needs of your specific context, it's simpler than trying to solve the rest of the organisation's needs as well.
The challenge here is to enable agility whilst avoiding the creation of disparate data silos that obscure the broader perspective. To prevent that, you’ll need to implement some governance.
Federated Governance
Federated governance is about striking the right balance between structure and flexibility. It sets up a framework of core principles and policies that guide the entire organisation, while also empowering individual teams to tailor their own detailed practices. Think of it as a shared vision where the broad strokes are defined, leaving room for each team to fill in the details with their unique colours and textures.
It's important to have a model that has a thin slice of central data governance, focused almost exclusively on ensuring that the mesh itself can function successfully, combined with distributed governance within the domains. The platform that underpins the mesh should enforce the principles and policies defined by this governance model, across the organisation.
This method ensures that your organisation operates as a united entity, without the constraints of a centralised, top-down governance model that can often dampen innovation and slow down decision-making.
Data as a product
Adopting a product-centric view of data in a data mesh environment means we handle it with the same care and attention that (for example) a retailer would with a real-world product – by having a clear description, exact specifications, origins, measurements, etc, the customer (data user) will know exactly what the product (data) can be used for and what to expect from it. Data becomes a living asset, continually enhanced to remain valuable and relevant, retired when it is superseded or obsolete.
This approach transforms data into a vital business tool, essential for informed decision-making and driving innovation. Those responsible for the data become its champions, focused on its integrity and alignment with the organisation's strategic goals. Short and sweet, it's about making data work harder and smarter for everyone involved.
Self-service platform
Despite the ambiguous name, the self-serve data infrastructure principle is perhaps the most transformative, as it enables the other principles to be implemented seamlessly. It is not prescriptive about the technologies employed; instead, it advocates for a platform that equips domain teams with the necessary tools and systems to independently develop, execute, and maintain data products. This domain-agnostic platform is the cornerstone upon which the data mesh is built, ensuring that the infrastructure supports the data products and services required by the business.
From ideation to implementation
While these principles provide a structural framework, the true essence of data mesh is found in the cultural shift it necessitates. It's a reimagining of the role of data within the organisation, fostering a mindset where data is treated with the same strategic consideration as other core business functions. This cultural evolution doesn’t happen overnight; it requires dedication to change management, persistent advocacy, and a willingness to redefine the norms of data stewardship.
The full benefits of data mesh might not be revealed if you’re working in smaller teams or individual departments because it thrives on scale. Without a substantial uptake, you may find the available data is limited, which constrains the mesh’s effectiveness. It’s crucial, however, to start with a manageable approach that still offers tangible benefits.
What does a good data mesh look like?
We’ve seen some organisations excel in defining what quality data products look like, crafting guidelines that outline what these products should achieve and how they should be governed. The challenge lies in ensuring these guidelines don’t appear too daunting at first glance. It’s about emphasising the essential steps and making it clear that additional quality elements can enhance a data product, without making them prerequisites from the start.
Remember we want to encourage participation in the mesh, without recreating the age-old problem of ungoverned silos. It should be straightforward to setup and register data products in the first instance but putting increased effort into managing the quality of data products should reap benefits. A crucial factor in the success of data mesh is ensuring there is appropriate incentive for data product owners to delight their customers across the organisation.
The goal is to foster adoption by minimising barriers to entry and simplifying the initial steps. It’s okay for early data products to be less than perfect; what’s important is establishing the process and understanding the data mesh framework. Over time, as more teams engage and contribute, the quality and value of the data products are expected to improve, driven by the evolving needs and feedback within the organisation. It’s a progressive journey, not a pursuit of immediate perfection.
Do you need data mesh?
The main argument for data mesh is for scalability across complex organisations. With businesses generating so much data, across so many domains, it’s becoming more difficult to centralise all aspects of that data’s management and analysis: central IT/data teams can become a bottleneck. It is unlikely to be the right approach for every organisation. Factors such as risk appetite and organisational culture will determine whether data mesh is the right approach. Further, hybrid approaches, which adopt the principles of mesh but not across all business or data domains, are already being adopted successfully by some organisations.
Data mesh empowers people across the organisation to access and get value from data. When implemented well and underpinned by an effective data catalogue, it can help reduce the time that analysts and data scientists spend searching for, wrangling and interpreting data.
The future of data mesh
We believe that the data mesh model holds great promise for certain organisations, though not universally for all. It's particularly potent when it meshes—no pun intended—with the principles of data fabric. These two concepts are distinct yet perfectly compatible, each enhancing the other's strengths.
Balancing mesh and fabric
While data mesh may be amongst many current buzzwords, it's crucial to consider it within the broader context of data management strategies. Data fabric, with its promise of a more interconnected and seamless data integration, holds considerable value for the future. Organisations should evaluate their readiness and strategic objectives to discern whether data mesh, data fabric, or a hybrid approach best suits their journey towards a data-centric future.
Data mesh’s big challenge
Data mesh is still in its infancy as an approach, and while a lot of organisations want to introduce aspects of it, many aren’t ready to introduce all four pillars. Moving away from centralised data management could lead to chaos if the organisational culture, governance and operating models are not right.
The secret sauce for a successful data mesh approach lies both in the cultural and organisational implementation but also in embracing advanced technologies and leveraging the metadata created by the participants in the mesh. Without this tech backbone, moving from a data mesh to a data fabric will be overwhelming and complicated. Those who tap into active metadata and let it guide the evolution of their data mesh will likely outpace competitors who don't.
So, here's a nugget of advice: if you're considering data mesh, don't overlook the technology and principles underpinning data fabric. Investing in these will not just serve you now, but also set you up for long-term success.
Let's talk Data Mesh
Need help defining your data mesh strategy? Already started but need support delivering it? Amplifi is actively engaged in multiple Data Mesh engagements with our customers and happy to help support you on your journey. Want to know why to choose Amplifi as your Data Mesh consultancy? Read more on our Data Mesh services page here or get in touch.
