The journey towards effective data management has been ongoing, with significant strides made in recent years. However, the current shift towards viewing data as a product is not just another phase in this journey – it’s a distinctive and progressive step forward. This approach elevates data from being a supportive element to a central strategic asset, demanding proactive cultivation, sophisticated management, and considered utilisation. Adapting to this evolved perspective is crucial for harnessing the full potential of an organisation’s data in a dynamic, globally competitive landscape. It’s about building on the lessons learned and moving towards a more integrated and value-driven data strategy.
How do you make a data product economy work within an organisation?
To effectively implement a data product economy within an organisation, similar to how products are managed on platforms like Amazon, it is crucial to ensure that data, treated as a product, possesses essential attributes, most notably comprehensive metadata.
This is vital because, in a well-functioning data mesh, some data products are utilised to create additional products, leading to a layered structure of products built upon one another. However, this interdependency also means that a data quality issue in one product can have a cascading effect, potentially compounding issues across multiple products. Therefore, monitoring and managing these interconnections is imperative for an organisation.
While data products need to maintain high standards, it's equally important to lower the barriers to entry into the Mesh. This approach encourages and incentivises individuals within the organisation to start creating and engaging with data products. Nonetheless, there should be a clear distinction and recognition of what constitutes a good versus a poor data product. Balancing the ease of access with the maintenance of quality standards is key to fostering a productive and efficient data product economy within an organisation.
Why do organisations need mesh?
Organisations need a data mesh to meet specific business requirements, from generating board reports to responding swiftly to market changes. It's not just about stating that different domains own their data; it's about how they can effectively deliver and access these data products. If a self-service platform isn't up to par, alternative methods need to be in place.
Crucially, in a data mesh, domains should have a system or process to manage costs associated with their data products. This economic aspect of the mesh allows it to function independently of individual projects. The challenge then lies in operationalising the mesh outside the usual organisational infrastructure. It requires setting clear governance, integration, and policies for data quality and access, ensuring that data management aligns with the organisation's broader goals.
How do you operationalise something that is outside of your organisation’s infrastructure?
The effective management of data as an asset entails a balance of economic incentives and governance principles, mirroring the "carrot or stick" approach. On the 'carrot' side, implementing a microtransaction model on self-service data platform aligns with the economic principle of ‘usage-based costs’. This model incentivises the creation and sharing of high-quality data products across various domains within the organisation:
- Incentivising through HR’s operational infrastructure: For instance, HR might create a data product like a headcount analysis. If other departments use this product, there are three possible scenarios: they could use it for free, pay a nominal fee, or contribute to its development cost. This incentivises HR to build good, reusable data products.
- Rewarding domain contributions: The board can encourage domains to develop valuable data products by offering financial rewards. This creates a positive cycle where the investment in creating quality data products is recognised and compensated.
- Cross-domain utilisation and compensation: Consider a domain creating a comprehensive customer list. Other departments might find this list valuable for cross-selling or upselling. They can pay to access this already developed data product, thus compensating the creating domain for their effort and resources.
- Cost and revenue model for data products: There's a cost to producing a data product, but also potential revenue when it's well-made and widely used. This economic model encourages domains to invest in creating useful and reusable data products.
- Discoverability and usage analysis in self-service platforms: For a self-service platform to be effective, it needs a robust system for discovering new products. This is where metadata becomes crucial. Tools like Denodo can track how often different data products are accessed, helping to gauge their value and utility. For example, a data product used ten times more than another might indicate higher value, though the importance of each product should be weighted differently (like board report figures which are less frequently used but highly valuable).
This approach to operationalising a data mesh encourages a culture of collaboration, quality, and efficiency in data product creation and utilisation, aligning closely with the organisation's broader strategic goals.
On the 'stick' side, there’s a suggestion to penalise poor data practices to maintain high data quality standards. This approach can take various forms within an organisation, acting as a deterrent against lax data management.
- Financial penalties for poor data quality: Some organisations opt to fine internal divisions for bad data. When data issues are identified, the responsible domain is penalised, either through direct fines or budget cuts for the following year. This method enforces the importance of maintaining high data quality.
- Consequences of non-compliance: Beyond financial implications, there are other repercussions for poor data management. For instance, a domain might be forced to relinquish ownership of certain data aspects if they consistently fail to meet quality standards.
- Data marketplace concept: In scenarios where data is the core business and product, operating in a data marketplace, the penalties for poor data can be more significant. The marketplace dynamics enforce a high standard, as poor data quality directly impacts the domain's credibility and economic success.
- Balancing quality with practicality: While imposing penalties for poor data quality, it's essential to balance these measures with practical limitations and internal checks. Implementing quality metrics and standards for data products helps maintain this balance.
- Starting the journey with data products: Organisations new to creating data products should focus on those that fill immediate reporting or regulatory gaps. These initial products act as a mainline pipe, demonstrating immediate value and utility. As these primary data products prove their worth, secondary and tertiary initiatives can branch out, leveraging and enhancing the initial data set.
- Building a positive reinforcement cycle: Each successful data product paves the way for future initiatives, creating a positive reinforcement cycle. This cycle leads to a gradual build-up of the data mesh, eventually achieving a critical mass where the benefits of the mesh are evident across the organisation.
Through the strategic use of both 'carrot' and 'stick' models, organisations can effectively operationalise a data mesh, ensuring the creation, maintenance, and utilisation of high-quality data products, which in turn support broader business objectives.
What makes a good or a bad data product?
To determine what a “good” or “bad” data product is, you must first establish clear quality metrics. These metrics help define the data's profile and form part of an implicit contract, informing users about the expected quality level of a data product. These metrics can be aggregated to give a headline score, such as 80% versus 70% quality. This way, users know what to expect in terms of data reliability.
Lifecycle management is another crucial aspect of a data product's quality. Data mesh highlights the need to keep the data platform clutter-free. Products that are no longer used or don't serve their intended purpose should be either upgraded or removed. This process is similar to how products are managed in a retail catalogue, allowing self-service users to make informed decisions based on the current relevance and quality of data products.
Federated governance plays a role in determining the quality of data products. With this model, a central layer of governance is minimally imposed across domains, each setting their own quality levels. The ultimate test of a product’s quality, however, comes from its use within the market. If a product is deemed poor but still in use, it signals a need for improvement, which can motivate the responsible domain to enhance the product's quality.
When it comes to specific quality tests, there are two key aspects to consider:
Quality Assessment
Certain errors in data are immediately apparent without needing additional context. Examples include mandatory fields left empty or filled with null values. Standard data profiling provides basic insights, such as identifying common but improbable values within a dataset. For instance, a birth date of 1/1/2001 being overly common might signal a data quality issue.
Accuracy Assessment
This involves deeper analysis and sometimes human review. For example, when two records look similar but have different addresses or phone numbers, determining their accuracy can require more than just algorithmic checks. It may involve cross-referencing with trusted sources or even direct verification from the concerned individuals.
In terms of enforcement, the 'stick' approach is more applicable in the earlier stages – especially in basic quality checks like uniqueness and rule adherence. These are more straightforward to monitor and enforce. Accuracy, being more complex and nuanced, presents greater challenges in terms of setting standards and implementing corrective measures.
How do you cost a product?
Pricing a data product typically comes under one of two primary schools of thought: cost-based pricing and value-based pricing. Each approach offers a different perspective on determining the price of a data product. In practice, the choice between these pricing strategies can depend on several factors, including the nature of the data product, the organisation's pricing philosophy, and the specific needs and expectations of the users.
Cost-Based Pricing
This method focuses on the cost of production. It looks at the total cost incurred to produce the data product and then factors in the expected usage to calculate the price. Additionally, considerations like the anticipated return on investment (ROI) over a certain horizon are included. For example, if producing a data product costs £10,000 and it's expected to be used 36 times over three years, the cost per use would be £277.78 (£10,000 / 36). This method is straightforward but has the limitation of not considering the product's value or impact on the user.
Value-Based Pricing
This approach shifts the focus from production costs to the perceived value of the product to its users. It involves assessing the value the product brings in solving a specific problem or fulfilling a need for the user. Factors like the cost savings from solving a problem, the potential increase in customer conversions, or even the cost of non-compliance (e.g., with GDPR) are considered. For instance, if a data product addresses a critical compliance issue that could otherwise cost an organisation £4 million in penalties, pricing the product at £1 million could be justified. This method is more complex but aligns the price with the product's value to the user and can lead to higher profitability.
Who decides the price?
In short, the market. Deciding the ‘right’ price for a data product is essentially guided by market dynamics, reflecting the principles of a free market. The initial pricing set by the domain, or the 'seller' of the data product, is just the starting point. The true test of this price comes from the market – the interaction between the seller and potential buyers. If the demand for a data product increases, indicating its utility and popularity, the domain can consider raising the price, aligning with the concept of dynamic pricing. Conversely, if a product is priced too high and fails to attract buyers, the market effectively signals that the price needs adjustment. This market-led approach ensures that the pricing of data products remains dynamic and responsive to real-time demand and usage patterns.
However, the autonomy granted to individual domains in setting prices does not mean the absence of overarching governance. In many organisations, broader pricing strategies and budget considerations are often guided by the board or CEO. This level of oversight ensures that the prices set by different domains align with the overall objectives and financial constraints of the organisation. It's a balancing act – maintaining the agility and autonomy of individual domains while ensuring that their actions dovetail with the organisation's strategic direction.
To successfully implement dynamic and market-responsive pricing, an organisation must first establish a solid foundation. This foundation is built on three key elements: robust data collection, well-established quality metrics, and effective governance frameworks. These elements are critical because they ensure that the advanced applications of a data mesh, such as dynamic pricing, are grounded in reliable and high-quality data.
Moreover, they guarantee that the pricing strategies developed are not only based on accurate and up-to-date information but also align with the broader goals and strategies of the organisation. This thorough groundwork is essential for an organisation to adaptively and effectively manage the pricing of its data products in a way that reflects market demands while staying true to its overall objectives.
Which organisations are data mesh and a data product economy right for?
The data mesh model and data product economy are most effective in large organisations, especially those with dedicated data science teams or specific data functions. These larger entities have the necessary resources and expertise to manage and leverage data products efficiently. In contrast, smaller organisations may struggle with the resource demands of maintaining a data mesh due to limited capacity.
Furthermore, organisations where data is a core part of their business, particularly those with numerous publicly facing data products, find significant value in adopting a data mesh. This model offers them a structured way to monetize their extensive data resources, potentially unlocking new revenue streams and competitive advantages. However, it's important to note that implementing a market-based approach to data management is complex and can pose cultural and political challenges, necessitating a careful and strategic approach.
Where should you go from here?
As you consider moving towards a data mesh model, it’s important to cover some foundational steps. The first is data collection and measurement – most organisations need to gather considerable data before they can effectively identify and measure key metrics. It’s essential to ask:
Have you conducted a cost analysis to determine how much your data production costs? Understanding the financial aspect of your current data processes is crucial. This analysis will provide a baseline to evaluate the feasibility and potential ROI of moving towards a data product economy.
Do you have established data quality metrics? Assessing and ensuring the quality of your data is fundamental. High-quality data is the bedrock of valuable data products and, consequently, a successful Data Mesh implementation.
Do you have the necessary self-service and metadata platforms to host and manage your data products? Platforms like Denodo for data virtualization, and Alation or data.world for data cataloging, are vital components. These tools will help you manage, share, and monetise your data effectively.
Not there yet? Begin with 'no regrets' actions. This means implementing frameworks and technologies that will be beneficial regardless of the specific direction your data strategy takes. These foundational elements serve as a springboard, enabling you to adapt and scale your data capabilities in the future. Some initial “no regrets” actions might be ensuring you have a robust data strategy and thorough metadata management - the strategy angle covering the top of the thought process, and metadata at the bottom, ensuring the detailed foundations you will need.
If your organisation is contemplating the adoption of a data mesh or is curious about the potential benefits of a data product economy, we can offer expert guidance. Our team is well-equipped to help you assess if these innovative strategies are in harmony with your organisational objectives and how they can be seamlessly integrated into your business framework. Get in touch through our contact form.
