Amplifi focuses on the core pillars of a modern data ecosystem. These pillars include People, Process, Technology, Data, Change Management, and Culture, each playing a critical role in creating a resilient and adaptable data environment. Among these, you must design with change in mind.
What is change?
Change is any significant alteration in technology, processes, or business environment that disrupts the status quo. It can be identified through new tools, processes, or strategic shifts and is often accompanied by stress, learning curves, and adaptation challenges. Changes can be either planned or unexpected, sudden or gradual, and may be welcomed or unwelcome. They can present opportunities to enhance our business practices or compel us to rethink our entire approach. For example, we might plan to adopt new technology to streamline processes, or, like a restaurant forced to adapt during the pandemic, we may need to pivot to takeaway and delivery services to survive.
Preparing for change
Change is inevitable. It is a fact of life. This is especially true in business and therefore it’s important to keep on top of it and react accordingly. Technology will always evolve, bringing unforeseen advancements that reshape our world. We've seen monumental shifts like the internet revolution, cloud disruption, and now AI - each of these changes has demanded significant adaptation.
Change is not confined to the professional world; it’s a part of our personal lives as well. Whether it’s relationships, career shifts, new systems, organisational changes, or even more disruptive events like wars or crises, change is inevitable. While some changes bring opportunities for growth and improvement, not all lead to positive outcomes. Many changes, like conflicts, can be deeply harmful, with few immediate benefits. Yet, regardless of the nature of change, our ability to adapt and respond plays a key role in how we navigate these transitions and move forward.
Effective communication is essential when navigating change, as people naturally resist what is unfamiliar. By clearly explaining the reasons for the change, its long-term benefits, and how individual roles will be affected, you can reduce resistance and manage expectations. Addressing these factors early on can help create a smoother transition and foster a more positive, cooperative environment for the change process.
How to design for and embrace change
Within the IT function, preparing for change involves efficient designing, proactive planning, clear communication, support systems, continuous learning, and feedback loops. By understanding and addressing these aspects, the IT team can effectively navigate and thrive amidst ongoing changes.
1. Design composable architectures to seamlessly enhance your ecosystem with new capabilities
When change happens, whether it’s gradual or sudden, technology often faces resistance, especially if you’re relying heavily on a single core technology vendor. While there are benefits for having the same technology vendor/provider across the board, such as Informatica, Microsoft and IBM there are limitations. One of the biggest drawbacks is that when an industry shift occurs, like the rise of AI, you must wait for your vendor to implement the necessary changes, meaning you can’t always just plug and play with best of breed.
If you’ve invested heavily in a certain technology, understand its integration capabilities and how it can be decomposed and augmented with new capabilities. The market is moving towards more composable services - systems that can be easily broken apart and swapped out. These services should integrate seamlessly, providing the flexibility to adapt to new technologies and industry shifts without being tied down. These architectures are designed to integrate seamlessly, providing the flexibility to adopt emerging technologies and adjust to industry shifts without the constraints of a monolithic system.
For example, with Microsoft Azure Data Factory, you pay for resources on demand rather than committing to long-term license fees, which can become sunk costs. This pay-as-you-go (PAYG) model aligns well with the principles of a modern data ecosystem. Unlike traditional approaches, PAYG models do not incur sunk costs, allowing for greater flexibility and agility. Embracing PAYG models equips organisations to adapt quickly to change without being constrained by long-term commitments, ensuring that your technology can evolve with the market and making your organisation more resilient and ready to embrace change.
2. Integration will be at the core of change
Integration is right at the core of all of this. If you have data silos, they will need to be able to communicate with each other eventually - the integration layer is where this happens. With point-to-point integrations, each system requires direct connections to every other system, resulting in a quadratic growth of necessary integrations. For example, two systems only require one integration, however 6 systems require 15 integrations, more than double, and so as the number of systems increases, the complexity and number of integrations multiply rapidly. This can lead to dozens if not hundreds of point-to-point integrations, making it extremely challenging to manage and adapt to change.
Decoupling is way to break down point to point solutions. It involves creating an "airlock" between systems. This means changes in one system do not directly impact another, allowing for easier amendments and updates without widespread disruption. A standard way to achieve this would be through the use of APIs in the integration layer over the top of business systems. By focusing on integration and decoupling, organisations can better prepare for and adapt to technological changes, ensuring a more resilient and flexible data ecosystem.
3. Define role-based security
To successfully embrace change, it's crucial to structure your ecosystem in such a way that enables people's roles to evolve as the needs of the business evolves. One common pitfall is not having security mechanisms set up at a role level.
In people-centric organisations, there is often a structure where everyone has a specific role. However, when it comes to security and data ownership structuring, it's frequently defined by a person rather than a position. This approach can lead to significant security and operational issues.
In a well governed, modern data ecosystem roles should be very well defined, with permissions assigned to the role rather than the individual. This way, when someone is onboarded or offboarded, the associated security and access permissions are managed automatically. Organisations often struggle with two key aspects: implementing role-based structures and properly offboarding security access.
By ensuring that roles are clearly- defined and permissions are role-based, you can mitigate the risks associated with incomplete offboarding processes. This approach not only enhances security but also makes it easier to move people around within the organisation, allowing for greater flexibility and adaptability in the face of change.
4. Documenting your business processes to make change possible
While IT change is beneficial and adapting to inevitable people changes is necessary, it's the business processes that truly drive an organisation. Your business must be able to react to changes—consider the print and media industry, where the shift from print to online is a fundamental change driven by market forces.
Understanding these processes, from traditional printing to implementing paywall content online, is essential. Technologically, this transition is complex and requires robust support. However, it's the business process changes that are most critical. Business model transformations and evolving processes are necessary to stay relevant. We've seen significant disruptions with social media, the internet, and now AI. Being agile in your business processes to embrace these changes is critical.
Documentation plays a vital role in this adaptability. Clearly recording who is responsible for what and mapping these responsibilities onto the necessary technology ensures that everything is understood and manageable. This thorough documentation is a crucial part of preparing for change.
5. Understanding and documenting the data lifecycle
Data will change; it’s always evolving. To effectively manage this, you need to understand the lifecycle of your data: when it is created, updated, removed, and shared. Each of these steps in the data lifecycle involves specific processes and technological considerations that must be thoroughly documented.
Using metadata management tools is essential for this:
Data Catalog: This tool documents what the data is and tracks how it can change. It provides a comprehensive inventory of your data assets, helping you understand and manage data throughout its lifecycle.
Data Lineage: This allows you to see how data is altered by different touchpoints—how, when, and by whom changes are made. It provides a detailed view of the data's journey, showing what it was before and after each change.
Metadata Activation: Who or what is using the data and when? Utilises multiple forms of metadata and analyses them to identify which data is valuable based on usage patterns and highlights "dead space" data that hasn’t been accessed in a certain number of years, allowing it to be archived or deleted.
Both data catalogs and lineage tracking reveal the various elements of data change, while metadata activation is the process of making metadata actionable by embedding it into workflows, systems, and decision-making processes. They are crucial for managing risks, ensuring security, maintaining traceability, and understanding data lineage. These aspects are fundamental to a modern data ecosystem and are essential for embracing change effectively.
Final thoughts
If this article has resonated with you, I wanted to share my advice for a few concrete first steps to start designing your ecosystem for change in your organisation:
Integration
Have you thought about an integration layer? Integration is critical to any modern data ecosystem and needs to be well-designed before implementation. However, it's essential to think big in terms of the overall structure but start small by focusing on high-priority use cases. Consider a flexible integration strategy that supports future growth and change. By decoupling your systems with an integration layer, you allow room for incremental improvements - so you can modify or scale one part without disrupting the whole. Begin by planning the architecture but implement iteratively, starting with the core systems or business functions that matter most now. As needs evolve, you can (and should) expand capabilities over time, adding more systems or functionalities seamlessly.
Start Capturing Metadata (if you haven’t already)
How do you prepare for strategy and change? If you have a messy heap of systems, it’s important to think big by envisioning a streamlined, efficient metadata management process, but start small by focusing on immediate, actionable steps. Start by documenting all the ins and outs - purposes, processes, people. This documentation will form the foundation for future improvements, even if you can’t implement sweeping changes right away. Starting small, you’ll standardise and streamline incrementally. Concentrate first on capturing metadata and process documentation for high-priority systems. Tools like data.world or Microsoft Purview can help scale this work, enabling you to expand as you grow, ensuring you’re always prepared for change.
Use expert advice
When preparing for change, thorough designs, proactive planning, clear communication, and adaptability are essential. Flexibility is key, as over-reliance on a single technology provider can hinder adaptability, making composable services and pay-as-you-go models ideal for an agile response. Integration is crucial, with decoupling and APIs helping to manage complexity. Role-based security enables flexibility in managing people, while clear documentation of business processes and the data lifecycle ensures adaptability. Effective communication is vital, explaining the reasons for change, how roles will be affected, and managing resistance through transparency.
By taking these steps, you can build a resilient and adaptable organisation ready to embrace change and thrive in an ever-evolving landscape.
If you’re considering how your organisation can prepare your data ecosystem to be more adaptable to change, we’re here to help. Our Modern Data Ecosystem Capability Assessment might be the best next step, get in touch here to speak with Matt, or another of our data experts.
To read more, download our guide to Implementing a Modern Data Ecosystem below.
