- Qualifications: Managing data across different assessment levels
- Learner data and transactions: Handling course registrations, grades, results, and key student record data
- Examination results: Keeping track of grade boundaries and student performance
- Regulatory compliance: Reporting to regulatory bodies on time and with complete accuracy
Meeting the reporting requirements of both regional government and regulatory bodies, presents a significant challenge due to strict deadlines, the need for absolute accuracy, and the highly prescriptive data formats required. Ensuring compliance often means navigating complex data preparation processes, which can make it difficult to react quickly to submission requests.
All of this, coupled with the nature of the complexities around the structure of qualifications, assessments, and papers, means the data landscape for NCFE is complex and intricate.
Data Challenges
NCFE was already on a journey to build a robust and scalable cloud architecture, alongside an optimized, future-ready data governance model. To support this, they brought in Amplifi to assess their data landscape and identify focus areas for improvement.
Amplifi began with a data capability assessment, informing a roadmap for NCFE to strengthen the foundations of their architectural and governance capabilities.
NCFE established three clear goals:
- Move their data to the cloud to make it more scalable, accessible, and efficient.
- Make it easier to find and track data, ensuring better integration across the organization.
- Strengthen data governance across different areas, improving consistency, compliance, and control.
The solution
Using Microsoft Purview, Amplifi and NCFE worked together to develop a unified data governance solution, ensuring data from both cloud and on-premises sources was discoverable, traceable, and properly governed across different domains and systems.
With Microsoft Fabric, Amplifi helped NCFE redefine its data approach, onboarding key domains into a highly integrated cloud architecture, making data easier to access, manage, and trust.
As with any Microsoft Fabric delivery, the best approach is to start small and scale up. The initial project focused on three key regulatory reports, working backwards from the final output to trace the data journey, identifying which sources, tables, and warehouses fed into each report. From there, a medallion architecture was implemented in Microsoft Fabric to bring structure and clarity to the data across 3 layers:
Bronze Layer: A direct copy of source data, ensuring all raw information is retained without transformation.
Silver Layer: Built with scalability in mind, making it easier to integrate future reports and data sources. This included restructuring it to introduce reference tables, and consolidating data into three clear categories: Products (qualifications, assessments etc), Learners, and Results.
Gold Layer: Designed to make reporting simpler by creating structured query outputs, which reduced the number of joins needed across multiple tables. It is also the home of the Enterprise Data Warehouse, designed to allow flexibility in reporting and analysis. This approach allows NCFE to efficiently produce reports and creates a strong foundation for future expansion within Microsoft Fabric.
Governance with Microsoft Purview
At the same time, Amplifi implemented Microsoft Purview, connecting it to Microsoft Fabric, allowing it to automatically scan and catalog all Microsoft Fabric tables. This gives NCFE a clear view of their data, making it easier to define classifications and strengthen governance. It’s the first step in a more structured approach to data governance, ensuring data is well-documented, easy to find, and simple to work with.
Emily Heath, Data Integration Practice Director at Amplifi, commented, “This project highlights how well Microsoft Fabric and Microsoft Purview can work together, even though they serve different purposes. Microsoft Fabric structures and processes the data, while Microsoft Purview brings visibility and governance. Delivering both at the same time takes the right expertise, balancing architecture and governance to create a more organized, efficient, and scalable data approach.”
The results
Since the outset of Amplifi’s engagement with NCFE, they have significantly strengthened the foundations of their architectural and governance capabilities. This is highlighted through the ease of access and reporting, and the trust that NCFE can now place in their data landscape. Some key achievements include:
- Significantly faster data access and processing times
- Straightforward access to answer complex data queries, through code and logic being stored within Microsoft Fabric
- Better data quality and traceability for sustainable data quality practices
NCFE’s Director of Technology, Data & PMO, Nick Evans commented, “The expertise and alignment that Amplifi have brought to our Enterprise Data warehouse project has been significant. We have vast amounts of data stored, through multiple disparate systems, and the ability to have “one source of the truth” allows us to trust and make strategic decisions based on those data sets.
Data Governance began as an immature capability, but through the implementation of Microsoft Purview, alongside our Microsoft Fabric medallion architecture, we can locate quality issues at source, through data lineage, and drive KPI’s against the resolution of these issues. We can now make informed data validation changes through our origin data creation points, which is a game changer.
We’ve enjoyed the personability of Amplifi and see them as an extension of our team, as opposed to a traditional transactional relationship.”
What’s next?
Seeing the impact of the work so far has given NCFE a clear vision of what’s possible, and they’re now considering how they can build further on their approach to Microsoft Fabric.
Learn more about our expertise implementing Microsoft Fabric and Microsoft Purview in our guide below, on our Microsoft partner page, or get in touch to speak about your organization's unique needs.
