Deeply Personalised Learning with data and AI
At my school I am blessed to work with exceptional teachers who provide a nurturing environment where each student can thrive and develop. My challenge is to support teachers by applying modern methods that add additional value to the student's growth.
Leveraging Microsoft Fabric to securely enable machine learning.
As part of my role I assess many data points concerning a student's growth from Kindy to Year 12. Recently I have discovered Microsoft Fabric, a tool that provides an all-in-one secure platform, covering everything from data movement to data science.
Schools collate a huge amount of data concerning their students. For example: academic performance/assessment, extra-curricular achievement, attendance, pastoral care notes, and report comments. These labels can all come together to create a holistic view of a student, and teachers should have access to a descriptive overview of each student’s progress.
But there are a few challenges:
Often the data is split across multiple databases.
In the case of Seqta data, a query can only be executed from a single, 'white-listed IP address'. This makes data analysis in third-party products hard.
It is difficult to develop predictive algorithms or leverage Artificial Intelligence using Python or R while maintaining security.
Collaboration on projects is almost impossible, with many artifacts split between various sources and folders. And normally the intellectual property (IP) of the project is claimed by one person and when that person leaves the organisation, the IP goes with them.
When ingesting data the data analyst typically repeats the same steps: extracting the data, transforming through filters and cleansing, and then loading with table relationships to create a semantic model. This repetition is inefficient and prone to error.
Microsoft Fabric solves these issues and provides business continuity and project sustainability. Furthermore, the data is incredibly secure, never leaving your Microsoft tenant and residing right next to you school’s SharePoint and OneDrive data.
The image below outlines the workflow that I have used to overcome the above challenges and modernise many of my data projects. Outcomes include Predictive Analytics with ATAR and reference writing with Azure Open AI.
The gateway acts as a bridge between your on-premises data sources and Fabric. The gateway is installed on a server within your network, and it allows Fabric to connect to your data sources through a secure channel without the need to open ports or make changes to your network.
Data flows and lake houses
Moving data to the lake house opens almost unlimited possibilities. You don't need to work with complex code to achieve this (and CoPilot can assist you to overcome most problems).
In short: The data flow connects to your gateway. The interface is a Power Query editor, so you can easily stage and monitor your transformations. Dataflows execute simple SQL queries and they can be set to refresh at specific intervals. For example I capture enrolment data every 12 hours.
The output of the data flow is a data lake house. Once the tables reside in the lake house you can create a semantic model. This model can be loaded directly into Power BI, both online (fully enabling Mac users) or from Power BI desktop.
Lake houses can be used with other powerful Microsoft services including Azure Synapse (Anaconda style notebooks) and Azure Open AI (using the Python Open AI modules). You can also link your dataverse environments to the lake house so you can leverage Power App solutions driven by your school data.
Machine learning with notebooks
Fabric notebooks allow you to explore your data and train machine learning experiments. In the below image I am exploring historical ATAR data and creating linear regression models to predict current student, course based, scaled scores.
Notebooks easily connect to your data lake houses and centralise your machine learning experiments. Collaboration with other users in the organisation is easy and this enables project sustainability.
Other projects of note
Scaling Azure Open AI with student data is a game changer. The Synapse notebooks enabled us to write 143, Year 12 student references (each one around 400 words) using data from our local Seqta database. The script took 28 minutes to run and cost the oganisation $13.87.
Conclusion
The amount of data we harvest reagarding each student is huge. Only machine learning can evaluate this at scale.
Data driven insights and AI solutions don't just save teachers valuable time, sometimes the machine can discover trends that are not always obvious on the surface level. If we can anticipate and see things before they happen, then we can ensure the best outcomes, tailored for each individual student.
Get started here: https://www.microsoft.com/en-us/microsoft-fabric