Our suite of education solutions at Advanced takes advantage of the latest technological advances. In this blog, we take a look at how Artificial Intelligence (AI) has been implemented in our bksb and Smart Coach solutions to improve assessment and learner risk analysis.
The science behind bksb
Using decades of data and industry-leading expertise, the team behind bksb have designed the solution to intelligently assess, measure and promote learning for English, Maths and Digital Skills. We have integrated AI and Item Response Theory (IRT) into the assessment system, allowing the software to analyse the difficulty of each question, choose the most suitable questions to make up an assessment and measure each learner’s progress using complex algorithms based on probability and best-fit calculations.
Measuring and Promoting Improvement
Our complex AI assessment tools are only useful if we have equally detailed and intelligent metrics to measure a learner’s progress. To this end, we divide the standard achievement levels into 10 decimal points, as in the diagram below. This allows us to not only define what level a learner sits at currently but also how close they are to reaching the next level. This information is then put to use by informing the level of the questions given to a learner.
For example, where a learner is identified as working at L1.1 (a low level 1), they will be given access to both Level 1 and Level 2 content for that module, but the platform will recommend that they study at Level 1. However, when the same learner improves to L1.5 (midway between level 1 and 2), the recommendation will change to Level 2. The system recommends content at the closest level to the learner’s current achievement, introducing higher content when the student is ready for it. By managing the journey from level to level this way, learners are stretched and challenged in a sensible and realistic manner.
From ‘progress’ to improvement
Our more detailed measurement system also allows for improvement to be tracked at a more granular level, with trends that would otherwise be missed used to analyse achievement. Where other learning systems would track progress according to whether or not a certain module has been ‘completed’ by the learner, bksb moves beyond this checklist-stye tracking to understand a learner’s improvement trends at a deeper level.
For example, where a learner may have reached entry level for three consecutive assessments, at the basic level no progression would be identified. However, using bksb’s metrics, we might observe a move from E.0 to E.3 to E.7, indicating that the learner is improving consistently and well on their way to level 1.
Using Item Response Theory to build assessments
The measurement systems explained above are implemented in initial and diagnostic assessments within bksb. However, for these measurements to be accurate and fair, certain considerations must be taken when choosing which questions make up an assessment.
The answer is not as simple as compiling a certain amount of random questions for a given level and subject into an assessment, as this will not result in fair results across your learners. This is due to two reasons:
- No two questions will be exactly equal in terms of difficulty – an element of luck will therefore always be involved in a random question selection, meaning that learners may receive similar results after taking assessments which vary in difficulty.
- Any multiple-choice questions have a chance of being guessed correctly – an assessment with more multiple-choice questions will put any candidates at an advantage compared to an assessment with more free-text questions.
Adaptive assessment and Item Response Theory
Rather than trying to make every question exactly equal in difficulty, or allocate different marks for questions depending on difficulty, bksb uses adaptive assessment. This process allocates questions to a learner in real-time, based on how they are performing within the assessment, to focus in on a specific ability group to categorise the learner. This personalised approach ensures a consistent assessment experience for all your learners and ensures that every single answer is providing useful data involved in calculating the learner’s achievement.
Our adaptive assessment process is implemented using Item Response Theory (IRT). IRT has been in existence long enough to be critically appraised by leading psychometricians, statisticians and mathematicians, with the academic consensus confirming that IRT is by far the most reliable and accurate method for testing ability. Indeed, it is now accepted that it is impossible to create any form of adaptive assessment without implementing the IRT algorithms.
These algorithms perform a series of calculations to assess the suitability of each question, including factoring in the chance of successfully guessing a multiple-choice question. These algorithms are used in conjunction with a pass rate for each question derived from our wealth of historic answer data. All of this is then used with a ‘best fit’ algorithm to identify the ideal next question for the learner.
This is used along with constant adjustments to the learner’s ‘ability estimate’ following every question. Rather than being random, the best question for each learner is chosen each time after their previous responses have been analysed. In contrast to competing systems, which merely total students’ scores and use boundaries to determine their results, this use of artificial intelligence is a massive step forward. Learners and teaching staff can be assured that the values attributed to each question are entirely valid, having been obtained through the manipulation of large datasets containing hundreds of millions of student answers.
At bksb, innovative tech and complex behind-the-scenes functions are used to deliver immediate and powerful results for assessments. By personalising assessment and providing meaningful insight, learners remain engaged and teachers are empowered to make the most of all the data collected during the assessment process.
AI and machine learning in Smart Coach
Smart Coach is a new module from Smart Assessor that uses AI and Machine Learning (ML) to identify early signs of learner attrition and helps tutors and assessors get at-risk learners back on track.
Developed in collaboration with researchers at Aston University and our own data science experts, Smart Coach analyses every aspect of learner data to produce a sophisticated model of learner risk measurement.
Using AI to predict attrition
Drawing on a large dataset, Smart Coach applies its risk model to each individual learner, using metrics such as number of logins, pieces of evidence signed off, and session attendance to produce a risk value between 0 and 1 to two decimal places. Assessors can then plot this risk over time, using individual snapshots and a trend line, as well as viewing the average risk over time for an individual learner or their entire caseload.
Empowering tutors and assessors
This flexibility offers much more than isolated data points, giving assessors data that is sophisticated enough to make informed decisions and predictions. By leveraging learner data with Smart Coach, tutors and assessors can ensure that learners are helped to get back on track with early interventions. To support these interventions, the Smart Coach interface offers quick action buttons with recommended strategies for success, such as booking meetings, recording action points, or reviewing evidence files.
While it is no replacement for engaged and informed staff, and cannot take external circumstances into account, Smart Coach is a valuable AI tool that empowers tutors, assessors and more to make the most of the data at their fingertips and take informed action to reduce learner attrition.
Innovation at Advanced
At Advanced, our passion for creating best-in-class solutions means that we are always looking to the latest developments in tech to empower staff and solve problems. With bksb and Smart Coach, we have taken some of the most exciting aspects of AI and ML to offer personalised and powerful learner assessment and monitoring tools solutions.
