The development of an artificial intelligence classifier to automate assessment in large class settings: Preliminary results

This evidence based practice paper presents preliminary results in using an artificial intelligence classifier to mark student assignments in a large class setting. The assessment task consists of an approximately 2000 word reflective essay that is produced under examination conditions and submitted electronically. The marking is a simple pass/fail determination, and no explicit feedback beyond the pass/fail grade is provided to the students. Each year around 1500 students complete this assignment, which places a significant and time-constrained marking load upon the teaching faculty.

This paper presents a Natural Language Process (NLP) framework/tool for developing a machine learning based binary classifier for automated assessment of these assignments. The classifier allocates each assignment a score representing the probability that the assignment would receive a passing grade from a human marker. The effectiveness and performance of the classifier is measured by investigating the accuracy of those predictions.

Several iterations and statistical analyses were carried out to determine operational thresholds that balance the risks of false positives and false negatives with the required quantity of human marking to assess the assignment.

The resulting classifier was able to provide accuracy levels that are potentially feasible in an operational context, and the potential for significant overall reductions in the human marking load for this assignment.