Abstract

This evidence based practice paper presents preliminary results in using an artificial intelligence classifier to mark student assignments in a large class setting. The assessment task consists of an approximately 2000 word reflective essay that is produced under examination conditions and submitted electronically. The marking is a simple pass/fail determination, and no explicit feedback beyond the pass/fail grade is provided to the students. Each year around 1500 students complete this assignment, which places a significant and time-constrained marking load upon the teaching faculty.

This paper presents a Natural Language Process (NLP) framework/tool for developing a machine learning based binary classifier for automated assessment of these assignments. The classifier allocates each assignment a score representing the probability that the assignment would receive a passing grade from a human marker. The effectiveness and performance of the classifier is measured by investigating the accuracy of those predictions.

Several iterations and statistical analyses were carried out to determine operational thresholds that balance the risks of false positives and false negatives with the required quantity of human marking to assess the assignment.

The resulting classifier was able to provide accuracy levels that are potentially feasible in an operational context, and the potential for significant overall reductions in the human marking load for this assignment.

Original languageEnglish
Article number37769
JournalASEE Annual Conference and Exposition, Conference Proceedings
Number of pages12
ISSN2153-5965
Publication statusPublished - 25 Jun 2023
Event2023 ASEE Annual Conference and Exposition - The Harbor of Engineering: Education for 130 Years, ASEE 2023 - Baltimore, United States
Duration: 25 Jun 202328 Jun 2023

Conference

Conference2023 ASEE Annual Conference and Exposition - The Harbor of Engineering: Education for 130 Years, ASEE 2023
Country/TerritoryUnited States
CityBaltimore
Period25/06/202328/06/2023

Bibliographical note

Publisher Copyright:
© American Society for Engineering Education, 2023.

Keywords

  • Automated Grading
  • Natural Language Processing
  • Reflection

Fingerprint

Dive into the research topics of 'The development of an artificial intelligence classifier to automate assessment in large class settings: Preliminary results'. Together they form a unique fingerprint.

Cite this