Machine or Human?: An Inspection of the Scoring of Writing Essays
حوزههای تخصصی:
Automated Writing Evaluation (AWE) systems are used to evaluate measurable characteristics of written texts, thereby creating a scoring model based on a compilation of essays. While considerable research has focused on the feedback provided by AWE systems, there is a conspicuous absence of studies examining these tools specifically in the Iranian context. Therefore, this research aimed to investigate the consistency of scores obtained from automated systems and human raters. Furthermore, it sought to explore the perceptions of EFL learners regarding the application of AWE in their writing practices. To facilitate this investigation, 30 male and female IELTS students participated, each writing two essays: one selected from topics provided by the AWE system and the other derived from Cambridge Official IELTS past papers. The essays were assessed by both My Access and three human raters. For the topics designated for the AWE system, a significant and robust positive correlation was identified between the ratings assigned by human raters and the machine. A similar significant and strong positive correlation was also found for the second essay, which did not utilize pre-defined topics. The results of two linear regression analyses demonstrated that the scores produced by the machine could significantly predict human scores for both pre-defined and non-pre-defined topics. Additionally, the findings indicated that My Access Home Edition is perceived to significantly enhance students' accuracy and autonomy, although it does not contribute to improved interaction. This study presents important implications for writing instructors and the field of second language education.