Test fairness

Validation is an important enterprise especially when a test is a high stakes one. Demographic variables like gender and field of study can affect test results and interpretations. Differential Item Functioning (DIF) is a way to make sure that a test does not favor one group of test takers over the others. This study investigated DIF in terms of gender in the reading comprehension subtest (35 items) of a high stakes test using a three-step logistic regression procedure (Zumbo, 1999). The participants of the study were 3,398 test takers, both males and females, who took the test in question (the UTEPT) as a partial requirement for entering a PhD program at the University of Tehran. To show whether the 35 items of the reading comprehension part exhibited DIF or not, logistic regression using a three step procedure (Zumbo, 1999) was employed. Three sets of criteria of Cohen’s (1988), Zumbo’s (1999), and Jodin and Girel’s (2001) were selected. It was revealed that, though the 35 items show “small” effect sizes according to Cohen’s classification, they do not display DIF based on the other two criteria. Therefore, it can be concluded that the reading comprehension subtest of the UTEPT favors neither males nor females.

This study examined the effect of rater training on promoting inter-rater reliability in oral language assessment. It also investigated whether rater training and the consideration of the examinees’ expectations by the examiners have any effect on test-takers’ perceptions of being fairly evaluated. To this end, four raters scored 31 Iranian intermediate EFL learners’ oral performance on the speaking module of the IELTS in two stages (i.e. pre- and post-training stage). Furthermore, following Kunnan’s (2004) Test Fairness Framework, a questionnaire on fairness in oral language assessment was developed, and after pilot testing and validating, it was administered to the examinees at both stages. The examinees’ expectations were taken into account in the second round of the speaking test. The results indicated that rater training is likely to promote inter-rater reliability and, in turn, enhances the fairness of the decisions made based on the test scores. It was also concluded that considering students’ expectations of a fair test would improve their overall perceptions of being fairly evaluated. The results of this study sought to provide second language teachers, oral test developers, and oral examiners and raters with useful insights into addressing fairness-related issues in oral assessment.

Iranian National University Entrance Exam (INUEE) as a nationwide high-stakes test is held annually to screen Iranian high school graduates and admit them into higher education programs in universities. This high-stakes examination has a wide range of impacts on test takers as the primary stake-holders and the parents, teachers, and high school principals as the secondary stakeholders. This study reports the impacts of INUEE on high school teachers and principals. To this aim, 27 teachers and 18 principals from three western provinces of Iran sat for a structured interview. Each interview lasted nearly 30 minutes. All the interviews were audio-recorded and transcribed. Next, following the Grounded Theory (Glaser & Strauss, 1967) as the basis of analysis, the transcriptions were subjected to content analysis to extract common patterns and recurring themes. Content analysis was applied to codify the transcribed interview data through an inductive process of frequent moving back and forth to extract common patterns and recurring themes of the data. After coding and 'quantitizing' the data (Dörnyei, 2007), the basic themes were identified, frequency counted, and tabulated. The results indicated that from the majority of the participants' perspective, the INUEE has detrimental consequences for students, teachers, school principals, and the educational curriculum. The findings of the study underscore the consequential invalidity and unfairness of the test and its negative impacts on different aspects of the educational system. The findings provide practical implications for educational policy-makers, school principals, and teachers highlighting the necessity of their awareness of negative consequences of INUEE.