rating scale

The application of Multifaceted Rasch Measurement (MFRM) in rating test takers’ oral language proficiency has been investigated in some previous studies (e.g., Winke, Gass, & Myford, 2012). However, little research so far has ever documented the effect of test takers’ genders on their oral performances and few studies have investigated the relationship between the impact of raters’ gender on the awarded scores to male and female test takers. Thus, this study aimed to address the above-mentioned issue. Twenty English as a Foreign Language (EFL) teachers rated the oral performances of 300 test takers. The outcomes demonstrated that test takers’ gender differences did not have any significant role in their performance differences when they were rated by the raters of the same or opposite gender. The findings also reiterated that raters of different genders did not demonstrate bias in rating test takers of the opposite or same gender. Moreover, no significant difference was observed regarding male and female raters’ biases towards the rating scale categories. The outcomes of the study showed that both male and female raters assign fairly similar scores to test takers. This suggests no evidence based on which either male or female raters must be excluded from the rating process. The findings imply that there is no need to worry about the impact of gender for a more valid and reliable assessment.

Developing rating scales for assessing interactional performance is demanding since it is a relatively complicated procedure. The present study investigated the psychometric characteristics of the CAP rating scale (Wang, 2015) for assessing interactional competence at micro-level. To this end, 160 Iranian intermediate EFL learners were selected based on their performance on TOEFL iBT test from a language institute in Tabriz. Four interaction tasks were used to elicit students’ performance on interactional competence using the CAP rating scale. Five raters were recruited in the study to assign score to each individual’s performance. The participants were pretested and post-tested at the beginning and the end of the term through the same scale. The Pearson correlations were computed in order to estimate the test-retest reliability indices of the scale. In addition, five separate exploratory factor analysis (EFA) through the varimax rotation method were conducted in order to investigate the underlying constructs of the communication functions individually and as a total. The results revealed that the CAP rating scale enjoys a reasonable reliability indices and the four functions i.e. building argument, developing discussion, offering support, and shaping connection can be appropriate predictors of interactional competence. Some pedagogical and assessment implications are presented as well.  

A longitudinal study was planned, and a scale was suggested for assessing EFL learners’ oral presentations. The scale had three major evaluation components: `Preparation’, `Organization’, and `Presentation’. The students were informed about the rating scale against which their performances would be evaluated. Throughout the course each student had five performances on different occasions. The results of the study indicated that: (a) significant improvement was observed in the learners’ performances, (b) ANOVA results indicated that the five performances were significantly different, (c) there were high correlations between each paired performances, and (d) the rating scale was a reliable and consistent measure by means of which the instructor could assess the student’s speaking ability.

The objective was threefold: (a) to investigate the difference among online teaching performance of EFL student teachers at Iran University of Science and Technology who received peer feedback, expert feedback, and reflected on their own teaching practice; (b) to examine the difference between rating scores given by student teachers and those by instructor to online teaching; and (c) to determine student teachers’ best experiences, challenges, and lessons they learnt from expert feedback, peer observation, and reflection. The results of this mixed methods experimental research revealed no statistically significant difference in the three group’s scores on their second teaching practice; however, there was statistically significant difference between the scores of the first and second practices for all three groups. The results also showed that student teachers mostly underrated their first teaching, while they overrated their second teaching. The findings also indicated that their best experiences were related to using technological tools, working with Adobe Connect, making interesting materials, and using games and songs. Their challenges included lack of face-to-face communication, engaging all participants, and preparing suitable materials, whereas teaching new vocabulary in context, simplifying songs by pre-teaching their unknown words, and using online games and websites were the lessons they learnt.

Performance testing including the use of rating scales has become widespread in the evaluation of second/foreign oral language assessment. However, no study has used Multifaceted Rasch Measurement (MFRM) including the facets of test takers’ ability, raters’ severity, group expertise, and scale category, in one study. 20 EFL teachers scored the speaking performance of 200 test-takers prior and subsequent to a rater training program using an analytic rating scale consisting of fluency, grammar, vocabulary, intelligibility, cohesion, and comprehension categories. The outcome demonstrated that the categories were at different levels of difficulty even after the training program. However, this outcome by no means indicated the uselessness of the training program since data analysis reflected the constructive influence of training in providing enough consistency in raters’ rating of each category of the rating scale at the post-training phase. Such an outcome indicated that raters could discriminate the various categories of the rating scale. The outcomes also indicated that MFRM can result in enhancement in rater training and functionality validation of the rating scale descriptors. The training helped raters use the descriptors of the rating scale more efficiently of its various band descriptors resulting in a reduced halo effect. The findings conveyed that stakeholders had better establish training programs to assist raters in better use of the rating scale categories of various levels of difficulty in an appropriate way. Further research could be done to make a comparative analysis between the outcome of this study and the one using a holistic rating scale in oral assessment.