پوریا بقایی

شواهد بیانگر این می باشد که تغییر پذیری در ارزیابی نگارش دانشجویان فقط نتیجه تفاوت مهارت نوشتاری آنها نیست بلکه عوامل بیرونی خاصی در این امر دخیلند و عوامل مرتبط با مصحح، فعالییت، موقعییت یا تعامل هر یک از اینها میتوانند در تصمیم گیریها و استنباطها در مورد توانایی نگارش فراگیران تاثیر بگذارند. هدف این تحقیق این است که مساله تغییرپذیری را در داوری مصحح به عنوان منبع خطای سنجش در ارزیابی نگارش فراگیران زبان انگلیسی به عنوان زبان خارجه مورد بررسی قرار دهد. به این منظور مقالات 32 دانشجوی ایرانی زبان انگلیسی بوسیله شش مصحح ارزشیابی و نتایج با استفاده از مدل رش و برنامه فست مورد تحلیل قرار گرفتند. یافته ها نشان دادند که تفاوتهای معنی داری در میزان سختگیری مصحح ها وجود دارد و تعامل مصحح با فراگیر باعث ایجاد تورش در ارزشیابی میشود. از آنجایی که ارزیابی مهارت نوشتن امری ذهنی میباشد و بر اساس داوری مصحح می تواند متغیر باشد، بررسی و شناخت عواملی که باعث عینی تر شدن این ارزیابی میشود امری ضروری به شمارمی رود. ذهنی بودن روند ارزیابی مهارت نوشتن تهدیدی برای روایی آزمون است و سبب می شود که نمره فراگیر نشان دهنده مهارت واقعی وی نباشد. این تحقیق نشان می دهد چطور ارزیابی می تواند کارامد و سودمند باشد و چگونه می توان انصاف و صحت عملکرد ذهنی را سنجید.

Cognitive diagnostic models (CDMs) have received sustained attention in educational settings because they can be used to operationalize formative assessment to provide diagnostic feedback and inform instruction. A large number of CDMs have been developed over the past few years. An important component of all CDMs is a Q-matrix that specifies a particular hypothesis about the relationship between each test item and its required attributes. The purpose of this study was to construct and validate a Q-matrix for the reading comprehension section of the Islamic Azad University English Proficiency Test (IAUEPT), as an advanced English placement test designed to measure language ability of Ph.D. candidates who tend to pursue their studies in the IAU. To achieve this, using item responses of 1152 candidates to twenty items of the reading section of the test, an initial Q-matrix was constructed based on theories and models of second/foreign language (L2) reading comprehension, previous applications of CDMs on L2 reading comprehension, and brainstorming and consensus of five content experts. Then, the initial Q-matrix was empirically validated using the method proposed by de la Torre and Chiu (2016) and checking mesa plots, and heatmap plot. Five attributes were derived for the reading comprehension section: vocabulary, grammar, making an inference, understanding specific information, and identifying explicit information. Finally, the analysis of the Generalized Deterministic Inputs, Noisy “and” Gate (GDINA) regarding absolute fit at item- and test-level as well as three residual-based statistics showed the accuracy of the Q-matrix and a perfect model-data fit.

Language assessment literacy has been addressed in a wealth of research. However, many studies have attempted to measure teachers’ assessment literacy, there is still a gap that prompted us to investigate the area from the EFL teachers' assessment literacy needs perspective. To accomplish the purpose, in line with the changes in classroom assessment over the past decades, this study was an attempt to develop and validate an inventory on Teachers Assessment Literacy Needs (TALNs). As the first stage, a set of items was generated through an extensive review of the relevant studies. In the quantitative phase, the developed inventory was administered to 159 English as a foreign language teachers selected through convenience sampling. An inventory construction and validation framework consisting of exploratory analyses was used to examine the construct validity of the proposed inventory. The results indicated that the inventory can be best explained by four components which are knowledge of language assessment literacy, consequences of language assessment literacy, processes of language assessment literacy and teachers’ expectations of Continuing Professional Development (CPD) programs. The TALNs inventory developed in this study aimed to help practitioners and researchers to investigate teachers’ needs in assessment literacy. Fulcher’s (2012) assessment literacy framework was drawn on as the analytic model guiding the study.

Testing organizations are faced with increasing demand to provide subscores in addition to the total test score. However, psychometricians argue that most subscores do not have added value to be worth reporting. To have added value, subscores need to meet a number of criteria: they should be reliable, distinctive, and distinct from each other and from the total score. In this study, the quality of subscores from two subscore augmentation models (Wainer and Yen) were compared in terms of distinctness and variability. The reliabilities of the Wainer-augmented subscores were also examined. The models were applied to a high-stakes English language proficiency test in Iran. The results of the study showed that Yen better satisfied subscore distinctness while Wainer best preserved variability and had high-reliability subscores. In other words, Yen-augmented subscores had lower correlations while Wainer-augmented subscores better discriminated examinees with different ability levels. Thus, none of the examined models of subscoring satisfied all criteria. The results of the study are discussed and suggestions for future research are provided.

The purpose of this study was to compare the functioning of five restrictive CDMs, including DINA, DINO, A-CDM, LLM, and RRUM, against the G-DINA model to identify the best-fitting CDM which can better explain the interaction underlying the attributes of the reading comprehension section of an Iranian high-stakes language proficiency test. To achieve this aim, item responses of 1152 examinees to the items of the test were examined. The six CDMs were initially compared in terms of relative and absolute fit statistics at test-level to choose the best model. It was found that the G-DINA model outperformed compared to the restrictive models; thus, it was selected for the second phase of the study. Concerning the second purpose of the study, the G-DINA was used to identify strengths and weaknesses of the examinees. The results revealed that making an inference and vocabulary are the hardest attributes for examinees of the test, and understanding the specific information is the easiest attribute. Finally, the models were also compared at item-level. The presence of a combination of L2 reading attributes was found.