Cognitive Diagnostic Assessment

Of paramount importance in the study of cognitive diagnostic assessment (CDA) is the absence of tests developed for small-scale diagnostic purposes. Currently, much of the research carried out has been mainly on large-scale tests, e.g., TOEFL, MELAB, IELTS, etc. Even so, formative language assessment with a focus on informing instruction and engaging in identification of student’s strengths and weaknesses to guide instruction has not been conducted in the Iranian English language learning context. In an attempt to respond to the call for developing diagnostic tests, this study explored developing a cognitive diagnostic reading comprehension test for CDA purposes. To achieve this, initially, a list of reading attributes was prepared based on the literature and then the attributes were used to construct 20 reading comprehension items. Then seven content raters were asked to identify the attributes of each item of the test. To obtain quantitative data for Q-matrix construction, the test battery was administered to 1986 students of a General English Language Course at the University of Tehran, Iran. In addition, 13 students were recruited to participate in think-aloud verbal protocols. On the basis of the overall agreement of the content raters’ judgements concerning the choices of attributes and results of think-aloud verbal protocol analysis, a Q-matrix that specified the relationships between test items and target attributes was developed. Finally, to examine the CDA of the test, the Fusion Model, a type of cognitive diagnostic model (CDM), was used for diagnosing the participants' strengths and weaknesses. Results suggest that nine major reading attributes are involved in these reading comprehension test items. The results obtained from such cognitive diagnostic analyses could be beneficial for both teachers and curriculum developers to prepare instructional materials that target specific weaknesses and inform them of the more problematic areas to focus on in class in order to plan for better instruction.

Although Diagnostic Classification Models (DCMs) were introduced to education system decades ago, it seems that these models were not employed for the original aims upon which they had been designed. Using DCMs has been mostly common in analyzing large-scale non-diagnostic tests and these models have been rarely used in developing Cognitive Diagnostic Assessment (CDA) from scratch. Despite the prevalence of retrofitting CDA studies, true applications of CDA are believed to be rare since, firstly, a coherent framework to conduct such studies had not been available and, secondly, researchers were not able to analyze various DCMs according to the same model fit indices and criteria. This paper presents a summary of different types of DCMs and reviews true and retrofitting CDA studies. Having examined the limitations of previous CDA studies, the present study argues for the implication and application of Ravand and Baghaei’s (2019) framework to conduct true CDA studies. This framework is of importance since not only does it fit into prominent frameworks in education assessment such as Cognitive Design System and Assessment Triangle, but also it can provide test-developers with practical steps in conducting valid cognitive diagnostic tests.

Investigating the processes underlying test performance is a major source of data for supporting the explanation inference in the validity argument (Chappelle, 2021). One way of modeling the cognitive processes underlying test performance is through the construction of a Q-matrix, which is essentially about summarizing the attributes explaining test takers’ response behavior. The current study documents the construction and validation of a Q-matrix for a high stakes test of reading within a generalized-deterministic inputs, noisy “and” gate (G-DINA) model framework. To this end, the attributes underlying the 20 items of the reading comprehension test were specified through retrospective verbal reports and domain experts’ Delphi techniques. In the ensuing stage, the Q-matrix thus developed along with item response data of 2625 test-takers were subjected to empirical analysis using the procedure suggested by de la Torre and Chiu (2016). Item-level results showed that, except for one item, the processes underlying the other items were captured by compensatory and additive models. This finding has significant implications for model selection for DCM practitioners.

The documentation of test takers’ achievements has been accomplished through large-scale assessments to find general information about students’ language ability. To remove subjectivity, Cognitive Diagnostic Assessment (CDA) has recently played a crucial role in perceiving candidates’ latent attribute patterns to find multi-diagnostic information rather than single proficiency classification. However, there are some gaps in the literature about in detail investigation of test takers’ listening comprehension language ability in responding to placement test items of a public English language center. The present study aims to validate an English placement test at a language center through a retrofitting process. In an exploratory mixed-method design, 449 participants from the same language center, including 274 females and 175 males, were selected. The performance of randomly selected participants in a language center placement test was analyzed by applying the GDINA model from R-studio packages, to detect Differential Item Functioning (DIF). Results of the study revealed DIF in some items since there is some bias in test items. The implication of this study is to provide meaningful interpretations of respondents’ attributes and improve teaching and learning by finding the strengths and weaknesses of candidates. For this purpose, the findings derived from the result of the study can raise the awareness of test developers in preparing unbiased items for the placement test, and at the same time, assist test-takers to become more critical of their English language achievements. It is also helpful for materials developers to become aware of developing materials free from bias.

The role of learning styles in academic performance has long been the question of many educationalists. Sensory learning styles, which categorize learners into three groups of visual, auditory, and tactile students, have been said to be likely to play parts in academic performance. The purpose of the present study is twofold. Initially, this study aimed to see what weaknesses Iranian university students have in reading comprehension task. The next step was to seek if possessing different sensory learning styles can lead to a significant difference regarding reading comprehension performance. In this study, Cognitive Diagnostic Assessment was applied to provide us with comprehensive mastery reading subskill profiles of everyone. To do so, a reading comprehension test along with a learning style questionnaire were given to 301 Iranian university students, the responses were all divided into either correct or incorrect responses, and according to examinees’ questionnaire, they were categorized into three groups of visual, auditory, and tactile learners. According to the present study, Iranian university students were found to have difficulty dealing with implicitly stated information, understanding difficult vocabulary, and summarizing the textual information. Regarding the second question of this study, visual learners performed significantly better than their auditory counterparts in four skills of Basic Linguistic Knowledge, Implicitly Stated Information, Understanding Difficult Vocabulary, and Understanding Complex Text. However, no significant difference was found between auditory and tactile participants. This result reinforces the prominent role of learning styles in academic and educational settings, to develop efficient instructions and curriculums that best meet learners’ needs.