Cognitive diagnostic assessment as an alternative measurement model Vahid Aryadoust (National Institute of Education, Singapore) |
"A great asset of CDA is that it does not require unidimensionality. The unidimensionality precondition seems particularly problematic in the language assessment field because most measurement models force the test data to be unidimensionally asymptotic . . ." |
[ p. 2 ]
[ p. 3 ]
Q = {q_{i}k}. When sub-skill k is required by item 1, then q_{i}k = 1, and when the sub-skill is not required, then q_{i}k = 0. Table 1 represents a hypothetical Q-matrix:Items | Identified sub-skills | ||
a | b | c | |
1 | 0 | 0 | 1 |
2 | 1 | 1 | 0 |
3 | 1 | 0 | 0 |
4 | 0 | 1 | 1 |
X_{ij} = response of test taker j to test item i (0 = incorrect; 1 = correct);The sub-skills should be identified on the basis of "test specifications, content domain theories, analysis of item content, think-aloud protocol analysis of test takers' test taking process, and other relevant research results" (Lee & Sawaki, 2009, p. 176). An equally useful method of defining the Q-matrix is iterative runs of the FM to specify the matrix (Sawaki et al., 2009). That is, a panel of experts develops a few rival Q-matrices, whose specified sub-skills will have commonalities and points of departure depending on their decisions. The researcher should consider multiple factors to partial out rival Q-matrices and retain the best fitting matrix.
α_{j} = vector of sub-skill mastery (if test taker j has mastered sub-skill k, then α_{jk} = 1, and if not, then α_{jk} = 0);
θ_{j} = overall ability of test taker j, which is not specified by the Q-matrix; unlike the ability parameters of IRT, which have continuous θ indices (i.e., data that can hold any value, ranging from minus infinity to infinity), the θ_{j} index in the FM is a categorical parameter (i.e., data that can only take certain values) (Lee & Sawaki, 2009); -∞ < θ_{j} < +∞;
Π^{*}i = probability of correctly applying the required sub-skills in answering the i^{th} item when the test taker has mastered all relevant sub-skills, or the difficulty of item i according to the Q-matrix; this index ranges from 0 to 1;
R^{*}_{ik} = , the discrimination parameter of item i and skill k; it ranges from 0 to 1. For each item i, there are k sub-skill values of r^{*}_{ik} and k_{i} is the number of sub-skills specified in the Q-matrix as being required to answer item i correctly;
q_{ik} = specification of mastery of sub-skill k which is required to answer item i;
c_{i} = the degree of reliance of item performance on c_{j} in addition to the sub-skills identified in the Q-matrix. This index ranges from 0 to 3. = probability of correctly applying the sub-skills which are not specifiedθ in the Q-matrix. This index is estimated though the Rasch model.
[ p. 4 ]
Among the aforementioned FM parameters Π^{*}i, r^{*}_{ik} and c_{i} have important roles because they not only provide diagnostic information about each test taker and item, but also highlight the properties of the Q-matrix and the misspecifications observed. An ideal matrix produces estimates r^{*}_{ik} below .90, indicating that the item discriminates masters from non-masters sufficiently; values below 0.50 are regarded as sub-skills highly necessary to answer the question correctly (Roussos, Xueli, & Stout, 2003). In addition, high Π^{*}i indices are desirable, as they indicate that masters have a higher probability of successfully applying the sub-skills required by that item. The parameter c_{i} is a "completeness index" ranging from 0 to 3 (Montero, Monfils, Wang, Yen, & Julian, 2003). It will approach 3 if the sub-skills required to successfully answer the item are fully specified in the Q-matrix, and 0 if they are not specified in the matrix.[ p. 5 ]
Jang, E. E. (2008a). A framework for cognitive diagnostic assessment. In C. A. Chapelle, Y._R. Chung, & J. Xu (Eds.), Towards adaptive CALL: Natural language processing for diagnostic language assessment (pp. 117_131). Ames, IA: Iowa State University.[ p. 6 ]