Design of the polytomous simplest complete Q matrix based on the reachability matrix

Subjects: Psychology >> Psychological Measurement submitted time 2024-06-14

Tang Xiaojuan Peng Zhixia Qin Shanshan Ding Shuliang Mao Mengmeng Li Yu

Abstract： The identifiability of cognitive diagnosis models relies heavily on the completeness of the Q matrix. However, existing test designs primarily focus on dichotomously-scored items, neglecting the importance of polytomous cognitive diagnostic test design. Moreover, this limitation poses a significant obstacle to the advancement of cognitive diagnosis. To bridge this gap, this paper aimed to introduce novel designs for the construction of polytomous structured and unstructured simplest complete Q matrices (SSCQM/USCQM). Our proposed approach considered all ideal response patterns (IRPs) of knowledge states (KSs) on the reachability matrix as research objects, with the objective of minimizing the number of columns selected from the reachability matrix. This ensured one-to-one correspondence between the set of KSs and the set of IRPs, thereby enhancing the completeness of the SSCQM. Additionally, we derived a polytomous USCQM by considering the relationship between the SSCQM and the sub-matrix of the corresponding identity matrix while ensuring that each row contains at least one 1 . Interestingly, the construction process revealed that there were more USCQMs than SSCQMs. This innovative approach expanded the possibilities for polytomous cognitive diagnostic test design.
This study focused on the design and evaluation of cognitive diagnostic tests using polytomous structured and unstructured Q matrices (SSCQM/USCQM). We conducted two studies to comprehensively examine the influence of factors such as the number of attributes, attribute hierarchies, and item parameters on the precision of the SSCQM, USCQM, and reachability matrix. In the first study, variations in attribute structures and item parameter values were investigated to understand their impact on Q matrix accuracy. On the other hand, the second study explored the effects of attribute hierarchies and the number of attributes on the precision of the SSCQM, USCQM, and reachability matrix.
Both simulation studies and actual measurement data were utilized to assess the robustness and efficacy of the two methods. Firstly, the simulation results revealed several key findings. Notably, increasing the number of SSCQMs or USCQMs positively influenced the accuracy of the results. In the context of long tests, the USCQM demonstrated higher Pattern Match Ratio (PMR) and Marginal Match Ratio (MMR) compared to the SSCQM and the reachability matrix. This trend was particularly evident when there was an increase in item parameters, attribute numbers, or a change in attribute hierarchy. However, it is noteworthy that, regardless of these various factors, the PMR and MMR of the three tests exhibited minimal differences. On the other hand, in short tests with good item quality, the SSCQM achieved the best performance compared to other methods. This highlights the importance of considering specific test characteristics and item quality when selecting the appropriate Q matrix type. These findings provide valuable insights into the factors that influence the precision of Q matrices. They emphasize the benefits of increasing the number of matrices, understanding the impact of item parameters, and recognizing the performance disparities among different matrix types. Obtaining a comprehensive understanding of these relationships is vital for optimizing the design and implementation of cognitive diagnostic testing, ultimately guaranteeing accurate assessments of individual knowledge states. Secondly, analysis of the actual measurement data showed high identification repetition rates for the SSCQM and the reachability matrix, with a minimal difference in attribute mastery ratio.
 In summary, both the SSCQM and the USCQM demonstrate adequate performance when compared to other Q matrices under similar conditions. These findings emphasize the significance of prioritizing completeness in cognitive diagnostic testing. This research seeks to contribute to the advancement of cognitive diagnosis by addressing the limitations of existing test designs and introducing new techniques for constructing polytomous Q matrices. In addition, the findings presented in this paper offer valuable insights for researchers and practitioners seeking to design high-quality cognitive diagnostic tests that accurately assess individual knowledge states.

YES

 Hits 305  Downloads 81  Comment 0