Subjects: Psychology >> Developmental Psychology submitted time 2023-03-28 Cooperative journals: 《心理科学进展》
Abstract： Full-information item bifactor analysis is an important statistical method in psychological and educational measurement, which can be seen as a rediscovery of the classical bifactor model and has seen wide applications in the past two decades. The item response model of full-information item bifactor analysis is described upon the introduction of the conception and characterization of full-information item bifactor analysis. Further, we introduced the dimension reduction method used in parameter estimation. Then, examples are provided for applications of the full-information item bifactor model in test measurement structure exploration or confirmation, score interpretation, and computerized adaptive testing. The measurement structure of full-information item bifactor analysis is accordance with most tests in the areas of psychology, education, and medical science. With the advantage in dimension reduction, it is believed that the full-information item bifactor analysis could be valuable and useful in various situations. At last, some future research directions and suggestions are put forward including parameter estimation, test linking, differential item functioning, model fit testing, and application of bifactor item response theory to computerized adaptive testing.
Subjects: Psychology >> Developmental Psychology submitted time 2023-03-28 Cooperative journals: 《心理科学进展》
Abstract： Dual-objective cognitive diagnostic computerized adaptive testing (CD-CAT), which considers knowledge status and ability simultaneously, has become more and more popular with the theoretical and practical development of CD-CAT. Item selection methods play a key role in CD-CAT. This paper systematically reviews existing item selection methods on traditional and dual-objective CD-CAT, and summarizes the types, characteristics, relations, and performance of these methods. Furthermore, several future research directions were illustrated. First, it is necessary to study item selection strategy with general cognitive models and under complex test conditions. Second, it is important to develop indexes representing items and test characteristic of dual-objective diagnostic testing. Finally, it is meaningful to conduct research on non-parametric item selection methods and practical applications of CD-CAT.
Subjects: Psychology >> Social Psychology submitted time 2023-03-28 Cooperative journals: 《心理科学进展》
Abstract： The Q-matrix, which represents important item characteristics by mapping attributes to items has been proved to be the core factor affecting the accuracy of cognitive diagnostic classification. It is of great value to study the methods of Q-matrix estimation (validation). First, the existing methods of Q-matrix estimation and validation are classified into 1) parameterized methods in the CDM perspective, including item differentiation, model-data fit index and parameter estimation; and 2) non-parametric methods in the statistical perspective, including the distance between observed and expected response vector, abnormal responses index and factor analysis. Then, these methods are introduced in terms of differences and relations, characteristics and performance. The advantages and disadvantages of each method are commented. At last, several future research directions are proposed. It is necessary to compare the Q-matrix estimation (validation) methods systematically under complex test conditions. It is also of vital importance to propose Q-matrix estimation (validation) methods by combining multiple thoughts and ways based on the calibration of knowledge state and parameter estimation error. It is meaningful to further study the Q-matrix estimation (validation) methods for polytomous scoring items, mixed test models, polytomous scoring attributes, unknown number of attributes and even continuous Q-matrix.
Subjects: Psychology >> Social Psychology submitted time 2023-03-27 Cooperative journals: 《心理学报》
Abstract： Cognitive diagnostic theory (CDT) can provide fine-grained and multidimensional process assessment results, which has important research and practical values. The Q-matrix that represents the relationship between items and attributes, is the basis of CDT. The accuracy of the Q-matrix is an important factor that affects the accuracy of items parameter estimation and participants’ diagnosis. Therefore, it is of great significance to check the correctness of the Q-matrix or to validate it. A lot of studies have been carried out on the estimation or validation of Q-matrix, and a variety of methods have been proposed from different perspectives, each having their advantages and disadvantages. The methods based on model-data fit can provide rich test information without the need of complex parameter estimation and time-consuming and tedious calculation. Following this line of thinking, this study used Gini coefficient to express the purity of expected numbers proportion distribution, and constructed a simple and efficient Q-matrix validation method, called the optimization of response distribution purity (ORDP) method, which is suitable for both simplified model and saturated model. Residual index (R), root mean square error approximate (RMSEA) and hamming distance (HD) were compared to evaluate the performances with varied influencing factors, under the conditions of two different distribution of knowledge states (KS) (uniform distribution, multidimensional normal distribution), two different sample sizes (300, 1000), two different test lengths (20, 30), Q-matrix error rates (20%, 40%), item qualities ([0.05, 0.25], [0.05, 0.24]) and attribute hierarchical structures (independent structure, linear structure, convergent structure, and branched structure). The specific algorithm of Q-matrix validation is as follows. Firstly, the initial Q-matrix is represented by Q0. When validating the first item j, the initial q-vector of item j in Q0 is replaced with one of all possible q-vectors, leaving the rest of the items intact. Then, the EM algorithm is used to estimate the item parameters and the knowledge states of the participants. Lastly, the q-vector that minimizes ORDP, R, RMSEA, or HD for the q-vector of the item is selected. Simulation results demonstrate that: (1) The distribution of KS affects the performance of each method. Specifically, when the KS is uniformly distributed, ORDP method is superior to other methods, HD method is the next, followed by RMSEA and R methods; When the KS follows multivariate normal distribution, there is no significant difference between RMSEA and ORDP. RMSEA method is slightly better than ORDP method except independent structure, followed by HD and R method; (2) The validation effect of these methods under multivariate normal distribution is not as good as that under uniform distribution; (3) The validation rates of the four methods all affected by sample sizes, test lengths, Q-matrix error rates, item qualities and attribute hierarchical structures. If the smaller the number of respondents, the shorter the test length, the higher the Q-matrix error rates, or the lower the item quality, the worse the performance of each method will be, and vice versa; (4) The validation results based on the fractional subtraction data of Tatsuoka (1984) show that the Q-matrix modified by ORDP method has the best model-data fit. In this study, the ORDP index representing the purity of the expected numbers proportion distribution was constructed based on the Gini coefficient. Simulation and empirical studies show that this method has a high validation rate for Q-matrices under different conditions. On the whole, the new method proposed in this study validates the Q-matrix through data analysis, which can reduce the workload of experts and thus improve the correctness of the Q-matrix.
Subjects: Psychology >> Psychological Measurement submitted time 2020-08-19
Abstract： Dual-objective cognitive diagnostic computerized adaptive testing (CD-CAT), which considers knowledge status and ability simultaneously, has become more and more popular with the theoretical and practical development of CD-CAT. Item selection methods play a key role in CD-CAT. This paper systematically reviews existing item selection methods on traditional and dual-objective CD-CAT,and summarizes the types, characteristics,relations,and performance of these methods.Furthermore, several future research directions were illustrated. First, itis necessary to study item selection strategy with general cognitive models and under complex test conditions. Second, it is important to develop indexes representing items and test characteristic of dual-objective diagnostic testing. Finally, it is meaningful to conduct research on non-parametric item selection methods and practical applications of CD-CAT.
Subjects: Psychology >> Psychological Measurement submitted time 2020-08-19
Abstract： Item replenishment is essential for item bank development and maintenance, where new items’ parameter calibration plays a significant role. Two core techniques of item replenishment under the circumstances of computerized adaptive testing (CAT) are: 1) online calibration design; 2) online calibration method. The former investigates the administration way of new items, while the later explores parameter estimation methods. This paper aims to clarify the development ideas and contexts of online calibration design and online calibration method. Additionally, their characteristics, relations and performance were illustrated and evaluated in details. At the end, several future research directions were pointed out. It is important to further study online calibration design based on different information indicators and online calibration methods based on joint estimations and error corrections. Moreover, future study could explore the online calibration technique in cognitive diagnostic CAT(CD-CAT) and multidimensional CAT(MCAT), as well as the empirical applications of item replenishment. "
Hits
Hits
Downloads
Comment