Subjects: Psychology >> Psychological Measurement submitted time 2023-04-22
Abstract： Test mode effect (TME) refers to the difference in test function caused by the administration of the same test in different test modes. The existence of TME will have an impact on test fairness, selection criteria and test equating, so it is of great significance to accurately detect and interpret TME. By systematically sorting out the source, detection (including the experimental design and detection methods) and research results of TME, the methodology of TME research is comprehensively demonstrated. Further interpretation of the TME model, expansion of the test modes in TME research, and application of TME research results to largescale educational assessment programs in China, are important future development directions in the field of TME.
Subjects: Psychology >> Social Psychology submitted time 2023-03-28 Cooperative journals: 《心理科学进展》
Abstract： Computerized classification testing (CCT) can adaptively classify test-takers into two or more different categories, and it has been widely used in qualifying tests and clinical psychology or medical diagnosis. As an essential part of CCT, the termination rule determines when the test is to be stopped and to which category the test-taker is ultimately classified into, directly affecting the test efficiency and classification accuracy. According to the theoretical basis of the termination rules, existing rules can be roughly divided into the likelihood ratio, Bayesian decision theory, and confidence interval rules. And their core ideas are constructing hypothesis tests, designing loss functions, and comparing the relative positions of confidence intervals, respectively. At the same time, when constructing specific termination rules, the requirement of different test scenarios (e.g., the number of categories and the number of tests’ dimensions) should also be considered. There are advantages and disadvantages to each of the three types of termination rules. Specifically, the likelihood ratio rule is based on the likelihood ratio test, with better theoretical properties. However, the method requires prior determination of the indifference interval and the type I and II error rates, introducing the impact of subjective factors. Also, it is more challenging to extend the method in complex test situations, such as multidimensional and multicategory CCT. Bayesian decision theory rules make classification decisions based on the loss function. It can dynamically optimize the decision from a more global perspective since it works backward from the final stage of the test. In addition, the variety of loss functions makes the method very flexible in form and makes it easy to be applied to different test situations. However, in practice, the flexibility will inevitably result in the uncertainty of the choice of loss function, and the inappropriate loss function may be biased. The confidence interval method is the most straightforward because of its relatively simple principle and low computational effort. However, this method is less robust and has a relatively low test efficiency. Currently, CCT is mainly applied in eligibility tests and clinical medicine questionnaires. In eligibility tests, all three types of termination rules have the potential to be widely applied. However, in practice, the principles of the likelihood ratio rule and the Bayesian decision theory rule are not easily understood by the general public, and these methods are also accompanied by the problem of over-exposure of items for their preference of cut-point based item selection methods. Therefore, the confidence interval rule, which is relatively simple in principle and has alleviated item exposure, has been widely used in existing qualifying tests. Bayesian decision theory rules are more applicable in clinical questionnaires because of their finer control over various classification losses. The following can be considered for future research on CCT termination rules. First, Bayesian decision theory rules can be improved by considering non-statistical constraints with the help of the flexibility of its loss function. Second, termination rules can be developed for multidimensional and multicategory CCT to meet more practical needs. Third, termination rules that integrate response time can be developed to improve test efficiency and classification accuracy. Fourth, it is possible to construct termination rules under the framework of machine learning.
Subjects: Psychology >> Psychological Measurement submitted time 2021-11-16
Abstract： Computerized classification testing (CCT) has been widely used in eligibility testing and clinical psychology since it can efficiently classify participants. As an essential part of CCT, the termination rule determines when the test is to be stopped and what category the participants are ultimately classified into, directly affecting the test efficiency and classification accuracy. The existing termination rules can be roughly divided into the likelihood ratio, Bayesian decision theory, and confidence interval rules. And their core ideas are constructing hypothesis tests, designing loss functions, and comparing the relative positions of confidence intervals, respectively. Based on these ideas, in different test situations, CCT termination rules have various specific forms. Future research can further extend Bayesian rules, construct rules for multicategory MCCT, integrate process data into termination rules, and build rules under the framework of machine learning. In addition, from the perspective of practical requirement, all three types of rules have the potential to be applied in the eligibility test, while the clinical questionnaire tends to choose Bayesian rules.
Peer Review Status:
Awaiting Review
Hits
YES
Hits
Downloads
Comment