Subjects: Psychology >> Psychological Measurement submitted time 2023-12-21
Abstract： Situational Judgment Tests (SJTs) have gained popularity for their unique testing content and high face validity. However, traditional SJT formats, particularly those employing multiple-choice (MC) options, have encountered scrutiny due to their susceptibility to test-taking strategies. In contrast, open-ended and constructed response (CR) formats present a propitious means to address this issue. Nevertheless, their extensive adoption encounters hurdles primarily stemming from the financial implications associated with manual scoring. In response to this challenge, we propose an open-ended SJT employing a written-constructed response format for the assessment of teacher competency. This study established a scoring framework leveraging natural language processing (NLP) technology to automate the assessment of response texts, subsequently subjecting the system's validity to rigorous evaluation. The study constructed a comprehensive teacher competency model encompassing four distinct dimensions: student-oriented, problem-solving, emotional intelligence, and achievement motivation. Additionally, an open-ended situational judgment test was developed to gauge teachers' aptitude in addressing typical teaching dilemmas. A dataset comprising responses from 627 primary and secondary school teachers was collected, with manual scoring based on predefined criteria applied to 6,000 response texts from 300 participants. To expedite the scoring process, supervised learning strategies were employed, facilitating the categorization of responses at both the document and sentence levels. Various deep learning models, including the convolutional neural network (CNN), recurrent neural network (RNN), long short-term memory (LSTM), C-LSTM, RNN+attention, and LSTM+attention, were implemented and subsequently compared, thereby assessing the concordance between human and machine scoring. The validity of automatic scoring was also verified.
This study reveals that the open-ended situational judgment test exhibited an impressive Cronbach's alpha coefficient of 0.91 and demonstrated a good fit in the validation factor analysis through the use of Mplus. Criterion-related validity was assessed, revealing significant correlations between test results and various educational facets, including instructional design, classroom evaluation, homework design, job satisfaction, and teaching philosophy. Among the diverse machine scoring models evaluated, CNNs have emerged as the top-performing model, boasting a scoring accuracy ranging from 70% to 88%, coupled with a remarkable degree of consistency with expert scores (r= 0.95, QWK=0.82). The correlation coefficients between human and computer ratings for the four dimensions—student-oriented, problem-solving, emotional intelligence, and achievement motivation—approximated 0.9. Furthermore, the model showcased an elevated level of predictive accuracy when applied to new text datasets, serving as compelling evidence of its robust generalization capabilities.
This study ventured into the realm of automated scoring for open-ended situational judgment tests, employing rigorous psychometric methodologies. To affirm its validity, the study concentrated on a specific facet: the evaluation of teacher competency traits. Fine-grained scoring guidelines were formulated, and state-of-the-art NLP techniques were used for text feature recognition and classification. The primary findings of this investigation can be summarized as follows: (1) Open-ended SJTs can establish precise scoring criteria grounded in crucial behavioral response elements; (2) Sentence-level text classification outperforms document-level classification, with CNNs exhibiting remarkable accuracy in response categorization; and (3) The scoring model consistently delivers robust performance and demonstrates a remarkable degree of alignment with human scoring, thereby hinting at its potential to partially supplant manual scoring procedures.
Subjects: Psychology >> Other Disciplines of Psychology submitted time 2019-09-22
Abstract： Death awareness refers to thinking about and the recognition of the inevitability of personal death. As a critical component of the human-unique ability of autonoetic consciousness, death awareness can be viewed as a cognitive adaptation for time management. We hypothesize that activating death awareness may affect intertemporal choice, in which people make tradeoffs between rewards across different time points. Such effects of death awareness on intertemporal choice may be mediated by time perception, a subjective assessment of the speed of time passage. In this research, we investigate the impact of death awareness on time perception and intertemporal choice, and the relationships among them. Study 1 examined the relationship between death awareness and time estimation. Eighty-three college students were randomly assigned to either a death awareness activation group where mortality was made salient to the participants or a control group where the participants imagined their toothache experience. After a word-search distraction task, the participants in both groups completed a time-passage (400ms, 800ms, 1200ms, 1600ms) estimation task. The results showed that the participants in the group of death awareness activation gave significantly shorter estimates than the participants in the control group. Study 2 (n = 123) extended the measure of time perception to a more extended period and also measured the delay discounting rate of the participants from their intertemporal choices between a smaller-and-sooner reward and a larger-and-later reward. The participants were randomly assigned to either a death awareness activation group or a toothache awareness activation group. The participants then indicated how long ten years was to them by marking on a line with the statement “10 years is very short” on the left end side of the line and the statement “10 years is very long” on the right end side. The participants in the death-awareness activation group marked the line closer to the left end (“life is short”) than those in the control group. As predicted, the participants in the death-awareness activation group had a lower delay discounting rate and were more future-oriented in making intertemporal choices. Moreover, bootstrapping analysis revealed a partial mediation effect of time-passage estimation between death awareness and delay discounting. In conclusion, death awareness serves adaptive functions in time management. Activating death awareness makes people feel that time passes more quickly and promotes future-oriented decisions.
Hits
YES
Hits
Downloads
Comment