Empowering psychometrics with generative large language models: Advantages, challenges, and applications

Abstract: Generative Large Language Models (LLMs), a class of artificial intelligence models pre-trained on vast corpora of textual data, present unprecedented opportunities and challenges for the field of psychometrics. This paper synthesizes the developmental trajectory of interdisciplinary research between AI and psychology to summarize the significant advantages of LLMs in empowering psychometrics, identify key challenges in their application, and propose future research directions. Specifically, LLMs’ ability to generate coherent, context-aware natural language text has the potential to transform traditional assessment interaction paradigms. Their advanced capabilities in processing extensive texts and multimodal data allow for the comprehensive capture and analysis of participants’ psychological information. Furthermore, LLMs facilitate real-time analysis and personalized feedback, promoting a shift from outcome-based to process-oriented evaluation. Despite facing practical challenges related to stability, creativity, and scalability, LLMs demonstrate substantial promise in applications such as Situational Judgment Test generation, collaborative problem-solving assessment, intelligent mental health diagnostics, and test item quality analysis.