Objective: Reliable and accurate measurement is fundamental to scientific progress; however, the dominant measurement practices in psychology, clinical psychology, and prediction research often lack rigor. Improving measures using Rasch Measurement Theory (RMT) offers advantages by fulfilling the key psychometric properties of unidimensionality, local independence of items, ordering of response categories, and invariance. Ordinal-level sum scores can be transformed into interval-level latent trait scores, thereby improving the measurement precision. However, the impact of using psychometrically advanced questionnaires with latent trait scores, as opposed to traditional sum scores, in predictive models is still unclear. This study evaluates whether using latent trait scores as predictors and outcomes, in accordance with RMT, improves predictive performance compared to using traditional sum scores when predicting treatment outcomes during psychological treatment.

Methods: Self-rated symptom data from three different questionnaires, collected over the first 4 weeks of psychological treatment from 6,464 patients undergoing a 12-week treatment program, were used to predict post-treatment outcomes on the same questionnaires. This was done in two ways: (1) using sum scores as the questionnaires were originally developed and (2) using a reformulated, more psychometrically robust version of the questionnaires based on Rasch analysis, which was also shorter. The prediction models used were linear regression, Bayesian ridge regression, and random forest. Multiple imputations were used to address missing data, and nested cross-validation was employed for hyperparameter tuning and scoring.

Results: Latent scores calculated using the psychometrically optimized shorter version, which comprises 23% of the full scale, showed similar predictive performance compared to the sum score of the full scale. Overall, there was a statistically significant but practically negligible difference of 0.007-0.008 in the root mean squared error (RMSE) when comparing the original sum score to the latent trait scores.

Conclusion: Initial findings comparing psychometrically improved questionnaires with the original ordinal sum scores within a predictive framework indicate that using latent trait scores derived from these improvements showed the predictive performance similar to the sum score of the full scale. The small differences suggest that the improved versions remain valuable owing to their enhanced psychometric qualities and the reduction in response burden by using considerably fewer items. Further research is needed to explore the use of latent trait scores compared to ordinal sum scores in predictive research.