Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer

Abstract

Measuring subjective constructs in naturally occurring social media text requires annotation procedures that are theoretically grounded, empirically validated, and transferable to an encoder model for scalable prediction. Using posts annotated according to Schwartz's theory of basic human values, we investigate how different LLMs and prompting strategies, which we call annotation regimes, operationalize the expression of values in text. Beyond standard classification metrics, we evaluate structural alignment, annotation ambiguity, error patterns, and stability across repeated runs. We find that different LLMs produce different value interpretations, and iterative prompt calibration through error analysis reduces misattributions and improves alignment with expert annotations. Error patterns are further used to derive targeted expert-verification rules for corpus annotation. We transfer soft LLM labels to an encoder model for prediction, retaining information about ambiguity in value expression. Finally, a sensitivity analysis on more than one million posts shows that regime-specific annotation differences propagate into predicted levels of value expression, whereas standardized temporal dynamics and the direction of major event responses are more robust.