Ryan L Boyd | Search arXiv

arXiv2026

What Users Cannot See: Evaluating LLM Emotional Support Beyond User Preference

People increasingly turn to LLMs for emotional support, yet common evaluations reward responses that feel helpful and may miss consequential response behaviors. We introduce a theory-informed measurement framework that decomposes LLM emotional-support responses into Soothe (affective comfort), Reframe (cognitive perspective-shift), and Endorse (agreement with a user's causal or moral framing). Across 9,000 GPT-5.6 responses to 3,000 venting and advice-seeking Reddit posts, friend- and therapist-style personas both increased Soothe relative to default but moved Reframe and Endorse in opposite directions: friend prompting increased Endorse and reduced Reframe, a pattern that could reinforce users' existing appraisals and plausibly contribute to escalation risk, whereas therapist prompting did the reverse. Two independent LLM judges substantially agreed with clinically trained raters, yet lay raters detected only 60% of expert-confirmed Endorse instances and rated the personas similarly helpful and desirable. Preference-based evaluation thus has a blind spot: supportive presentation can obscure appraisal-reinforcing, potentially escalatory behavior.

cs.HC↗

arXiv2026

From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP

While NLP typically treats documents as independent and unordered samples, in longitudinal studies, this assumption rarely holds: documents are nested within authors and ordered in time, forming person-indexed, time-ordered $\textit{behavioral sequences}$. Here, we demonstrate the need for and propose a longitudinal modeling and evaluation paradigm that consequently updates four parts of the NLP pipeline: (1) evaluation splits aligned to generalization over people ($\textit{cross-sectional}$) and/or time ($\textit{prospective}$); (2) accuracy metrics separating between-person differences from within-person dynamics; (3) sequence inputs to incorporate history by default; and (4) model internals that support different $\textit{coarseness}$ of latent state over histories (pooled summaries, explicit dynamics, or interaction-based models). We demonstrate the issues ensued by traditional pipeline and our proposed improvements on a dataset of 17k daily diary transcripts paired with PTSD symptom severity from 238 participants, finding that traditional document-level evaluation can yield substantially different and sometimes reversed conclusions compared to our ecologically valid modeling and evaluation. We tie our results to a broader discussion motivating a shift from word-sequence evaluation toward $\textit{behavior-sequence}$ paradigms for NLP.

cs.CL↗