Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.23530

EduWild: How Teacher-Configured Chatbots Fail in Real-World Use and How to Fix Them

Ziqi Liu·Ruiwei Xiao·Yuxuan Li·John Stamper
View original source ↗Read source document ↗Export metadata ↓

Abstract

Teachers increasingly configure educational chatbots through natural-language instructions, but how these bots fail in real-world use remains poorly understood. We introduce EduWild, a dataset of 1,726 conversations with 217 teacher-configured chatbots, each paired with the instructions and configuration behind it, including 900 conversations from the intended end users of 35 deployed chatbots. We develop a 29-code taxonomy covering factual, workflow, diagnostic, and pedagogical failures and apply it using a human-calibrated LLM judge. The most frequent failures are lapses in following teacher intent --- bots skip required workflow steps (53.8% of end-user conversations), omit expectation setup (33.4%), and overload learners with information or questions (37.1%). Guided by these findings, we design a prompt-based harness targeting the six most frequent failure modes. In a study with 200 college students across ten bots, the harness reduces the mean number of target failure codes per conversation from 1.406 to 0.489, a 65.2% relative reduction with bots weighted equally. These findings show how real-world failure analysis can guide practical improvements to educational chatbots.

Explore related subjects

cs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Ziqi Liu, Ruiwei Xiao, Yuxuan Li, John Stamper. 2026-10-16. EduWild: How Teacher-Configured Chatbots Fail in Real-World Use and How to Fix Them. https://arxiv.org/abs/2610.23530

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Enriching Patent Claim Generation with European Patent Dataset

Drafting patent claims is time-intensive, costly, and requires professional skill. Therefore, researchers have investigated large language models (LLMs) to assist inventors in writing claims. However, existing work has largely relied on datasets from the United States Patent and Trademark Office (USPTO). To enlarge research scope regarding various jurisdictions, drafting conventions, and legal standards, we introduce EPD, a European patent dataset. EPD presents rich textual data and structured metadata to support multiple patent-related tasks, including claim generation. This dataset enriches the field in three critical aspects: (1) Jurisdictional diversity: Patents from different offices vary in legal and drafting conventions. EPD fills a critical gap by providing a benchmark of European patents to enable more comprehensive evaluation. (2) Quality improvement: EPD offers high-quality granted patents with finalized and legally approved texts, whereas others consist of patent applications that are unexamined or provisional. Experiments show that LLMs fine-tuned on EPD significantly outperform those trained on previous datasets and even GPT-4o in claim quality and cross-domain generalization. (3) Real-world simulation: We propose a difficult subset of EPD to better reflect real-world challenges. Results reveal that all tested LLMs perform substantially worse on challenging samples, which highlights the need for future research.

cs.CL↗
arXiv2026

POSID: Target-Task-Unsupervised Person-Attribute Extraction from Heterogeneous Public-Safety Text

Person descriptions in public-safety text can be compact and regular, or distributed across heterogeneous narratives in which evidence boundaries and person ownership are ambiguous. We present POSID, a target-task-unsupervised framework that decomposes person-attribute extraction into candidate localization, provenance-controlled extraction scope, structured recovery and traversal, and person association, without supervised training or fine-tuning on the target task. We evaluate six localization configurations under two traversal policies across three real-use cohorts and two controlled synthetic diagnostics. On structurally regular descriptions, the strongest configurations reach 1.000/.972 Attr-Only (AO)/Attr-Value (AV) F1 on Compact500. Across the real-use cohorts, every RE-containing configuration benefits from conservative early stopping, while semantic-only retrieval can benefit from continued traversal when localization is less constrained. On the heterogeneous multi-person Challenge14 cohort, the best automatic AO and AV results are .500 and .382; replacing automatic candidates with verified evidence raises the frozen extractor to .522/.410 under early stopping, showing that localization is one of the material bottlenecks. We further introduce provenance-aware scoring for collective and underspecified evidence and a separate person-aware diagnostic for ownership and fragmentation. Together, the results establish a measurable operating regime for inspectable person-attribute extraction and distinguish errors due to localization, traversal, structured recovery, and attribution. Code: https://github.com/ksolaiman/hart-posid.

cs.CL↗
arXiv2026

Reasoning for Hierarchical Text Classification: The Case of Patents

Hierarchical text classification (HTC) assigns documents to multiple levels of a pre-defined taxonomy. Automated patent subject classification represents one of the hardest HTC scenarios because of professional difficulties and extensive labels. Prior approaches only output a flat label set, which offers little insight into the reason behind predictions. Therefore, we propose Reasoning for Hierarchical Classification (RHC), a novel framework that reformulates HTC as a step-by-step reasoning task to sequentially deduce hierarchical labels. RHC trains large language models (LLMs) in two stages: a cold-start stage that aligns outputs with chain-of-thought (CoT) reasoning format and a reinforcement learning (RL) stage to enhance multi-step reasoning ability. RHC demonstrates four advantages in our experiments. (1) Effectiveness: RHC surpasses previous baselines and outperforms the supervised fine-tuning counterparts by approximately 3% in accuracy and macro F1. (2) Explainability: RHC produces natural-language justifications before prediction to facilitate human inspection. (3) Scalability: RHC scales favorably with model size with larger gains compared to standard fine-tuning. (4) Applicability: Beyond patents, we further demonstrate that RHC achieves state-of-the-art performance on other widely used HTC benchmarks, which highlights its broad applicability.

cs.CL↗