HintEval: An Open-Source Python Toolkit for Hint Generation and Hint Evaluation

Abstract

Large Language Models (LLMs) increasingly provide direct answers to user questions, raising concerns about reduced engagement in critical thinking and problem-solving. Hint generation offers an alternative by guiding users toward answers without revealing them, while hint evaluation assesses the quality of such guidance. Research in this area is hindered by fragmented datasets, inconsistent annotation formats, and evaluation tools that are often dataset-specific or unavailable. To address these challenges, we introduce HintEval, an open-source Python library for unified hint generation and evaluation. HintEval standardizes access to diverse hint datasets, supports answer-aware and answer-agnostic generation methods, and implements multiple evaluation metrics within a shared data model. The toolkit enables reproducible experimentation, cross-dataset analysis, and multi-dimensional evaluation with minimal engineering effort. We further demonstrate its utility through human studies in which participants assess generated hints and use them to answer questions, showing that hints can effectively support users in reaching correct answers. HintEval is accompanied by comprehensive documentation, an executable Google Colab notebook for rapid experimentation, and a demonstration video. By promoting consistent evaluation practices and lowering barriers to entry, it facilitates systematic research on hint-based question answering (QA) in NLP and IR.