Adaptive Question Generation for Ambiguous Robot Instructions Based on Information Gain and Human Cognitive Characteristics

Ryo Terashima; Yuga Yano; Hakaru Tamukoh

Home/Archive/arXiv

arXiv · 2610.23874

Adaptive Question Generation for Ambiguous Robot Instructions Based on Information Gain and Human Cognitive Characteristics

Ryo Terashima·Yuga Yano·Hakaru Tamukoh

View original source ↗Read source document ↗Export metadata ↓

Abstract

In this study, we propose an adaptive question generation mechanism that addresses uncertainty arising from ambiguous instructions. Natural-language instructions for service robots often contain ambiguities. In scenes with multiple similar objects, the robot must identify the target object by asking questions. However, existing question generation methods focus on probabilistic discrimination efficiency and do not sufficiently account for user ease of response. Asking questions without considering the user's ease of response may cause incorrect or uncertain answers. Such incorrect or uncertain answers lead to additional clarification turns and degraded dialogue efficiency. The proposed mechanism integrates probabilistic discrimination efficiency and human cognitive characteristics to generate questions that users can intuitively and easily answer. First, the proposed mechanism extracts visual attributes of candidate objects using a vision-language model (VLM). Next, the proposed mechanism computes the information gain (IG) of each attribute and selects the visual attribute (e.g., color, shape) that most reduces uncertainty in the candidate set. When multiple attributes have the same IG, the proposed mechanism applies a visual attribute filter based on cognitive characteristics to select the visual attribute that users can most intuitively perceive. The proposed mechanism then prompts the VLM with the selected attribute and the corresponding attribute values of each candidate object to generate a concise natural-language question. This process is repeated until the target object is uniquely identified. The proposed mechanism achieved a target identification success rate of 88.5% in the evaluation using a dataset of 120 scenes. Furthermore, the proposed mechanism received high ratings in subjective evaluations of response ease.

Explore related subjects

cs.RO

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Ryo Terashima, Yuga Yano, Hakaru Tamukoh. 2026-09-19. Adaptive Question Generation for Ambiguous Robot Instructions Based on Information Gain and Human Cognitive Characteristics. https://arxiv.org/abs/2610.23874

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Synergistic Action and Style Priors for Learning Legged Locomotion

Deep Reinforcement Learning (DRL) enables robots to learn complex behaviors through interaction with the environment. However, due to the unrestricted nature of the learning algorithms, the resulting solutions are often brittle and appear unnatural. This is especially true for learning direct joint-level torque control, as inductive biases are difficult to integrate into the learning process. We propose an inductive bias for learning locomotion that is especially useful for torque control: latent actions learned from a small dataset of expert demonstrations. This prior allows the policy to directly leverage knowledge contained in the expert's actions and facilitates more efficient exploration. We observe that the agent is not restricted to the reward levels of the demonstration, and performance in transfer tasks is improved significantly. Latent action priors combined with style rewards for imitation lead to a closer replication of the expert's behavior. Videos are available at https://sites.google.com/view/latent-action-priors.

cs.RO↗

arXiv2026

PAKE: Learning Whole-Body Loco-Manipulation with Partial Kinematic Embeddings

Loco-manipulation has recently shown promising capabilities; however, achieving high-precision control, managing the high-dimensional action space induced by many degrees of freedom (DoFs), and fully exploiting the inherent redundancy of whole-body systems remain challenging. In this paper, we propose a novel whole-body control framework that effectively addresses these challenges by decomposing the complex loco-manipulation problem into partial reference motion generation and low-level imitation control. We introduce a new Kinematic Normalizing Flow (KNF) model, trained on a large-scale kinematic dataset, that generates diverse partial kinematic references. A high-level controller is then trained to navigate the KNF's latent space to exploit redundant solutions, while a low-level controller learns dynamically consistent motion execution. We validate our approach on the quadrupedal robot equipped with a six-DoF robotic arm. In simulation, PAKE outperforms the evaluated baselines in tracking accuracy and feasible workspace coverage on the 4,000-target benchmark. For hardware deployment, we evaluate the system over 24 episodes across 8 different mobile loco-manipulation tasks. The system achieves end-effector pose-tracking errors of 4.5 cm and 0.14 rad, while maintaining accurate locomotion tracking with linear and angular velocity errors of 0.1 m/s and 0.01 rad/s, respectively. Our method represents a practical and powerful solution for accurate and generalized whole-body loco-manipulation in high-DoF robotic systems, with promising potential for diverse downstream robotic tasks.

cs.RO↗

arXiv2026

Learning Where it Matters: Autonomous Data Collection and Offline Reinforcement Learning for Local Contact-Rich Manipulation

Learned policies trained end-to-end on large datasets often remain brittle in high-precision contact-rich tasks. Further, they typically require large amounts of manually collected data and still struggle to generalize to scenes outside of the training scenarios. Classical motion planning, in turn, generalizes well but lacks the precision and contact awareness for tight-tolerance tasks. To reliably solve challenging contact-rich tasks, we propose to leverage robot data and learning where it matters in a modular approach. First, we introduce an autonomous data collection scheme for the real world that collects dense data locally in the contact-rich segment of the task, eliminating reliance on a teleoperator's skill. Second, we show that using offline RL, the collected data can be translated into policies with high success rates. We then integrate our local policies with traditional motion planning to solve long-horizon assembly tasks. Across four real-world tasks, using only 2-2.5 h of autonomous data collection, we achieve an average success rate of 96% for long-horizon tasks, compared to 55% for a planning-only baseline and 37% for the strongest end-to-end baseline. Our local policies also reduce applied forces by 49%. Notably, performance remains at 90% for scene-level out-of-distribution scenarios, while end-to-end approaches struggle. In total, we run over 1500 real-world evaluation rollouts. Our results pave the way for targeted data collection in contact-rich tasks and for high success rates in precision applications. Videos and qualitative results are available at https://learnsyslab.github.io/data_and_learning_where_it_matters/

cs.RO↗