Vision-Based Tactile Sensing for the Perception of the Object's Compliance and Hardness

Abstract

Object compliance perception enables the identification of soft materials, supporting tasks such as fruit detection and assisted medical palpation. Compliance perception requires sensing an object's deformation and contact forces. Existing vision-based tactile sensing for compliance perception usually depends only on force or deformation. However, deformation-based approaches cannot reliably quantify object compliance, while force-based methods are overly sensitive to geometric variations. To address these limitations, this paper presents a framework that fuses temporal force sequences and deformation field information. Specifically, time-varying contact forces are inferred from tactile image sequences via a neural network, while deformation characteristics are encoded using depth maps. Crucially, the temporal force sequence is used as a dynamic force-response feature for compliance recognition. In standard experiments, the force prediction error reaches 0.06 N within a measurement range of 12 N. The proposed method achieves a 98.0% Shore-hardness classification accuracy for samples ranging from 10 HA to 80 HA. In practical scenarios, including abnormal fruit detection and soft matter, the overall accuracy exceeds 98.5%. This method improves the compliance perception ability of artificial tactile systems and facilitates their deployment in embodied perception applications.