Search arXiv⌕ Search

arXiv subjects

Hokin Deng

Publications and source records attributed to Hokin Deng.

21 records · Page 2Linked to original sources
arXiv2025

Core Knowledge Deficits in Multi-Modal Language Models

While Multi-modal Large Language Models (MLLMs) demonstrate impressive abilities over high-level perception and reasoning, their robustness in the wild remains limited, often falling short on tasks that are intuitive and effortless for humans. We examine the hypothesis that these deficiencies stem from the absence of core knowledge--rudimentary cognitive abilities innate to humans from early childhood. To explore the core knowledge representation in MLLMs, we introduce CoreCognition, a large-scale benchmark encompassing 12 core knowledge concepts grounded in developmental cognitive science. We evaluate 230 models with 11 different prompts, leading to a total of 2,530 data points for analysis. Our experiments uncover four key findings, collectively demonstrating core knowledge deficits in MLLMs: they consistently underperform and show reduced, or even absent, scalability on low-level abilities relative to high-level ones. Finally, we propose Concept Hacking, a novel controlled evaluation method that reveals MLLMs fail to progress toward genuine core knowledge understanding, but instead rely on shortcut learning as they scale.

cs.CL↗
arXiv2024

Use-dependent Biases as Optimal Action under Information Bottleneck

Use-dependent bias is a phenomenon in human sensorimotor behavior whereby movements become biased towards previously repeated actions. Despite being well-documented, the reason why this phenomenon occurs is not yet clearly understood. Here, we propose that use-dependent biases can be understood as a rational strategy for movement under limitations on the capacity to process sensory information to guide motor output. We adopt an information-theoretic approach to characterize sensorimotor information processing and determine how behavior should be optimized given limitations to this capacity. We show that this theory naturally predicts the existence of use-dependent biases. Our framework also generates two further predictions. The first prediction relates to handedness. The dominant hand is associated with enhanced dexterity and reduced movement variability compared to the non-dominant hand, which we propose relates to a greater capacity for information processing in regions that control movement of the dominant hand. Consequently, the dominant hand should exhibit smaller use-dependent biases compared to the non-dominant hand. The second prediction relates to how use-dependent biases are affected by movement speed. When moving faster, it is more challenging to correct for initial movement errors online during the movement. This should exacerbate costs associated with initial directional error and, according to our theory, reduce the extent of use-dependent biases compared to slower movements, and vice versa. We show that these two empirical predictions, the handedness effect and the speed-dependent effect, are confirmed by experimental data.

q-bio.NC↗