Discounting and Drug Seeking in Biological Hierarchical Reinforcement Learning

Abstract

Despite a strong desire to quit, individuals with long-term substance use disorder (SUD) often struggle to resist drug use, even when aware of its harmful consequences. This disconnect between knowledge and compulsive behavior reflects a fundamental cognitive-behavioral conflict in addiction. Neurobiologically, differential cue-induced activity within striatal subregions, along with dopamine-mediated connectivity from the ventral to the dorsal striatum, contributes to compulsive drug-seeking. However, the functional mechanism linking these findings to behavioral conflict remains unclear. Another hallmark of addiction is temporal discounting: individuals with drug dependence exhibit steeper discount rates than non-users. Assuming the ventral-dorsal striatal organization reflects a gradient from cognitive to motor representations, addiction can be modeled within a hierarchical reinforcement learning (HRL) framework. However, integrating discounting into biologically grounded HRL remains an open challenge. In this work, we build on a model showing how action choices reinforced with drug rewards become insensitive to the negative consequences that follow. We address the integration of discounting by ensuring natural reward values converge across all levels in the HRL hierarchy, while drug rewards diverge due to their dopaminergic effects. Our results show that high discounting amplifies drug-seeking across the hierarchy, linking faster discounting with increased addiction severity and impulsivity. We demonstrate alignment with empirical findings on temporal discounting and propose testable predictions, establishing addiction as a disorder of hierarchical decision-making.