Deletion of the 5-HT3A receptor reduces behavioral persistence and enhances flexibility

Abstract The 5-HT3A receptor is the only ionotropic serotonin receptor and has been implicated in cognitive functions, yet its specific role remains unclear. To investigate the contribution of the 5-HT3A subtype, we trained wild-type (C57BL/6J) and 5-HT3A receptor knockout (Htr3a −/− ) mice across a series of operant conditioning tasks and compared their behavioral performance. Following nose-poke training, both groups underwent a rule-switching task, extinction tests under fixed ratio (FR) and variable ratio (VR) reinforcement schedules, and a progressive ratio (PR) task to assess persistence. We found that 5-HT3A receptor knockout mice exhibited reduced responding during the extinction and PR tasks, suggesting diminished behavioral persistence. Notably, however, knockout mice acquired the new rule in the switching task significantly faster than wild-type controls, indicating enhanced cognitive flexibility. These findings suggest that the 5-HT3A receptor plays a role in regulating the balance between behavioral persistence and flexibility, normally biasing this balance toward persistence under normal physiological conditions. This mechanism may underlie the therapeutic effect of 5-HT3A receptor antagonists in treatment-resistant obsessive-compulsive disorder (OCD). Highlights Cognitive abilities of 5-HT3A receptor-deficient (Htr3a−/−) mice were examined using an operant conditioning paradigm. 5-HT3A receptor knockout mice exhibited reduced responding during extinction and progressive ratio (PR) tasks, indicating diminished behavioral persistence. 5-HT3A receptor knockout mice showed more rapid acquisition in a rule-switching task compared to wild-type mice, suggesting enhanced flexibility. The 5-HT3A receptor contributes to the balance between behavioral persistence and flexibility. This function may underlie the therapeutic effects of 5-HT3A receptor antagonists in treatment-resistant obsessive-compulsive disorder (OCD).