Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study

Совершенствование подготовки по психиатрии с использованием имитационного инструмента консультации на основе агентного искусственного интеллекта: проспективное когортное исследование
Sanjeev Sockalingam, Guillaume Dumas, Venkat Bhat, Bill Kapralos, Adam Dubrowski, Alexandre Hudon, Mohammad Amin Kamaleddin, Lisa Burback, Andrew James Greenshaw, Sridhar Krishnan, Alice Rueda, Divya Sharma, Bazen Gashaw Teferra, Huda F. Al-Shamali, Zack Cote, Niloy Roy, Reinhard Janssen‐Aguilar, Jithin Joseph, Olga Winkler, Andrei Torres, Yanbo Zhang
2026-07-21

STACER assessmentagentic AIlarge language modelspsychiatry trainingsimulated patient encounters
Background: Canadian psychiatry residents must demonstrate consultation competency, assessed using the standardized assessment of a clinical encounter report (STACER). However, opportunities to practice these skills and receive constructive assessment remain limited in clinical settings. Objective: This study aimed to evaluate the technical feasibility of an agentic AI system designed to support psychiatry residents' consultation competence through simulated patient encounters with a patient agent and structured feedback from a rater agent. Methods: We conducted a two-phase technical feasibility prospective single-arm cohort study of the STACER Agentic System, a large language model-based platform integrating a patient agent and a rater agent. Phase 1 involved automated evaluation of the patient agent using a psychiatrist agent across 227 synthetic major depressive disorder cases. Performance was assessed using DeepEval metrics (correctness, clarity, medical faithfulness, turn relevance, and role adherence) with descriptive statistics and 95% CIs. Phase 2 involved a preliminary user study with 14 convenience-sampled participants: a total of 5 members of the clinical research team and 9 psychiatry residents from the University of Alberta. Participants completed simulated diagnostic interviews and case presentations. Performance was evaluated using STACER-based scoring by the rater agent and 2 psychiatrists. Interrater reliability was assessed using intraclass correlation coefficients (α=.05). Participants rated realism, behavioral consistency, psychiatric nuance, and feedback utility using Likert scales and free-text answers. Results: The patient agent demonstrated high behavioral (51/56, 91.07%) and symptom fidelity (105/110, 95.45%), with strong automated performance (medical faithfulness mean 0.99, 95% CI 0.99-1.00; turn relevance 0.99, 95% CI 0.986-0.992). Participants rated simulations as psychiatrically plausible and diagnostically useful, particularly for depressive symptom representation, although rapport building was moderate (mean 2.78, SD 1.56 to mean 3.00, SD 1.41, out of 5.00) due to limited nonverbal cues. The rater agent generated structured STACER-aligned feedback with high intrarater consistency, especially at the section subtotal level. Interrater reliability with psychiatrists was poor at the item level (intraclass correlation coefficient range=0.25-0.49) but improved to good-to-excellent agreement at the section level for psychiatry resident sessions (intraclass correlation coefficient range=0.89-0.93). The rater agent's scores fell between those of the 2 psychiatrists for the clinical research team and were lower than both human raters for psychiatry residents. Conclusions: The STACER Agentic System demonstrates the technical feasibility of using agentic AI to simulate psychiatric consultations and deliver STACER-aligned formative feedback. By combining adaptive multiturn psychiatric simulation with competency-based evaluation, it shows promise in supporting cognitive aspects of consultation, though it remains limited in facilitating relational skills such as rapport building. These findings suggest agentic AI could expand scalable, low-risk opportunities for deliberate practice and formative feedback in competency-based psychiatric education. Further controlled studies are needed to evaluate educational effectiveness and integration into residency training.
1
Across 227 synthetic major depressive disorder cases, the patient agent showed high behavioral fidelity (91.07%) and symptom fidelity (95.45%).
2
Automated evaluation indicated strong patient-agent medical faithfulness (mean 0.99) and turn relevance (mean 0.99).
3
In a preliminary study of 14 participants, simulations were rated psychiatrically plausible and diagnostically useful, supporting technical feasibility for consultation practice.
4
The STACER Agentic System integrates a simulated patient agent and rater agent to support psychiatry consultation training and structured feedback.
5
The abstract reports evaluation using STACER-based scores from an AI rater and two psychiatrists, with interrater reliability assessed by intraclass correlation coefficients.

The STACER Agentic System for simulated psychiatry consultations, including its patient and rater agents

The system’s technical feasibility and its ability to support psychiatry residents’ consultation competence through realistic simulated encounters and structured feedback

Publication Details
Publication Date
2026-07-21
Journal
Publisher
ISSN
Cited by
1
Access Type
Author Information
Authors
Sanjeev Sockalingam
Guillaume Dumas
Venkat Bhat
Bill Kapralos
Adam Dubrowski
Alexandre Hudon
Mohammad Amin Kamaleddin
Lisa Burback
Andrew James Greenshaw
Sridhar Krishnan
Alice Rueda
Divya Sharma
Bazen Gashaw Teferra
Huda F. Al-Shamali
Zack Cote
Niloy Roy
Reinhard Janssen‐Aguilar
Jithin Joseph
Olga Winkler
Andrei Torres
Yanbo Zhang
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%