Assessment of real-world evidence research competencies in federated research networks: a maternal health fellowship program evaluation

Abstract Background and Significance Generating real-world evidence (RWE) approaches across federated evidence networks based on real-world data requires learning new methodological skills. However, standardized frameworks for defining and assessing competencies for large-scale RWE research remain limited. This study aims to develop an assessment framework for RWE studies and evaluate the competencies of RWE research professionals. Methods We adapted the Joint Task Force (JTF) Core Competency Framework for RWE research using the Observational Medical Outcomes Partnership (OMOP) Common Data Model. Through iterative expert review, 4 new domains were created, including Ethics and Governance, Protocol Development, Study Operations, and Data Analysis and Informatics, and 4 original JTF domains were adapted. Participants completed pre- and post-training self-assessments on a 10-point scale. Paired t-tests with effect sizes were used to evaluate changes in competency scores. Results Seventeen participants from academic, data, and clinical roles completed the assessment. Mean self-reported competency scores increased by 2.5 points across the 8 competency domains, with statistically significant changes observed in all domains. The largest gain was observed in Leadership and Professionalism (mean increase 3.66; 95% CI, 2.69-4.62). Discussion The RWE research competency assessment tool captured changes in self-reported skill proficiency, perceived readiness to conduct RWE studies, and methodological knowledge gaps. These findings suggest the potential to identify competency gaps, tailor educational interventions, and inform quality benchmarks for RWE research training programs. Conclusion This study developed and evaluated a competency assessment framework specifically designed for RWE research using standardized healthcare data, providing a tool for training program evaluation and workforce development support.