Psychometric assessment of three newly developed implementation outcome measures

Психометрическая оценка трех недавно разработанных показателей результатов внедрения
Bryan Jeffrey Weiner, Byron J. Powell, Cara C. Lewis, Cameo Stanick, Marcella H. Boynton, Caitlin N. Dorsey, Alecia Clary, Heather Halko
2017-08-29

Acceptability of Intervention Measure (AIM)Feasibility of Intervention Measure (FIM)Intervention Appropriateness Measure (IAM)implementation outcome measurespsychometric assessment
BACKGROUND: Implementation outcome measures are essential for monitoring and evaluating the success of implementation efforts. Yet, currently available measures lack conceptual clarity and have largely unknown reliability and validity. This study developed and psychometrically assessed three new measures: the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). METHODS: Thirty-six implementation scientists and 27 mental health professionals assigned 31 items to the constructs and rated their confidence in their assignments. The Wilcoxon one-sample signed rank test was used to assess substantive and discriminant content validity. Exploratory and confirmatory factor analysis (EFA and CFA) and Cronbach alphas were used to assess the validity of the conceptual model. Three hundred twenty-six mental health counselors read one of six randomly assigned vignettes depicting a therapist contemplating adopting an evidence-based practice (EBP). Participants used 15 items to rate the therapist's perceptions of the acceptability, appropriateness, and feasibility of adopting the EBP. CFA and Cronbach alphas were used to refine the scales, assess structural validity, and assess reliability. Analysis of variance (ANOVA) was used to assess known-groups validity. Finally, half of the counselors were randomly assigned to receive the same vignette and the other half the opposite vignette; and all were asked to re-rate acceptability, appropriateness, and feasibility. Pearson correlation coefficients were used to assess test-retest reliability and linear regression to assess sensitivity to change. RESULTS: All but five items exhibited substantive and discriminant content validity. A trimmed CFA with five items per construct exhibited acceptable model fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings (0.79 to 0.94). The alphas for 5-item scales were between 0.87 and 0.89. Scale refinement based on measure-specific CFAs and Cronbach alphas using vignette data produced 4-item scales (α's from 0.85 to 0.91). A three-factor CFA exhibited acceptable fit (CFI = 0.96, RMSEA = 0.08) and high factor loadings (0.75 to 0.89), indicating structural validity. ANOVA showed significant main effects, indicating known-groups validity. Test-retest reliability coefficients ranged from 0.73 to 0.88. Regression analysis indicated each measure was sensitive to change in both directions. CONCLUSIONS: The AIM, IAM, and FIM demonstrate promising psychometric properties. Predictive validity assessment is planned.
1
A trimmed model with five items per construct showed acceptable fit (CFI = 0.98, RMSEA = 0.08) and high factor loadings ranging from 0.79 to 0.94.
2
Content-validity testing showed that all but five of the 31 items demonstrated substantive and discriminant validity among implementation scientists and mental health professionals.
3
The abstract is truncated before reporting complete results for known-groups validity, test-retest reliability, and sensitivity to change.
4
The five-item scales demonstrated high internal consistency, with Cronbach’s alpha values between 0.87 and 0.89.
5
The study developed three implementation outcome measures: the Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure.

new implementation outcome measures for the acceptability, appropriateness, and feasibility of adopting evidence-based practices

the psychometric properties of the measures, including content and structural validity, reliability, known-groups validity, test-retest reliability, and sensitivity to change

Publication Details
Publication Date
2017-08-29
Journal
Publisher
ISSN
Cited by
2822
Access Type
Author Information
Authors
Bryan Jeffrey Weiner
Byron J. Powell
Cara C. Lewis
Cameo Stanick
Marcella H. Boynton
Caitlin N. Dorsey
Alecia Clary
Heather Halko
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%