Multi-level Determinants of Implementation and Sustainment in the Education Sector

The goal of this study is to help understand how to successfully implement and sustain evidence based programs (EBP) in the education sector. The study will look at how questions such as what teacher, program, and school characteristics impact sustainment of the EBP BEST in CLASS, what modifications to BEST in CLASS represent a sustained program, and how do teacher, program, and school characteristics influence sustainment of BEST in CLASS.

• BEST in CLASS (Behavioral): BEST in CLASS is a Tier-2 evidence based program (EBP) that has demonstrated effectiveness at reducing children's Disruptive Behavior Problems (DBPs) and risk for Emotional and Behavioral Disorders (EBDs).

Primary outcomes

• Best in Class (BiC) Adherence: Sustainment Average (SU1-SU4) vs. Implementation Post-Baseline Average (IM2-IM4) (Teacher Level Sample) [3 implementation post baseline timepoints: 5 weeks, 10 weeks, 14 weeks AND 4 sustainment timepoints: sustainment baseline, 5 weeks, 10 weeks, 14 weeks]: The difference in the degree to which the 52 teacher sample adheres to the BiC treatment during the post-baseline implementation phase from the prior efficacy trial, in comparison to the same sample's adherence during the present sustainment trial. Adherence is measured during observations as measure by independent observers on the TREATMENT INTEGRITY INSTRUMENT FOR ELEMENTARY SETTINGS (TIES) Adherence scale. TIES-A is a scale of BEST in CLASS practices. Teachers are rated on the presence (use) of each practice on a scale from 1 (not at all) to 7 (extensively). Higher scores are associated with greater adherence.
• Best in Class (BiC) Competence: Sustainment Average (SU1-SU4) vs. Implementation Post-Baseline Average (IM2-IM4) (Teacher Level Sample) [3 implementation post baseline timepoints : 5 weeks, 10 weeks, 14 weeks AND 4 sustainment timepoints: baseline, 5 weeks, 10 weeks, 14 weeks]: The difference in the competence with which the teacher sample delivers the BiC treatment during the post-baseline implementation phase from the prior efficacy trial, in comparison to the same sample's competence during the present sustainment trial. Competence is measured during observations as measure by independent observers on the TREATMENT INTEGRITY INSTRUMENT FOR ELEMENTARY SETTINGS (TIES) Competence scale. TIES-C is a scale of BEST in CLASS practices. Teachers are rated on the competence (quality) of each practice on a scale from 1 (not at all) to 7 (extensively). Higher scores are associated with greater competence.
• Social Skills Improvement System (SSIS) to Measure Change in Student Disruptive Behavior Problems (Student Level Sample) [Baseline of sustainment intervention baseline and week 14 of sustainment intervention]: Problem Behaviors (Standard Score) The mean difference between the teacher-reported problem behavior standard score exhibited by the focal student sample (78 enrolled, 72 completed) from Fall (SU1) to Spring (SU4) of the sustainment year. Student (72 sample size) problem behaviors will be measured by the Social Skills Improvement System (SSIS) problem behavior scale. Higher scores represent more problem behaviors. SU1 v. SU4 (sustainment baseline v. sustainment 14 weeks). The core composite scores for Social Skills and Problem Behaviors have a mean of 100. Frequency Ratings: Items are rated on a 4-point scale: 0 = Never, 1 = Seldom, 2 = Often, 3 = Almost Always, 4=Always.
• Social Skills Improvement System (SSIS) Social Skills Scale to Measure Change in Student Prosocial Behavior (Student Level Sample) [Baseline of sustainment intervention and week 14 week of sustainment intervention]: The mean difference in teacher-reported social skills exhibited by the focal student sample (78 enrolled, 72 completed) from Fall (SU1) to Spring (SU4) of the sustainment year. Student (72 sample size) prosocial behaviors will be measured by the Social Skills Improvement System (SSIS) social skills scale. Social Skills (Standard Score). Higher scores represent higher level of social skills. SU1 v. SU4 (sustainment baseline v. sustainment 14 weeks). The core composite scores for Social Skills and Problem Behaviors have a mean of 100. Frequency Ratings: Items are rated on a 4-point scale: 0 = Never, 1 = Seldom, 2 = Often, 3 = Almost Always, 4=Always

Sex: All, ages: 5 Years to any, accepts healthy volunteers

Inclusion Criteria:

Teacher participants

* teach in general education Kindergarten, 1st, 2nd, or 3rd grade
* consent to participate
* proficient in English
* participated as a BEST in CLASS teacher who received coaching in the prior year

Children participants

* enrolled in a general Kindergarten, 1st, 2nd or 3rd grade classroom
* proficient in English
* at elevated risk for EBD
* demonstration of externalizing behaviors that interfere with participation in the classroom (e.g., disruption, aggression) as indicated by teacher rating
* parental/guardian consent to participate
* Assent to participate if the child is 7 years or older

Exclusion Criteria:

Teacher participants

* do not teach in general education Kindergarten, 1st, 2nd, or 3rd grades
* were not part of the BEST in CLASS Elementary intervention condition
* do not consent to participate
* do not speak fluent English
* do not have at least one student that meets the inclusion criteria

Children participants

* caregivers do not provide permission for participation
* caregiver(s) do not speak proficient English or Spanish (caregiver consent provided in both languages)
* are not proficient in English
* do not screen in as demonstrating risk for EBD
* participated as a focal child in the classroom of a participating treatment teacher in the previous efficacy trial

Participant flow

Outcome results

• Best in Class (BiC) Adherence: Sustainment Average (SU1-SU4) vs. Implementation Post-Baseline Average (IM2-IM4) (Teacher Level Sample) (Primary), Mean, score on BiC: Teacher Implementation (51 Subject Sample) -.075 +/- .454
• Best in Class (BiC) Competence: Sustainment Average (SU1-SU4) vs. Implementation Post-Baseline Average (IM2-IM4) (Teacher Level Sample) (Primary), Mean, score on BiC: Teacher Implementation (52 Subject Sample Enrolled, 50 Subjects Completed This Outcome) .001 +/- .631
• Social Skills Improvement System (SSIS) to Measure Change in Student Disruptive Behavior Problems (Student Level Sample) (Primary), Mean, score on SSIS: Focal Student (72 Subject Student Sample) -3.260 +/- 11.646
• Social Skills Improvement System (SSIS) Social Skills Scale to Measure Change in Student Prosocial Behavior (Student Level Sample) (Primary), Mean, score on SSIS: Focal Student (72 Subject Student Sample) 4.603 +/- 10.892

Adverse events

Time frame: During both of the study timeframes (IM-Implementation) and SU (Sustainment): IM2 = 5 weeks of implementation IM3 = 10 weeks of implementation IM4 = 14 weeks of implementation SU1 = baseline of sustainment SU2 = 5 weeks of sustainment SU3 = 10 weeks of sustainment SU4 = 14 weeks of sustainment

• Study Protocol and Statistical Analysis Plan (PDF, 2022-03-18)
• Informed Consent Form (PDF, 2022-08-26)

• Bryce D McLeod, PhD, Principal investigator, Virginia Commonwealth University

NIH grants: R01MH124439