Skip to main content
Download PDF
  • Study protocol
  • Open access
  • Published:

Study protocol: cluster randomized trial of consultation strategies for the sustainment of mental health interventions in under-resourced urban schools: rationale, design, and methods

BMC Psychology volume 10, Article number: 24 (2022) Cite this article

Abstract

Background

The school is a key setting for the provision of mental health services to children, particularly those underserved through traditional service delivery systems. School-wide Positive Behavioral Interventions and Supports (PBIS) is a tiered approach to service delivery based on the public health model that schools use to implement universal (Tier 1) supports to improve school climate and safety. As our prior research has demonstrated, PBIS is a useful vehicle for implementing mental and behavioral health evidence-based practices (EBPs) at Tier 2 for children with, or at risk for, mental health disorders. Very little research has been conducted regarding the use of mental health EBPs at Tier 2 or how to sustain implementation in schools.

Methods/design

The main aim of the study is to compare fidelity, penetration, cost-effectiveness, and student outcomes of Tier 2 mental health interventions across 2 sustainment approaches for school implementers in 12 K-8 schools. The study uses a 2-arm, cluster randomized controlled trial design. The two arms are: (a) Preparing for Sustainment (PS)—a consultation strategy implemented by school district coaches who receive support from external consultants, and (b) Sustainment as Usual (SAU)—a consultation strategy implemented by school district coaches alone. Participants will be 60 implementers and 360 students at risk for externalizing and anxiety disorders. The interventions implemented by school personnel are: Coping Power Program (CPP) for externalizing disorders, CBT for Anxiety Treatment in Schools (CATS) for anxiety disorders, and Check-in/Check-out (CICO) for externalizing and internalizing disorders. The Interactive Systems Framework (ISF) for Dissemination and Implementation guides the training and support procedures for implementers.

Discussion

We expect that this study will result in a feasible, effective, and cost-effective strategy for sustaining mental health EBPs that is embedded within a multi-tiered system of support. Results from this study conducted in a large urban school district would likely generalize to other large, urban districts and have an impact on population-level child mental health.

Trial registration ClinicalTrials.gov identifier number NCT04869657. Registered May 3, 2021.

Peer Review reports

Background

A major goal of dissemination and implementation research is to identify implementation strategies that can help to sustain the gains that occur as a result of the initial implementation of evidence-based practices (EBPs) [1]. The school is a key setting for the delivery of mental health services to children, particularly those who are underserved [2]. School-wide positive behavioral interventions and supports (PBIS) [3] is a service delivery framework based on the public health model used in schools to improve school climate and safety. This framework is a useful vehicle for implementing targeted (Tier 2) mental health EBPs for children with or at risk for mental health disorders [4,5,6,7,8]. PBIS is effective in reducing office discipline referrals (ODR) and improving children’s behavior [9] and overall school climate [10]. Very little research has been conducted on using mental health EBPs at Tier 2 or how to sustain their implementation [11]. The main aim of the study is to compare implementation, cost-effectiveness, and student outcomes of Tier 2 mental health interventions across 2 sustainment approaches for school implementers in 12 K-8 schools.

Sustainment of EBPs in schools

EBPs can be sustained in under-served settings when there is internal capacity and ongoing opportunities for training. Sustainment has been defined as the maintenance of health benefits over time [12] and as the continued implementation of an intervention or prevention program that maintains fidelity to the core program principles after the supplemental resources that supported initial training are removed [13]. Studies examining implementation of PBIS have noted that adequate and ongoing training opportunities are frequently identified as “essential practices” needed to successfully sustain interventions [14, 15]. A lack or significant reduction of training resources has been associated with practice abandonment in the school setting [16]. Also, urban schools are typically unable to sustain mental health programs unless they are provided ongoing expert support [17]. Continuous training leads to the development of school-level expertise that can mitigate the negative effects of staff turnover [18]. Such training may increase participants’ abilities to implement interventions with fidelity and adapt the practice as needed without eliminating key components [13]. Preparing for sustainment concomitantly with initial implementation [19] with a goal of infusing the intervention into the school organizational culture is key [1]. Analyses of interventions in both mental health and school settings suggest that sites are more likely to sustain interventions if they are implemented with high rates of fidelity [20], and if implementation and student-outcome data are used to inform decision-making [20, 21]. Sustainment is more likely when EBPs are supported at the district level and resources are dedicated to developing a model for implementation [22]. Reviews of the literature have consistently identified the importance of a program “champion” in intervention sustainment [23]. Also, school principals have been identified as critical to intervention sustainment [24] and can provide important support as both instructional and school-culture leaders.

Proposed sustainment strategies

Little is known about what models of support for sustainment are effective, feasible, and cost-effective. In order to achieve the potential public mental health benefits of targeted mental health interventions in schools, an efficient, scalable, cost-effective and sustainable model of training and consultation that can be transferred from experts to the school district is needed. A number of studies have shown that more experienced school personnel can successfully serve as coaches and supervisors to those implementing EBPs [25,26,27]. Sustainment of Tier 2 interventions is complicated by the fact that most urban schools do not have an established training approach for professionals already working in the schools [11, 17]. The lack of access to effective EBP training in low-income communities is contributing to major service disparities.

Based on the extant literature, our sustainment approach will (a) prepare for sustainment during initial implementation, (b) use existing supervision arrangements in the schools to develop and reinforce internal capacity for training and supervision, (c) train with the goal of maintaining high fidelity, (d) use implementation champions (i.e., district coaches already present at each school), and (e) maintain buy-in and support by instituting an Advisory Board of school district stakeholders.

We will test two strategies for sustainment: (a) Prepare for Sustainment (PS)—sustainment supported by school district coaches who, in turn, receive technological enhanced consultation from external consultants, and (b) Sustainment as Usual (SAU)—sustainment supported by school district coaches alone. The first strategy involves gradual removal of support coupled with the use of technology. This innovative strategy reduces cost of implementation while concurrently creating internal capacity for sustainment.

Preliminary studies

The three EBPs that will be used in this study have been found to be effective in multiple studies. Also, our group has found the interventions to be effective specifically with students in the School District of Philadelphia (SDP) [7, 8, 28]. Our training and consultation approach has been used in two prior randomized trials in the SDP and shown to be feasible, clinically-effective and acceptable by therapists and supervisors [7, 8, 28]. We contacted administrators and school counselors from schools participating in a study using mental health EBPs embedded within PBIS to determine whether schools continued to implement the EBPs after the end of the study. We found that 50% of the schools continued to implement one of two EBPs one year after the end of the study, and 33% of schools continued to implement one of the EBPs two years after the end of the study. Only one school continued to implement both programs during both years. The main reason given for the discontinuation was lack of training support during the sustainment period.

In one of our prior studies, research staff provided interactive consultation to clinical supervisors on how to conduct effective supervision of school-based therapists on the implementation of two group therapy EBPs. The consultation for one of the conditions was conducted via Zoom. Therapists or research assistants video-recorded child sessions and supervision sessions. The videos were uploaded onto a dedicated server. A usability analysis of Zoom, plus the dedicated server and REDCap used in 12 schools, showed that Zoom was compatible with school and hospital computers as well as mobile devices. All scheduled consultation sessions were completed within the expected timeline and we were able to securely upload, view, and store all recorded session content on a dedicated server at the hospital. This technology allowed our team to successfully provide secure remote consultation, including video feedback and access to training and review materials, to agency supervisors beyond the initial on-site training workshop. We will use similar processes and technologies in the present study.

In our prior studies, we found that schools, coaches and Tier 2 implementers differ widely with regard to their exposure to EBPs, and their experience supervising other therapists on the implementation of mental health interventions. As such, members of the research team, who are highly experienced in the implementation of the EBPs used in the study and in training Masters-level therapists, will conduct the initial training and subsequent consultation in order to ensure that all schools and all supervisors and implementers are equally prepared for the sustainment phases of the study.

Mediators and moderators of consultation type on fidelity

It is important to identify mechanisms (mediators) through which implementation strategies (i.e., consultation type) work and specific moderation factors that can be used to target implementation strategies to the appropriate Tier 2 implementer. In addition, it is critical to study the role of perceived appropriateness, feasibility and acceptability of consultation support and the individual level variables related to how Tier 2 implementers perceive their jobs because these constructs are modifiable. We are interested in the role of perceived consultation appropriateness (our main mediator variable) because this construct has been shown to be a “leading indicator” of implementation success [29]. We plan to examine whether a consultation strategy for sustainment implemented by school district coaches, who are in turn supported by expert trainers, would be more appropriate for Tier 2 school personnel and thus associated with higher EBP fidelity, than a consultation strategy implemented by school district coaches without support. We are also interested in assessing moderator effects of certain individual-level variables on EBP fidelity. Perceived burnout (emotional exhaustion, depersonalization, personal accomplishments), role conflict, and role overload have been shown to affect school personnel’s ability to teach and implement EBPs in the school setting [17, 30]. In our previous work, we have found that school personnel in under-resourced schools often feel overextended and exhausted and, in some cases, this affects their performance in implementing EBPs [8, 28]. Thus, we also will examine whether the relationship between perceived appropriateness and EBP fidelity varies depending on staff perceived burnout. Regardless of the specific results, by testing the hypothesized relationships, the findings will directly inform the design of future, targeted interventions. In particular, the findings will indicate whether interventions should focus on modifying the training and consultation approach in order to make them more appropriate, feasible and/or acceptable.

Methods

Participants

Participants will be school staff and students attending urban public schools that are already implementing PBIS in Philadelphia, PA.

Inclusion criteria

The inclusion criteria for participants include:

Tier 2 team

School personnel already assigned by the school principal to the Tier 2 team to support students in Tier 2. The Tier 2 team will conduct team meetings to review referrals for Tier 2, determine appropriate interventions for students, and track student progress.

Tier 2 coach

School or district personnel with Masters-level education selected by administrators. The Tier 2 coaches will train and supervise Tier 2 team members and Tier 2 implementers during the sustainment phases.

Tier 2 implementer

These are school staff with and without prior mental health training selected by the school principal. Tier 2 implementers will implement Tier 2 interventions.

Tier 2 student

These students must be (a) attending one of the participating schools, (b) in grades 4–8; (c) identified by the Tier 2 Team as not responding to Tier 1 intervention, thus needing Tier 2 support; and (d) scoring > 1 standard deviation above the mean on the Emotional Symptoms or Conduct Problems of the Strengths and Difficulties Questionnaire (SDQ) [31] completed by a parent or a teacher. Students will receive Tier 2 support by participating in one of the Tier 2 interventions.

Exclusion criteria

School personnel not employed by the school district and students not enrolled in Tier 2 will not participate. Students with a special education classification of “Intellectual Disability,” students who have diagnoses that make participation in the study clinically inappropriate (i.e., current substance abuse disorder, psychotic or autism spectrum disorders, based on school records) because they would be unlikely to benefit from the interventions used in the study or who present as an acute risk to themselves or others, will be excluded.

Overview and randomization

The study uses a parallel 2-arm, cluster randomized controlled trial design. The two arms are: (a) Prepare for Sustainment (PS), and (b) Sustainment as Usual (SAU) (see Fig. 1).

Fig. 1
figure 1

CONSORT diagram

Full size image

We will recruit 12 schools, 145 school personnel (13 coaches, 72 Tier 2 members, 60 Tier 2 implementers), and 360 students in 3 waves (4 schools, 20 implementers, 120 students in each wave) in order to strategically deploy resources. During Phase 1 (all waves), Tier 2 implementers and Tier 2 team members will receive the same level of support provided by research consultants. At the end of Phase 1 of each wave, schools will be randomly assigned in 1:1 ratio to condition (PS [n = 6], SAU [n = 6]). Tier 2 implementers, school district coaches, and Tier 2 team members will participate in their school’s assigned condition. We expect that there will be 5 Tier 2 implementers per school. Schools will be stratified by school level (elementary or middle schools) to ensure that both conditions have the same number of school types. All schools will have similar levels of students qualifying for free or subsidized lunch.

Tier 2 interventions

All participating schools will already be implementing PBIS at Tier 1. None of the schools will have any significant experience prior to implementing mental health EBPs at Tier 2. The EBPs that schools will be able to use for Tier 2 in this study are shown in Table 1.

Table 1 Tier 2 interventions
Full size table

Decisions as to whether students are assigned to group intervention or individual intervention will be made by the Tier 2 team in each school. These decisions will be largely based on whether the Tier 2 team can assemble intervention groups of students of similar developmental level who present similar difficulties. In addition, the Tier 2 team might choose individual or group therapy for certain students based on the student’s presenting problems or other factors important to the school. The study team will provide guidance to the Tier 2 team about the type of students who would be appropriate for each specific program or for individual vs. group intervention.

Training components

Initial training is provided by research team consultants. Initial training components for all schools (Phase 1) will include (a) initial training workshop, (b) consultation on the use of the Team-Initiated Problem Solving (TIPS) [40] approach for assigning students to a specific Tier 2 intervention, and monitoring students’ progress, and (c) consultation on the use of Tier 2 interventions (CPP, CATS, CI/CO). Subsequent training support (Phases 2–3) will vary by year and condition (see Fig. 2). All materials, including Tier 2 interventions, training manuals and exemplar videos, will be accessible via an online platform to all participating school personnel. The purpose of Phase 1 is to create a baseline where all participating school personnel will be equally prepared for the sustainment phases.

Fig. 2
figure 2

Sustainment Support for School Personnel

Full size image

Preparing for sustainment (PS) condition

Following Phase 1 (implementation), schools will be randomly assigned to PS or to SAU. Following the retraining workshop during Phase 2, school district coaches (with research consultant support) will provide 5 separate on-site consultation sessions to each Tier 2 team; and 6 on-site sessions for Tier 2 implementers. The consultation sessions will be pre-scheduled at the beginning of the school year. With regard to content, the Tier 2 team will be expected to choose a particular Tier 2 team audio-recorded meeting and an intervention session that the team would like the school district coach to review and discuss during the consultation session. Also, the Tier 2 implementers will be expected to raise points that they would like to discuss and receive feedback regarding any aspect of the intervention (e.g., how to conduct a particular role play in session) or any implementation barrier. The remote support will be provided to the school district coaches via Webex and an online file share server will include consultation manuals, materials (e.g., TIPS forms for running effective Tier 2 team meetings using the TIPS approach; fidelity forms for Tier 2 interventions), and how to prepare for the consultation session with school personnel. The same coaches in PS will support the schools during Phase 3 without receiving any support from external consultants.

Sustainment as usual (SAU) condition

During Phases 2–3 of the study, school district coaches will provide separate on-site consultation to the Tier 2 team and Tier 2 implementers using a sustainment as usual approach. School district coaches will have participated in a training for coaches and as active participants during the initial training and consultation meetings for Tier 2 teams and Tier 2 interventions (Phase 1) for school personnel in SAU. They will not receive direct support from external consultants but they will have access to all training and consultation materials.

Training of coaches

Across the three study waves, 6 Tier 2 coaches (3 coaches in PS and 3 coaches in SAU) will participate in a 2-day training workshop on supervision strategies for the implementation of EBPs in school settings (e.g., learn a competency framework for supervisors [41], strategies for identifying children who could benefit from the service, and how to track their progress). They will also be trained on conducting consultation with Tier 2 implementers. External consultants from the research team will provide 1 h of consultation support to school district coaches in the PS condition for each consultation session school district coaches provide to school personnel during Phase 2. The district coaches will also assist research consultants during Phase 1 as they train and provide consultation to Tier 2 implementers. The 6 district coaches will support Tier 2 implementers during Phase 2 and Phase 3.

Implementation framework and training procedures

All activities related to the training of school personnel and implementation and sustainment of EBPs are guided by the Interactive System Framework for Dissemination & Implementation (ISF) [42] (see Fig. 3). ISF is intended to be a “heuristic for understanding key systems, key functions, and key relationships relevant to the dissemination and implementation process” [42] (p. 179). ISF is composed of three interrelated systems: Synthesis and Translation System (STS), Support System (SS), and Delivery System (DS). The function of STS is to distill information innovations and prepare them for implementation by service providers. SS supports the work of those who put the innovation into practice. The primary function of DS is the implementation of innovations in “real world” settings [42, 43].

Fig. 3
figure 3

Interactive Systems Framework for D&I

Full size image

Synthesis and translation system (STS)

An Advisory Board will be created to facilitate recruitment and buy-in, and to employ “climate embedding mechanisms” [44] for the implementation and sustainment of EBPs in the school district. The Advisory Board will be comprised of school district administrators, principals and counselors.

Synthesis (step 1) & translation (step 2)

We will present synthesized information about the implementation and effectiveness data of Tier 2 EBPs from literature reviews and previous studies conducted by our research team to the Advisory Board. We will work with the Advisory Board to create an easy-to-read document that shows the effectiveness of Tier 2 EBPs used in the study and share it with school staff and parents in order to seek stakeholder buy-in for sustainment. (Step 3) We will seek principal commitment to support and champion the integration of mental health supports at Tier 2. Based on our preliminary studies and PBIS literature, we will employ the TIPS approach to support the school Tier 2 teams, which will include Tier 2 implementers, as they incorporate Tier 2 mental health supports. (Step 4) We have developed training materials that will be disseminated among participating schools. The training materials include a series of video modules. Modules that focus on interventions include a discussion of the theoretical background of the particular EBP, its development and a detailed review of group sessions. Participants in both conditions will have access to these materials for the duration of the project via a dedicated online server housed at CHOP.

Support system (SS)

General capacity building (step 1)

The study will be conducted in the context of general capacity building for sustainment, specifically, training and consultation, to enable certain school personnel (e.g., faculty, counselors, social workers, climate specialists) to become effective users of the TIPS process for identifying and following students in Tier 2. Activities in the Support System and the bidirectional relationship of activities between the Support System and the Delivery System will be informed by the Evidence-Based System for Innovation Support (EBSIS) [43], a logic model developed by Abraham Wandersman and colleagues, to supplement the ISF [42, 43]. The EBSIS consists of a series of four sequential support components: Training, Tools, Technical Assistance, and Quality Assurance/Quality Improvement [43].

Training

Research consultants will be project staff with expertise in the implementation of PBIS with mental health supports. They will be trained and supervised by senior members of the research team on how to conduct training, retraining and consultation. Research consultants will conduct three days of on-site training in August at the outset of the study (Phase 1) for all school personnel involved in the TIPS approach and Tier 2 EBPs. District coaches will conduct retraining for PS after having received consultation from external consultants (Phase 2). District coaches in SAU will conduct retraining by themselves. Members of the Tier 2 teams and Tier 2 implementers will be introduced to the TIPS [40] approach to use data to identify and assign students at risk for behavioral and emotional disorders into Tier 2, and to measure fidelity to the EBP and intervention outcomes. The Tier 2 implementers will also be taught a competency framework [41, 45], strategies (e.g., in-service presentations to the faculty) for enhancing school personnel knowledge of mental health “red flags” among students, and how to access the online materials. Retraining (Phases 2–3) will include a refresher of the information contained in the initial training.

Innovation-specific capacity building, (step 2)—tools

During the initial training (Phase 1) and retraining, members of the Tier 2 team/Tier 2 implementers and district coaches will be instructed to use training manuals and adherence checklists for TIPS and Tier 2. The Tier 2 team will be trained on the use of a mental health screening instrument (SDQ) [31] and a parent rating scale (Behavioral Assessment System for Children, Third Edition [BASC-3]) [46] and other instruments used in Tier 2. Based on our prior studies and informed by TIPS [40], the Tier 2 team will also learn how to modify the standard PBIS leadership team meetings to integrate mental health interventions into the tiers of support, context for problem behaviors, how to identify students in need of higher levels of support, CICO, and other components used in PBIS training. [45, 47] During all phases of the study, members of the Tier 2 team will be expected to include Tier 2 implementers in their meetings and to support them by discussing, and solving as a group, any Tier 2 implementation barriers. Tier 2 implementers will be introduced to a competency model for CBT [48]. They will also learn about how to deal with implementation barriers (e.g., scheduling sessions, conducting exposure tasks) [49].

Delivery system (DS)

General capacity use (step 1)—technical assistance

During Phase 2 and Phase 3, Tier 2 teams and Tier 2 implementers, respectively, will receive a 3-h retraining by school district coaches. The purpose of the retraining will be to review key points from the training and to answer questions about any aspect of the implementation process. Through the initial training and retraining, the Tier 2 teams will have the capacity to use the TIPS process [40] to identify students using SDQ data. Tier 2 implementers will have the capacity to implement CI/CO and group CBT with fidelity.

Innovation-specific capacity use, (step 2)—consultation

Training and consultation content and procedures will be based on adult learning characteristics (e.g., propensity to learn from experience, capacity to reflect on performance and apply knowledge, and self-motivation) [50, 51]. Consultants will provide eight 50-min training sessions on assisting the Tier 2 team to run team meetings using the TIPS process. Consultation for Tier 2 will be provided for each intervention the Tier 2 implementer conducts. For example, consultants will provide one 50-min consultation session to Tier 2 implementers for each group session the staff member conducts with students (e.g., 12 sessions for Coping Power Program). Consultation for TIPS will include: (a) reflecting [52] on a prior team meeting using TIPS; (b) providing feedback based on TIPS fidelity measure; and (c) dealing with implementation barriers. Consultation for Tier 2 will include: (a) reflecting on previous group intervention session or CI/CO case; (b) discussion of intervention strategies; and (c) tailored problem solving for dealing with implementation barriers. In addition, consultation will dedicate time for didactics (e.g., preparing for next team meeting or group CBT session or CI/CO case; answering questions about the consultation manual).

Quality assurance / quality improvement

All Tier 2 team meetings and student Tier 2 group sessions will be audio-recorded using a device provided by the research team. A designated member of the Tier 2 team and Tier 2 implementers will be in charge of audio-recording and uploading audio files to an online server. If schools have difficulty doing this, the audio-recording and uploading will be conducted by a research assistant (RA). The audio files will be coded by independent coders (ICs) for fidelity. About 20% of the audio files will also be listened to by external consultants during Implementation; school district coaches in PS will listen to the audio files for the purpose of consultation. These procedures will ensure quality assurance / quality improvement for the consultation approach used in the study and should contribute to maintaining appropriate implementation and clinical outcomes. The audio files will be provided to school district coaches in SAU and to all district coaches during Phase 3 so that they can continue to be used to provide feedback to the Tier 2 team and Tier 2 implementers. Given that district coaches will operate totally independently of the research team in SAU and Phase 3 of PS and SAU, the research team will keep track of the use of audio files in consultation, as well as the frequency and length of consultation sessions via logs completed by school district coaches.

Measures and assessment procedures

Thought leaders and researchers in the area of sustainment [53] have identified four characteristics of optimal sustainability research: (1) using mixed quantitative and qualitative data approaches; (2) reflecting perspectives of multiple stakeholders invested; (3) capturing variables at multiple levels of health delivery; and (4) collected over multiple time points. We will use this approach to data collection. A list of measures is provided in Table 2.

Table 2 Measures by variable/construct, measure characteristics, timepoint, method and informant
Full size table

Data analysis by aim and hypothesis

Aim 1

To compare sustainment conditions for Tier 2 mental health interventions.

We hypothesize that randomly assigned Tier 2 implementers to the PS condition will maintain higher implementation fidelity at Phase 2 and Phase 3 compared to implementers assigned to the SAU condition.

To address implementation fidelity in Aim 1, Hypothesis 1, the unit of analysis is the 60 implementers who are nested within schools; strategies for sustainment groups are PS and SAU and time is the repeated measure collected by the end of Phase 1 (before randomization), Phase 2, and Phase 3. The linear mixed-effects model will be utilized with main effects of strategies for sustainment (PS, SAU), time (Baseline, Phase 2, Phase 3) and the group by time interaction will be the independent variables predicting implementation fidelity. Implementers across each school and each intervention’s year will be pooled and the models will include an implementer-level random effect to account for variability among implementers. For each outcome measure, the mixed effects modeling approach for a 2-level mixed effects linear regression model will be presented below for illustration purposes using the notation and descriptions given by Donner and Klar [65]: Yijl = U + Gi + Vij + eijl; Where i = 1, 2 (PS, SAU); j = 1, 2, …, ki (School j in group i) and l = 1,2, …, mij therapist in group I school j. Therefore, Yijl represents outcomes for participant l in school j condition i. U = response grand mean; Gi = fixed effect of the intervention condition; Vij = the random cluster effects which assumed to be normally distributed ~ N(0, σ2A); eijl is the error term assumed to be normally distributed ~ N (0, σ2w). The equation represents a mixed effects model since the strategies for sustainment occurs at the school level (the clusters = j) and participants are nested within schools. The intervention effect is fixed, and the cluster effect is random. The effects of Vij and eijl are assumed to be independent with a common intra-cluster correlation. The 2-level model presented above will be expanded to include students within implementers (level 3) as a cluster and its random effect will be estimated, assuming it is normally distributed ~ N (0, σ2z). In all models, we will consider adding pertinent covariates related to student level factors if randomization at the school level did not ensure comparability among implementors and/or students characteristics between the two groups.

Hypothesis 2

Compared to SAU, PS will maintain higher penetration at the school provider level (i.e., behavioral health staff involved in the implementation of EBPs) at Tier 2, and EBP penetration at the student service level (i.e., students receiving EBPs at Tier 2) during Phase 2 and Phase 3 of the study compared to Phase 1.

The information regarding penetration (Aim 1, Hypothesis 2) will be collected using questionnaires completed by school therapists and their coaches. The data will be presented descriptively, using mean, standard deviation, quartiles, minimum and maximum by PS and SAU. Changes in score between Phase 1 and Phase 2 and Phase 1 and Phase 3 will be presented using the 95% confidence intervals.

Hypotheses 3 & 4

At Phase 2 and Phase 3, students receiving mental health interventions in PS will show better symptom severity improvement and better academic engagement improvement compared to students receiving mental health interventions in SAU.

Regarding Aim 1, Hypotheses 3 & 4, we will determine the difference in student symptom severity improvement and academic engagement improvement between PS and SAU. Statistical analysis related to student outcomes will be done by phases. Phase 1 will be analyzed as a one group using pre/post changes in BASC-3 and descriptive statistics for EvsD. Phase 2 and Phase 3 will be analyzed using the mixed effect modeling approach described above with group, time (pre/post) and interaction term (group X time).

Differences in BASC 3 and EvsD between students receiving support by implementers under PS vs SAU will be estimated by mean ± standard deviation and the 95% confidence intervals (CI) of the differences and presented by Phase 2 and Phase 3.

Power analyses

We used PASS 3 Power Analysis and Sample Size Software [66] to determine the power and minimum detectable implementation fidelity effect sizes (MDES) based on a sample of 12 schools with 5 implementers per school (60 implementers). The effect size refers to the differences in implementation fidelity between PS and SAU (ES = (|µ1 − µ2|/σ)) in Phase 2 and Phase 3. Using the empirical results from our prior study on the effects of PBIS with mental health supports [7], the intra-cluster correlation coefficient (ICC) was set to 0.05. A group sample of 6 clusters (schools) per PS and SAU with 5 implementers per cluster, (i.e., 30 implementers in PS and 30 implementers in SAU; 60 total) achieve 85% power to reject the null hypothesis of equal implementation fidelity means when the population mean difference between PS and SAU is assumed to be 9.0%, standard deviations of 9.5, and with a significance level (alpha) of 0.05 using a two-sided two-sample t-test. We anticipate at total of 360 students will participate from 12 schools over three study phases (30 students per school).

Hypothesis 4

PS will be more cost-effective compared to SAU during Phases 2–3.

We will address Aim 1, Hypothesis 4 by calculating PS incremental cost-effectiveness ratio and determining whether it is significantly less than $150/[BASC-3 T-score]. This ratio is made up of the difference in cost divided by the difference in parent reported scores on the Aggression, Conduct Problems, and Anxiety scales of the BASC-3.

The difference in cost between PS and SAU will be assessed by use of the generalized linear model (GLM). Link and family functions for the GLM will be empirically fit to the data using diagnostic tests including the Modified Parks test, Pregibon-Link test, Hosmer–Lemeshow test [67]. Separate ratios will be calculated for Phase 2 and Phase 3 of the trial.

Aim 2

To explore mediators and moderators of consultation support on implementer fidelity.

Our primary hypotheses are that (2a) Tier 2 implementers’ perceived appropriateness of the consultation support will mediate the relationship between type of support they receive and implementation fidelity, and (2b) implementer burnout will moderate the relationship between the type of consultation support and fidelity (see Fig. 4).

Fig. 4
figure 4

Mediators and moderators of type of support on fidelity

Full size image

In addressing Aim 2, we will employ the regression modeling approaches proposed by Hayes [68]. These approaches utilize PROCESS macro suitable for SPSS and SAS. The models will reflect change scores in the mediators. To produce a precise estimate of mediation effects and properly account for the potential of regression to the mean, baseline measurement of the mediator will be included as a covariate.

To address the moderation effect, we will examine the impact of sustainment (PS vs. SAU) on implementation fidelity with the effect of implementer burnout on implementation fidelity. We will use a regression modeling approach with the outcome (Y) being the dependent variable, treatment condition (T) being the independent variable, a subgrouping variable or covariate (X) being the moderator tested, and the test of moderation reflected as the interaction of treatment by the covariate. The generic regression equation is Y = b0 + b1 T + b2 X + b3 (T*X). Where Y = implementation fidelity; T = sustainment group (PS, SAU), X = implementer burnout. A significant interaction term (T*X) of group by implementer burnout indicates that the treatment effect varies as a function of the covariate or subgroup variable. The Wald statistical test will be used for testing the interaction term. In planning and performing the analyses related to mediation and moderation, we will utilize the PROCESS modeling tool.

Aim 3

To qualitatively examine school personnel perceptions of mental health training support. Participants in Aim 3 will include Tier 2 implementers (n = 60), school district coaches (n = 6), and school leaders and administrators (n = 12). Interviews will be digitally recorded and transcribed with analyses supported by use of an NVivo database. Using an integrated approach [69] to codebook development, a priori codes will be developed to capture relevant implementation constructs including acceptability and feasibility of training and sustainment supports and implementation and sustainment processes. Additional codes will be added by the research team following a close reading of the first five transcripts [70]. We will develop a structured codebook. Each code will be defined and decision rules for their application included in the definition. Using NVivo, two members of the research team will separately code a sample of five transcripts and compare their application of the coding scheme to assess the reliability and robustness of the coding scheme. Disagreements in coding will be resolved through discussion and the codebook refined and applied to all transcripts. Coders will be expected to reach and maintain reliability at κ ≥ 0.85. Following coding, we will conduct member-checking, reviewing our conclusions with a multidisciplinary group of colleague clinical stakeholders to validate our analyses. Additionally, data will be triangulated with quantitative markers of feasibility, including proportion of schools and implementers invited to participate who enroll in the project and implementers retained throughout the sustainment phase.

Mixed methods analysis

We will integrate the quantitative data and interviews from Aim 1 using the following taxonomy: the structure is Quan → QUAL, the function is to expand upon the quantitative findings to understand the process of sustainment of Tier 2 interventions as experienced by stakeholders, and the process is connecting [71]. To integrate the quantitative and qualitative methods, we will follow the NIH guidelines for best practices [72]. We will use the quantitative data to identify patterns in the qualitative data. To do this, we will enter quantitative findings into NVivo as attributes of each participant (i.e., fidelity of the school the participant was drawn from). We will use the Content Fidelity Checklist (CFC) and the Check-In/Check-Out Fidelity Checklist as the measures of fidelity. The biostatistician will visually inspect the distribution of fidelity scores for each measure to determine logical cut points to classify schools as high, medium, and low fidelity and we will then enter these fidelity classifications into NVivo for each participant interview. These quantitative attributes will be used to categorize and compare important themes among subgroups.

Discussion

We will compare the effects of two sustainment strategies on implementation and student outcomes. The examination of mediators and moderators of consultation support on fidelity might lead to the identification of modifiable factors to improve training and consultation approaches. We plan to use rigorous methods to compare outcomes, using measures with strong psychometric properties, multiple data collection strategies (surveys, interviews, independent coding), quantitative and qualitative data, and sound analytical methods.

Innovations

This study contains several key innovations, including (a) examining sustainment, which is an understudied phase in implementation research in general, and in school mental health research in particular, (b) conducting the first study that seeks to test the effectiveness of a training and consultation support strategy for the sustainment of mental health EBPs within PBIS, (c) reducing cost for implementation while concurrently creating internal capacity for long-term sustainment, (d) using school stakeholders to prepare for sustainment concomitantly with initial implementation, (e) gradually removing supports in a way that is conducive to the transfer of knowledge from external consultants to school district coaches and Tier 2 implementers, (f) using the team problem-solving approach (TIPS) for mental health supports at Tier 2, and (g) conducting the first study investigating the cost and cost-effectiveness of two sustainment strategies in urban schools.

Limitations

We will not be able to test the effectiveness of Tier 2 interventions against a control intervention. However, the Tier 2 interventions used in the study have been found to be effective in multiple studies. The current study will not be able to obtain implementation or effectiveness data on students needing individualized supports (Tier 3). Collecting these types of data would be beyond the scope of the current study. Results may not generalize to non-urban schools because of the unique characteristics of under-resourced urban schools. However, results should generalize to any large urban school district in the country.

Conclusions

We expect that this study will result in a feasible, effective, and cost-effective strategy for the sustainment of mental health EBPs embedded in PBIS. Linking PBIS with state-of-the-art mental health supports and well-trained school personnel will likely lead to sustained improvement in student mental health and academic markers. Results from this study conducted in a large urban school district would likely generalize to other large, urban districts, and impact population-level child mental health.

Availability of data and materials

The datasets generated and/or analyzed during the current study will be available in the NIMH Data Archive repository, https://nda.nih.gov/.

References

  1. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;2(8):117.

    Article  Google Scholar 

  2. Garland AF, Hough RL, McCabe KM, Yeh M, Wood PA, Aarons GA. Prevalence of psychiatric disorders in youths across five sectors of care. J Am Acad Child Adolesc Psychiatry. 2001;40(4):409–18.

    Article  PubMed  Google Scholar 

  3. Sugai G, Horner R, Todd A, Lewis-Palmer T. School-wide positive behavior support. In: Bambara L, Kern L, editors. Individualized supports for students with problem behaviors: designing positive behavior plans. New York: Guilford Press; 2005. p. 359–70.

    Google Scholar 

  4. Duchnowski AJ, Kutash K. Integrating PBS, mental health services, and family driven care. In: Sailor W, Dunlap G, Sugai G, Horner R, editors. Handbook of positive behavior support. New York: Springer; 2009. p. 203–31.

    Chapter  Google Scholar 

  5. Mendez LMR, Stone B. Targeted interventions for students: Process and content. In: Mendez LMR, editor. Cognitive behavioral therapy in schools: a tiered approach to youth mental mental health services. New York: Routledge; 2017.

  6. Sugai G, Horner R, Dunlap H, Hieneman M, Lewis TJ, Nelson CM, et al. Applying positive behavioral support and functional behavioral assessment in schools. J Pos Beh Inter. 2000;2:131–43.

    Article  Google Scholar 

  7. Eiraldi R, et al. Implementation of targeted mental health interventions in urban schools: preliminary findings on the impact of training strategy on program fidelity. Evid Based Pract Child Adolesc Ment Health. 2020;5:437–51.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Eiraldi R, et al. Examining effectiveness of group cognitive-behavioral therapy for externalizing and internalizing disorders in Urban Schools. Behav Modif. 2016;40(4):611–39.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Bradshaw CP, Reinke WM, Brown LD, Bevans KB, Leaf PJ. Implementation of school-wide Positive Behavioral Interventions and Supports (PBIS) in elementary schools: observations from a randomized trial. Educ Treat Children. 2008;31(1):1–26.

    Article  Google Scholar 

  10. Bradshaw CP, Waasdorp TE, Leaf PJ. Effects of school-wide positive behavioral interventions and supports on child behavior problems. Pediatrics. 2012;130(5):e1136.

  11. Elias MJ, Zins JE, Graczyk PA, Weissberg RP. Implementation, sustainability, and scaling up of social-emotional and academic innovations in public schools. School Psychol Rev. 2003;32(3):303–19.

    Article  Google Scholar 

  12. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87–108.

    Article  PubMed  Google Scholar 

  13. Han SS, Catron T, Weiss B, Marciel KK. A teacher-consultation approach to social skills training for pre-kindergarten children: treatment model and short-term outcome effects. J Abn Child Psychol. 2005;33(6):681–93.

    Article  Google Scholar 

  14. Andreou TE, McIntosh K, Ross SW, Kahn JD. Critical incidents in sustaining school-wide positive behavioral interventions and supports. J Spec Educ. 2015;49:157–67.

    Article  Google Scholar 

  15. Bambara LM, Nonnemacher S, Kern L. Sustaining school-based individualized positive behavior support. J Posit Behav Interv. 2009;11(3):161–76.

    Article  Google Scholar 

  16. Santangelo R. Collaborative problem solving effectively implemented, but not sustained: a case for aligning the sun, the moon, and the stars. Except Child. 2009;75(2):185–209.

    Article  Google Scholar 

  17. Lyon AR, Frazier SL, Mehta T, Atkins MS, Weisbach J. Easier said than done: intervention sustainability in an urban after-school program. Adm Policy Ment Health. 2011;38(6):504–17.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41(3–4):327–50.

    Article  PubMed  Google Scholar 

  19. Pluye P, Potvinb L, Denis JL. Making public health programs last: conceptualizing sustainability. Eval Progr Planning. 2004;27(2):121–33.

    Article  Google Scholar 

  20. McIntosh K, Kim J, Mercer SH, Strickland-Cohen MK, Horner RH. Variables associated with enhanced sustainability of school-wide positive behavioral interventions and supports. Assess Effect Intervention. 2015;40(3):184–91.

    Article  Google Scholar 

  21. McIntosh K, Mercer SH, Hume AE, Frank JL, Turri MG, Matthews S. Factors related to sustained implementation of schoolwide positive behavior support. Except Children. 2013;79(3):293–311.

    Google Scholar 

  22. McIntosh K, Mercer SH, Nese RNT, Ghemraoui A. Identifying and predicting distinct patterns of implementation in a school-wide behavior support framework. Prev Sci. 2016;17:992–1001.

    Article  PubMed  Google Scholar 

  23. Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Evaluation. 2005;26(3):320–47.

    Article  Google Scholar 

  24. Handler MW, Rey J, Connell J, Thier K, Feinberg A, Putnam R. Practical considerations in creating school-wide positive behavior support in public schools. Psychol Schools. 2007;44(1):29–39.

    Article  Google Scholar 

  25. Sutherland KS, Conroy MA, Vo A, Ladwig C. Implementation integrity of practice-based coaching: preliminary results from the BEST in CLASS efficacy trial. School Ment Health. 2015;7(1):21–33.

    Article  Google Scholar 

  26. Pas ET, Bradshaw CP, Becker KD, Domitrovich C, Berg J, Musci R, et al. Identifying patterns of coaching to support the implementation of the good behavior game: the role of teacher characteristics. School Ment Health. 2015;7(1):61–73.

    Article  Google Scholar 

  27. Cook CR, Lyon AR, Kubergovic D, Wright DB, Zhang Y. A supportive beliefs intervention to facilitate the implementation of evidence-based practices within a multi-tiered system of supports. School Ment Health. 2015;7(1):49–60.

    Article  Google Scholar 

  28. Eiraldi R, et al. Group CBT for externalizing disorders in urban schools: effect of training strategy on treatment fidelity and patient outcomes. Behav Ther. 2018;49(4):538–50.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.

    Article  PubMed  Google Scholar 

  30. Maslach C, Leiter MP, Schaufeli W. Measuring Burnout. In: Cartwright S, Cooper CL, editors. The Oxford Handbook of organizational well being. Oxford, UK: Oxford University Press; 2008.

  31. Goodman R, Ford T, Simmons H, Gatward R, Meltzer H. Using the Strengths and Difficulties Questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. Br J Psychiatry. 2000;177:534–9.

    Article  PubMed  Google Scholar 

  32. Lochman JE, Wells KC, Lenhart L. Coping power child group program: facilitator guide. New York: Oxford University Press; 2008.

    Book  Google Scholar 

  33. Lochman JE, Wells KC. The coping power program for preadolescent aggressive boys and their parents: outcome effects at the 1-year follow-up. J Consult Clin Psychol. 2004;72(4):571–8.

    Article  PubMed  Google Scholar 

  34. Ellis ML, Lindsey MA, Barker ED, Boxmeyer CL, Lochman JE. Predictors of engagement in a school-based family preventive intervention for youth experiencing behavioral difficulties. Prev Sci. 2013;14(5):457–67.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Khanna MS, Eiraldi R, Schwartz B, Kendall PC. CBT for Anxiety Treatment in Schools. Unpublished Manuscript; 2016.

  36. Barrett P. Friends for Life: Group leaders’ manual for children. Sydney: Pathways Health and Research Centre, Australia; 2008.

    Google Scholar 

  37. Kendall PC, Hedtke KA. Cognitive-Behavioral Therapy for Anxious Children: Therapist Manual. Ardmore: Workbook Publishing; 2006.

    Google Scholar 

  38. Crone DA, Hawken LS, Horner RH. Responding to problem behavior in schools: The behavior education program. 2nd ed. New York: Guilford; 2010.

    Google Scholar 

  39. Hunter KK, Chenier JS, Gresham FM. Evaluation of check-in/check-out for students with internalizing behavior problems. J Emot Beh Disorders. 2013;22:135–48.

    Article  Google Scholar 

  40. Newton JS, Todd AW, Algozzine K, Horner RH, Algozzine B. The Team Initiated Problem Solving (TIPS) Training Manual, version 2. Unpublished training manual. Eugene, Oregon: Educational and Community Supports. University of Oregon; 2012.

  41. American Psychological Association (APA). Guidelines for clinical supervision in health service psychology. Am Psychol. 2015;70(1):33–46.

  42. Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. A J Comm Psychol. 2008;41(3–4):171–81.

    Article  Google Scholar 

  43. Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation support for implementing innovations with quality: tools, training, technical assistance, and quality assurance/quality improvement. A J Commun Psychol. 2012;50(3–4).

  44. Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35:255–74.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Lewis TJ, Barrett S, Sugai G, Horner RH. Blueprint for school-wide positive behavior support training and professional development (Version 3). Eugene, Oregon: University of Oregon, Center for Positive Behavioral Interventions and Supports; 2010.

    Google Scholar 

  46. Reynolds CR, Kamphaus RW. Manual for the Behavior Assessment System for Children - Third Edition (BASC-3). Bloomington, MN: Pearson, Psych Corp.; 2015.

  47. Sugai G, Horner R, Lewis-Palmer T, Rossetto Dickey C. PBIS Team Implementation Checklist - Version 3.1 (TIC 3.1). 2011.

  48. Sburlati ES, Schniering CA, Lyneham HJ, Rapee RM. A Model of Therapist Competencies for the Empirically Supported Cognitive Behavioral Treatment of Child and Adolescent Anxiety and Depressive Disorders. Clin Child Fam Psychol Rev. 2011;14(1).

  49. Beidas RS, Mychailyszyn MP, Edmunds JM, Khanna MS, Downey MD, Kendall PC. Training school mental health providers to deliver cognitive-behavioral therapy. School Ment Health. 2012;4:197–206.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Merriam S. The changing landscape of adult learning theory. In: Comings J, Garner B, Smith C, editors. Review of adult learning and literacy: connecting research, policy and practice. Mahwah: Lawrence Erlbaum Associates; 2004. p. 199–220.

    Google Scholar 

  51. Rakovshik SG, McManus F. Establishing evidence-based training in cognitive behavioral therapy: a review of current empirical findings and theoretical guidance. Clin Psychol Rev. 2010;30(5):496–516.

    Article  PubMed  Google Scholar 

  52. Denton CA, Hasbrouck J. A description of instructional coaching and its relationship to consultation. J Edu Psychol Consultation. 2009;19:150–75.

    Article  Google Scholar 

  53. Proctor E, Luke D, Calhoun A, McMillen C, Brownson R, McCrary S, et al. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Imp Sci. 2015;10:88.

    Google Scholar 

  54. Algozzine B, Newton JS, Horner RH, Todd AW, Algozzine K. Development and technical characteristics of a team decision-making assessment tool: decision observation, recording, and analysis (DORA). J Psychoed Assessment. 2012;30(3):237–49.

    Article  Google Scholar 

  55. Stone LL, Otten R, Engels RC, Vermulst AA, Janssens JM. Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4- to 12-yearolds: a review. Clin Child Fam Psychol Rev. 2010;13(3):254–74.

    Article  PubMed  PubMed Central  Google Scholar 

  56. Filter KJ, McKenna MK, Benedict EA, Horner RH. Check in/ check out: a post-hoc evaluation of an efficient, secondary-level targeted intervention for reducing problem behaviors in schools. Educ Treat Children. 2007;30(1):69–84.

    Article  Google Scholar 

  57. McIntosh K, Massar MM, Algozzine RF, George H, Horner R, Lewis TJ, et al. Technical adequacy of the SWPBIS Tiered Fidelity Inventory. J Posit Behav Inter. 2017;19(1):3–13.

    Article  Google Scholar 

  58. Weisz JR, Vaughn-Coaxum RA, Evans SC, Thomassin K, Hersh J, Lee EH, et al. Efficient monitoring of treatment response during youth psychotherapy: the behavior and feelings survey. J Clin Child Adolesc Psychol. 2019;18:1–15.

    Google Scholar 

  59. Skinner EA, Kindermann TA, Furrer CJ. A motivational perspective on engagement and disaffection: conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educat Psychol Meas. 2009;69(3):493–525.

    Article  Google Scholar 

  60. Skinner E, Furrer C, Marchand G, Kindermann T. Engagement and disaffection in the classroom: part of a larger motivational dynamic? J Educ Psychol. 2008;100(4):765–81.

    Article  Google Scholar 

  61. French MT, Dunlap LJ, Zarkin GA, McGeary KA, McLellan AT. A structured instrument for estimating the economic cost of drug abuse treatment. The Drug Abuse Treatment Cost Analysis Program (DATCAP). J Subst Abuse Treat. 1997;14(5):445–55.

  62. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Imp Sci. 2017;12:108.

    Google Scholar 

  63. Maslach C, Jackson SE, Leiter MP. The Maslach Burnout Inventory, 3rd Edition. In: Zalaquett CP, Wood RJ, editors. Evaluating stress: a book of resources. Lanham, MD: Scarecrow Education; 1997. p. 191–218.

    Google Scholar 

  64. Pettegrew LS, Wolf GE. Validating measures of teacher stress. Am Educ Res J. 1982;19:373–96.

    Article  Google Scholar 

  65. Donner A, Klar N. Design and analysis of cluster randomization trials in health research. London: Arnold; 2000.

    Google Scholar 

  66. Sample Size Software. NCSS, LLC, Kaysville, Utah, USA: ncss.com/software/pass. 2014.

  67. Glick HA, Doshi JA, Sonnad SS, Polsky D. Economic evaluation in clinical trials. Oxford: University Press; 2007.

    Google Scholar 

  68. Hayes AF. Introduction to mediation, moderation, and conditional process analysis: a regression-based approach. New York: Guilford Publications; 2018.

    Google Scholar 

  69. Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758–72.

    Article  PubMed  PubMed Central  Google Scholar 

  70. Charmaz K. Grounded theory: objectivist and constructivist methods. Thousand Oaks, CA: Sage; 2000.

    Google Scholar 

  71. Palinkas LA, Aarons GA, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed method designs in implementation research. Adm Policy Men Health. 2011;38(1):44–53.

    Article  Google Scholar 

  72. Creswell J, Klassen A, Plano Clark V, Smith K. Best practices for mixed methods research in the health sciences. Washington, DC: National Institutes of Health; 2011.

    Book  Google Scholar 

Download references

Funding

The study is being funded by the National Institute of Mental Health (NIMH), Award Number 1R01MH122465 to Eiraldi, R. (PI). The content is solely the responsibility of the authors and does not represent the views of NIMH. The funding agency was not involved in the design of the study, data collection, data analysis, interpretation of data, or in writing the manuscript.

Author information

Authors and Affiliations

  1. Children’s Hospital of Philadelphia, Roberts Center for Pediatric Research, 2716 South Street, Room 8293, Philadelphia, PA, 19146-2305, USA

    Ricardo Eiraldi, Quinn A. Rabenau-McDonnell, Rachel Comly, Tara Wilson, Kathryn Henson & Abbas F. Jawad

  2. Department of Pediatrics, University of Pennsylvania Perelman School of Medicine, 3400 Civic Center Boulevard, Philadelphia, PA, 19104, USA

    Ricardo Eiraldi

  3. Philadelphia College of Osteopathic Medicine, 4170 City Avenue, Philadelphia, PA, 19131, USA

    Barry L. McCurdy

  4. OCD and Anxiety Institute, 3138 Butler Pike # 200, Plymouth Meeting, PA, 19462, USA

    Muniya S. Khanna

  5. Department of Psychiatry, University of Pennsylvania Perelman School of Medicine, 3535 Market St., Philadelphia, PA, 19104, USA

    Courtney Benjamin Wolk

  6. Leonard Davis Institute of Health Economics, University of Pennsylvania, 3641 Locus Walk # 210, Philadelphia, PA, 19104, USA

    Courtney Benjamin Wolk & Henry A. Glick

  7. Wharton School, University of Pennsylvania, 3620 Locust Walk, Philadelphia, PA, 19104, USA

    Henry A. Glick

  8. Devereux Center for Effective Schools, 2012 Renaissance, Blvd., King of Prussia, PA, 19406, USA

    Laura E. Rutherford

  9. School District of Philadelphia, 440 North Broad Street, Philadelphia, PA, 19130, USA

    Jayme Banks, Steven A. Rufe & Kristina M. Popkin

  10. Department of Psychology, University of South Carolina-Columbia, Pendleton Street, Barnwell College, Suite #220, Columbia, SC, 29208, USA

    Abraham Wandersman

Authors
  1. Ricardo Eiraldi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barry L. McCurdy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muniya S. Khanna
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Courtney Benjamin Wolk
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Henry A. Glick
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Quinn A. Rabenau-McDonnell
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Rachel Comly
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Laura E. Rutherford
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jayme Banks
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Steven A. Rufe
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Kristina M. Popkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Tara Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Kathryn Henson
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Abraham Wandersman
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Abbas F. Jawad
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

RE conceived the study, drafted the manuscript, and approved all edits. BLM, MSK, CBW, HAG, and AFJ collaborated on the design of the study. AFJ prepared all quantitative analyses. CBW and RC prepared the qualitative analyses. HAG prepared the cost analyses. QARM serves as the project coordinator and IRB regulatory. LER, TW, KH, JB, SAR and KMP contributed to the development of training materials. LER, TW and KH will conduct the training with school personnel. AW is a consultant on this project. All authors reviewed and edited the manuscript. The final version of the manuscript was vetted and approved by all authors. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ricardo Eiraldi.

Ethics declarations

Ethics approval and consent to participate

Informed consent and child assent will be obtained from participants’ parents or legal guardians prior to data collection. Parents/guardians and children will be informed of the purpose of the study, that their participation is voluntary, and that there is no foreseeable risks or harms in the study. The study was approved by Children’s Hospital of Philadelphia, IRB #20-018185 and the School District of Philadelphia, RRC #2021-01-883.

Consent for publication

Consent to publish does not apply to this manuscript. The manuscript does not contain any individual person’s data in any form.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Eiraldi, R., McCurdy, B.L., Khanna, M.S. et al. Study protocol: cluster randomized trial of consultation strategies for the sustainment of mental health interventions in under-resourced urban schools: rationale, design, and methods. BMC Psychol 10, 24 (2022). https://doi.org/10.1186/s40359-022-00733-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40359-022-00733-8

Keywords

BMC Psychology

ISSN: 2050-7283

Contact us