Skip to content

Advertisement

  • Study protocol
  • Open Access
  • Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

A blended psychosocial support program for partners of patients with amyotrophic lateral sclerosis and progressive muscular atrophy: protocol of a randomized controlled trial

  • 1,
  • 2,
  • 3,
  • 4,
  • 5,
  • 1, 6 and
  • 1, 6Email author
BMC Psychology20186:20

https://doi.org/10.1186/s40359-018-0232-5

Received: 6 December 2017

Accepted: 16 April 2018

Published: 2 May 2018

Abstract

Background

Informal caregivers of patients with Amyotrophic Lateral Sclerosis (ALS) or Progressive Muscular Atrophy (PMA) face stressful demands due to severe impairments and prospect of early death of the patients they care for. Caregivers often experience feelings of psychological distress and caregiver burden, but supportive interventions are lacking. The objective of this study is to investigate the effectiveness of a psychosocial support program aimed at enhancing feelings of control over caregiving tasks and reducing psychological distress. This support program is based on an existing program for adult partners of people with cancer and is adapted to meet the needs of ALS caregivers.

Methods

This study is a randomized controlled trial using a wait-list control design. One hundred and forty caregiver-patient dyads, recruited from a nationwide database and through the website of the Dutch ALS Center, will be either randomized to a support program or a wait-list control group. The blended intervention is based on Acceptance and Commitment Therapy and consists of 1 face-to-face contact, 6 online guided modules and 1 telephone contact. The intervention can be worked through in 8 weeks. The effectiveness and the participants’ satisfaction with the intervention will be evaluated using a mixed method design. Caregivers and patients will be asked to fill in questionnaires on 4 occasions during the study: baseline, 3 months, 6 months and 9 months. The main study outcome is the psychological distress of the caregiver assessed with the Hospital Anxiety and Depression Scale. Secondary outcomes are caregiver burden, caregiver quality of life, quality of life of the patient and psychological distress of the patient. Group differences in primary and secondary outcomes at 6 months will be compared with linear mixed model analysis. In a subgroup of caregivers we will explore experiences with the support program through semi-structured interviews. Usage of the online modules will be logged.

Discussion

The study will provide insights into the effectiveness of a blended psychosocial support program on psychological distress of caregivers of patients with ALS or PMA, as well as into indirect relations with patients’ wellbeing.

Trial registration

Netherlands Trial Registry NTR5734, registered 28 March 2016.

Keywords

  • Caregivers
  • Amyotrophic Lateral Sclerosis (ALS)
  • Progressive Muscular Atrophy (PMA)
  • Acceptance and Commitment Therapy
  • Psychological distress

Background

Informal caregivers, usually the patient’s partner, are key figures in Amyotrophic Lateral Sclerosis (ALS) and Progressive Muscular Atrophy (PMA) care. They provide the majority of support to patients and are often the primary caregivers. ALS and PMA are fatal motor neuron diseases, ALS affecting both upper and lower motor neurons while PMA only affects lower motor neurons. Informal caregivers of patients with ALS or PMA face stressful demands due to the prospect of an early death and severe impairments of the patient. Patients experience a progressive decline of muscle strength resulting in paralysis, difficulty with speech and swallowing, possible cognitive and behavioral problems and ultimately, respiratory failure leading to death [13]. Patients become increasingly impaired and the amount of care that is needed accumulates [4].

Since PMA is a rare subtype of motor neuron disease, research studies on PMA caregivers are limited. However, PMA shows substantial overlap with ALS and is considered to be a form of ALS [5]; PMA caregivers are, therefore, likely to struggle with the same issues as ALS caregivers. The wellbeing of ALS caregivers has been studied more intensively and shows that as the disease progresses, ALS caregivers experience heightened feelings of psychological distress and burden [68], which is related to a diminished quality of life [9]. The wellbeing of ALS caregivers is critical because a high level of burden might predict a breakdown in care, leading to earlier placement of the patient in a care-home or hospice [10].Therefore, improving the psychological health of the caregivers may not only improve their quality of life but also that of the patient.

Previous research has shown that psychological distress and feelings of burden of ALS caregivers are associated with disease characteristics of the patient (i.e. physical and behavioral problems) but also with characteristics of the caregivers themselves, such as their coping style or whether they find positive meaning in caregiving [7, 9, 1115]. As the disease progresses, psychological and physical demands on the ALS caregiver increase. Patients become increasingly reliant on their caregiver, and caregivers have to take over responsibilities from the patient. Handling all these responsibilities, accepting a loved one’s illness and accepting the loss of the patient in the near future are examples of issues ALS caregivers struggle with [16].

ALS and PMA caregivers are faced with situations, yet may lack the relevant knowledge and skills, such as communicating about the disease and death, dealing with the patients’ behavioral changes, dealing with their own emotions or expressing their own boundaries [17]. Consequently, caregivers may not feel competent or in control with respect to their caregiving tasks, while the demands increase. From previous studies we know that a combination of high demands and feelings of insufficient control over caregiving is associated with poorer physical and psychological health outcomes of caregivers [18, 19]. Previous studies have indicated that there is a need for psychosocial interventions for caregivers, but such interventions are still lacking [6, 20, 21].

Acceptance and Commitment Therapy (ACT) is a form of cognitive behavioral therapy that encourages individuals to accept unwanted private events which are out of personal control (such as thoughts, feelings and memories) and to identify important values in life in order to engage in committed action to pursue these values [22]. The acceptance component in ACT makes this therapy valuable in contexts with circumstances that cannot be changed [23, 24], for instance, receiving a diagnosis of ALS or PMA. The values component supports caregivers to undertake action that is personally meaningful. This can assist them in adjusting to their situation, in moving on in life and in enhancing their psychological wellbeing [23]. Applying acceptance strategies and living up to personal values requires a different way of responding to situations and may increase the feeling of control [25].

ACT has proven to be effective in decreasing feelings of psychological distress in various target groups, including caregivers of other patient populations [2628]. Recently, ACT has also been proved to be effective when delivered via the internet [28, 29]. Since ALS and PMA caregivers are often preoccupied with the care for their home-bound patient, receiving care in a more accessible and time efficient manner may offer opportunities.

In this study, we will investigate the effect of a blended support program in which face-to-face contact and e-health will be combined. The support program is based on Acceptance and Commitment principles and focuses particularly on the needs of ALS and PMA caregivers. This support program aims to diminish caregivers’ psychological distress by increasing their feelings of control in fulfilling the caregiving tasks for patients with ALS or PMA.

Methods

The described protocol (Version 6, dated 27-07-2017) has been developed according to the Standard Protocol Items Recommendations for Interventional Trials (SPIRIT) and the Template for Intervention Description and Replication (TIDieR) [30, 31].

Design

This study is a randomized controlled trial in which caregiver-patient dyads will be randomly allocated to one of two groups:
  1. 1.

    Intervention group (support program during 8–12 weeks).

     
  2. 2.

    Wait-list control group, receiving care as usual (6-month monitoring preceding the support program).

     
This design enables us to investigate whether offering a support program in addition to usual care improves the wellbeing of caregivers compared to care as usual. Both caregiver and patient will be asked to complete online questionnaires at baseline (T0), 3 months after baseline (T1), 6 months after baseline (T2) and 9 months after baseline (T3), but only the caregivers will receive the support program. In Fig. 1, the flowchart of the study is presented. In a subgroup of caregivers, we will explore experiences with the support program through interviews.
Figure 1
Fig. 1

Flowchart study

Study population

Caregiver-patient dyads will be recruited through a nationwide ALS/PMA database and via the website of the Dutch ALS Center. The study will also be announced on websites of ALS/PMA patient associations. Patient and inclusion criteria are 1) the caregiver is the partner of the ALS or PMA patient; 2) the caregiver is 18 years or older; 3) caregiver and patient are proficient in Dutch to fill out the questionnaires; 4) caregiver and patient have internet access. When patients are not able or not willing to complete online questionnaires, caregivers are still eligible to join the support program provided the patient consents. The inclusion criteria remain in force.

Sample size

The sample size calculation was based on Hospital Anxiety Depression Scale (HADS) data of informal caregivers in a previously completed study [6]. The total number of caregiver-patient dyads needed to detect a clinically relevant difference [32] of 3.65 points, with a standard deviation of 7.3 between the groups at T2, with an alpha of 0.05 and a power of 80%, is 116 caregiver-patient dyads (58 dyads in each group). Taking into account an attrition rate of 20%, we aim to include 140 caregiver-patient dyads in total.

Procedure

Caregiver-patient dyads recruited via the national ALS/PMA database will be contacted by telephone. In this telephone call, the dyad will receive information about the study. When dyads are interested in participating, the inclusion criteria will be checked. Eligible dyads receive the study information letter per postal mail. One week after sending the information letter, caregivers will be contacted by telephone. Caregivers who do not want to participate will be asked for their reasons for not participating and we will assess their burden of caregiving with the one item, Self-Rated Burden scale (SRB) [33].

Caregiver-patient dyads can find information about the research and can apply to participate on the Dutch ALS Center website. Thereafter, the researcher will send the research information letter. One week after sending the information letter, caregivers will be contacted by telephone to answer questions and to check the inclusion criteria.

Dyads are asked to return the signed informed consents by postal mail. Once the informed consents have been received, participants are sent an invitation via e-mail to fill out the first assessment (T0).

Randomization

After completing the first assessment, dyads will be randomized into the wait-list control or experimental condition by the researchers according to a computerized programmed randomization scheme. Randomization will be stratified for the degree of functional impairment of the patient (Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised, using the cut off score for severe disabilities ≥ 24) [34, 35], presence of behavioral problems of the patient (Amyotrophic Lateral Sclerosis-Frontotemporal Dementia-Questionnaire, using the cut off score for mild behavioral changes ≥ 22) [36] and gender of the caregiver.

The intervention

The content of the support program for informal caregivers is based on an online intervention aimed at partners of patients with cancer [37, 38]. This intervention, based on ACT, was adapted to the specific needs of ALS caregivers. We interviewed 21 ALS caregivers about their support needs [17] and added information and exercises related to these needs to the support program. Next, we asked 6 ALS caregivers and professionals in ALS/PMA care and research (physicians, psychologists and researchers) to provide feedback on the content of the program. Based on their feedback, text materials were adjusted and the web-based application was developed. In a usability test, five partners were observed while using the web-based application and they were asked to evaluate it. Their feedback was used to improve the usability of the web-based application.

The support program consists of an introductory face-to-face appointment with a psychologist, 6 psychologist-guided online modules and one closing telephone contact with the psychologist. The total program can be completed in 8 weeks. If caregivers need more time due to personal circumstances, they have the opportunity to work through the total program in 12 weeks. The content of the support program, the topics and the goals of each part are presented in Table 1.
Table 1

Content intervention

Part of intervention

Topics

Goals

Key components

Face-to-face session

• The care situation

• Wellbeing caregiver

• Information about support program

• Log in online modules

To receive information about the care situation and establish a relationship between the psychologist and the caregiver. To inform about caregiver burden and start online modules.

• Psychoeducation

Online module 1

Coping with your emotions and thoughts

• Dealing with and expressing emotions

• Recognizing thoughts

To recognize emotions and encourage caregivers to allow, express and share emotions that can arise. To recognize dysfunctional thoughts and rumination. Change the way the caregiver relates to thoughts/to create distance from thoughts.

• Acceptance

• Cognitive defusion

• Mindfulness

Online module 2

The art of communication

• Communication style

• Communication about sensitive topics

• Communication about providing care

To improve the overall communication and to communicate with the patient about sensitive topics and providing care in the future.

• Communicating about what really matters

• Mindfulness

Online module 3

Your resilience plan

• Dealing with continuous stress

• Moments of relaxation

• Using your sources

To make a resilience plan that may allow caregivers to maintain health during this stressful period by taking care of themselves.

• Acceptance

• Mindfulness

Online module 4

What is really important

• Values in relationship

• Values in life

To identify the values of the caregiver in different areas of life and to plan action to meet these values.

• Values

• Commited action

• Mindfulness

Online module 5

Moments of joy

• Positivity during difficult times

• Celebrate the relationship

To seek, enjoy and cherish the positive moments in the relationship and in life.

• Committed action

• Mindfulness

Online module 6

A good last period

• Life story of the patient

• Communication in this last phase

• Beautiful memories

• Being grateful

To create a beautiful last period with the loved ones and to make memories with the patient for the future.

• Acceptance

• Communicating about what really matters

• Committed action

• Mindfulness

Telephone call

• Any questions

• Finish the support program

To offer support with regard to any issues and close the support program.

 

Face-to-face session

Participants will have a one-hour session with a psychologist before they start with the online modules at the residence of the caregiver. The session is protocolized: the psychologist briefly explains the purpose of the intervention, receives information about the caregiving situation, logs in and demonstrates the online program and establishes a working relationship with the participant. Following this session, the caregiver starts with the online modules.

Online modules

The online part consists of 6 online modules, each module is directed at a specific theme. All modules start with an introduction directed at the theme of the module, followed by psychological exercises. The content of the online modules is focused on the following key components: 1] acceptance (embracing the private events without unnecessary attempts to change them [39]), 2] values (identifying valued domains of life [40]), 3] committed action (actions to pursue one’s values [40]), 4] mindfulness (training conscious awareness and attention from one moment to the next moment [41]), 5] communication about what really matters, 6] cognitive defusion (change the way one interacts with or relates to thoughts by altering the contexts in which they occur [39]). Participants also receive practical information, tips and references to relevant websites, organizations and other sources of information and support associated with the theme of the module. They are able to get in contact with other participating caregivers of patients with ALS or PMA, using the online program. They have an online personal profile and can send each other private messages. Participants can also share tips and advice with fellow participants.

The same psychologist who visited the participant for the face-to-face session provides online feedback including feedback on the completed exercises, a reflection on the progress of the participant and a reaction to any questions or difficulties.

Telephone contact

The program ends with a telephone call with the psychologist. During this call, the caregiver can ask for advice for specific problems and discuss questions that came up after completing the last module.

Guidance

The support will be provided by psychologists who are trained to provide the intervention.

Assessments

All quantitative assessments are self-report measures and will be administered online. Overviews of the questionnaires for caregivers and patients and their time of assessment are provided in Tables 2 and 3. Participants who discontinue the intervention will be asked to complete study follow-up assessments. Semi-structured qualitative interviews will be conducted by telephone.
Table 2

Measurement overview caregivers

  

Measurementa

Outcome

Instrument

T0

T1

T2

T3

Socio-demographics

 Caregiver, patient, and care characteristics

iMTA Valuation of Informal Care Questionnaire

x

   

Primary outcome

 Psychological distress

Hospital Anxiety and Depression Scale

x

x

x

x

Secondary outcomes

 Quality of life

Care-related Quality of Life −7 + Care-related Quality of Life - VAS

x

x

x

x

 Burden

Zarit Burden Interview + Self-Rated Burden scale

x

x

x

x

Mediator

 Self-efficacy

Revised scale for caregiving self-efficacy

x

x

x

x

Covariates

     

 Satisfaction with relationship

Satisfaction Questionnaire

x

x

x

x

 Social Support

Multidimensional Scale of Perceived Social Support

x

x

x

x

 Behavioral changes patient

Amyotrophic Lateral Sclerosis-Frontotemporal Dementia- Questionnaire

x

x

x

x

 Physical functioning patient

Amyotrophic Lateral Sclerosis Functional Rating Scale- Revised

x

x

x

x

Evaluation

 Evaluation intervention

Client Satisfaction Questionnaire + Self developed scale

 

xb

 

xc

aT0 = Baseline, T1 = 3 months, T2 = 6 months, T3 = 9 months

bonly for the intervention group

conly for the wait-list control group

Table 3

Measurement overview patients

  

Measurementa

Outcome

Instrument

T0

T1

T2

T3

Secondary outcomes

 Quality of life

McGill Single Item Scale

x

x

x

x

Psychological distress

Hospital Anxiety and Depression Scale

x

x

x

x

Covariates

 Self-perceived as burden

Self-perceived Burden Scale - 1 item

x

x

x

x

aT0 = Baseline, T1 = 3 months, T2 = 6 months, T3 = 9 months

Primary outcomes measure

Caregivers’ psychological distress

Psychological distress will be measured using the HADS [42, 43]. This scale consists of 14 items reflecting symptoms of anxiety and depression by 7 items each. Items are scored on a 4-point scale and total scores range from 0 to 42. Furthermore, a total score for the subscales depression and anxiety can be calculated. The internal consistency for the total scale and both subscales is sufficient to high (Cronbach’s alpha ranging from .71 to .90). The test-retest reliability for the total scale and both subscales proved to be high (correlation coefficient ranging from .86 to .91) [42, 44].

Secondary outcomes measures

Caregiver burden

The Zarit Burden Interview (ZBI) will be used to measure caregiver burden by evaluating disease impact on caregivers’ quality of life, psychological suffering and impact on social and family relationships [45]. We will use a short version of 12 items, which has shown to have comparable psychometric properties to the full version that consists of 22 items [46]. The ZBI contains a 0–4 point scoring system with the following answering options: never, rarely, sometimes, quite frequently and nearly always. The questionnaire yields a maximum score of 48. A score ≥ 17 indicates a high burden. The ZBI short form shows good validity, internal consistency, and discriminative ability [47].

Burden of caregiving will additionally be measured with the Self-Rated Burden scale (SRB) [33]. The SRB is a single question in which informal caregivers are asked to give an overall assessment of the burden they experience from caring by using a visual analogue scale. The scores range between ‘0’ (not at all straining) and ‘10’ (much too straining). The SRB is a valid and reliable question and it can be used for a quick screening of caregivers at risk [33].

Caregivers' quality of life

Caregivers’ quality of life will be assessed using the Care-related Quality of Life (CarerQoL) [48]. The CarerQoL combines a description of the burden of caregiving on seven care dimensions (CarerQoL-7) with a valuation component (CarerQoL-VAS) assessing general quality of life in terms of happiness. The CarerQoL-7 provides answering categories ‘none’ (1), ‘some’ (2), and ‘many’ (3). The CarerQoL-VAS contains 0, ‘completely unhappy’ and 10 ‘completely happy’ as endpoints. The psychometric properties of the CarerQol were shown to be satisfactory [4850].

Patients’ quality of life

Patients’ self-rated quality of life will be measured using the McGill Quality of Life Questionnaire (MQOL) [51], which is designed to measure the quality of life of patients with a terminal illness. In order to burden the patients as little as possible, we will only use the single item scale which assesses the overall quality of life with answer scores ranging from 0 = very bad to 10 = excellent.

Patients’ psychological distress

Patients’ psychological distress will be measured with the HADS.

Mediator

Caregivers’ self-efficacy

Caregivers’ beliefs about their capacity to carry out caregiving tasks will be measured using the Revised Scale for Caregiving Self-Efficacy [52]. The original version of the instrument consisted of 15 items within 3 subscales; self-efficacy for obtaining respite, responding to disruptive patient behaviors, and controlling upsetting thoughts about caregiving. The disruptive patient behaviors scale is not suitable for our target population and is omitted.

Caregivers are asked to indicate on a scale of 0 (absolutely cannot do) to 100 (certainly can do) how confident they are with respect to items such as “how confident are you that you can control worrying about future problems that might come up with [patient]”. All subscales demonstrate strong internal consistency and adequate test-retest reliability [52]. We added 3 additional questions based on the Job Content Questionnaire aimed at the control that caregivers perceive over fulfilling the caregiver tasks [53].

Covariates

Caregivers’ social support

Caregivers’ experience of social support will be measured using the Multidimensional Scale of Perceived Social Support (MPSS) [54]. The MPSS consists of 12 items and is aimed at different sources of social support (family, friends, and significant others). The items are scored on a 7-point Likert scale ranging from 1 (very strongly disagree) to 7 (very strongly agree). The total score is calculated by adding up the scores of all items, resulting in a range of 12–84. A higher score indicates stronger social support, with scores ≥ 79 corresponding to an experience of strong support. MPSS has proven to be a psychometrically valid instrument, with good test-retest reliability and adequate validity among varying populations [5456].

Caregivers’ satisfaction with relationship

Caregivers’ satisfaction with the relationship with the patient will be assessed using the Satisfaction Scale [57]. The questionnaire consists of 4 satisfaction items which are rated on a scale ranging from 1 (not satisfied) to 5 (satisfied). A total score is calculated by adding up the scores of the 4 items, with a higher score indicating more satisfaction with the relationship. The items refer to caregivers’ experience during the last month. The satisfaction scale shows reasonable internal consistency [58].

Patients’ behavioral changes

Behavioral changes in patients will be assessed with the Amyotrophic Lateral Sclerosis-Frontotemporal Dementia-Questionnaire (ALS-FTD-Q) [36]. The questionnaire asks the caregiver to compare the patient’s current behavior with his/her behavior 3 years ago. It consists of 25 items with a total score range of 0–100 (≥ 22 indicating mild behavioral changes and ≥ 29 corresponding to significant behavioral changes). The ALS-FTD-Q shows good internal consistency (Cronbach’s  alpha = 0.92) as well as construct validity [36].

Patients’ physical functioning

The physical functioning of patients will be assessed using the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALS-FRS-R) [35]. The scale consists of 12 items with 0–4 point scores in order to measure limb, bulbar, and respiratory dysfunction. An example is the item “Walking” with answer scores 0 = normal to 4 = is unable to consciously move legs. Overall scores range from 0 to 48, with higher scores indicating better physical functioning. The ALS-FRS-R demonstrates strong internal consistency as well as construct validity [35]. This questionnaire will be completed by the caregiver.

Patients’ perception of being a burden

Patient’s own feelings of being a burden for the caregiver will be measured using one item of the Self Perceived Burden Scale (SPBS): “I feel that I am a burden to my caregiver” [59]. This statement is rated on a scale of how often patients feel this way, from “none of the time” (1) to “all of the time” (5). Higher scores indicate that the patients perceived themselves to cause a higher burden to their caregivers.

Evaluation of the intervention

Satisfaction with received support

To measure the satisfaction of the caregiver for the support they received, the 8-item Client Satisfaction Questionnaire (CSQ-8) is used [60]. All items are scored on a 4-point scale ranging from 1 to 4. Response options differ from item to item. An example is “How satisfied are you with the amount of help you have received?” (for which the response options range from 1 = “Quite dissatisfied” to 4 = “Very satisfied”). An overall score is calculated by summing and ranges from 8 to 32, with higher scores indicating greater satisfaction. The Dutch translated version of the questionnaire shows high internal consistency (.91) [61].

Evaluation support program

Additionally, a scale to evaluate the intervention was developed. The participant is asked to rate the intervention in general and the different components of the intervention such as the psychological exercises, contact with the psychologist who provided the feedback and contact with other informal caregivers. Participants are asked to rate every component on a 0–10 scale; the questionnaire consists of 9 questions.

Experiences with support program

Semi-structured interviews to explore the experiences of the caregivers with the support program will be carried out after subjects complete the support program. Participants will be selected via purposive sampling on demographic variables (age, sex, disease stage patient). Interviews will be held by a researcher, using a topic list with the following topics: experiences with support program, user-friendliness, use of the support program, valuable, missing and redundant elements of the support program and recommendations for change. The interviews will last approximately 1 h and will be recorded. Participants will be included until data saturation is reached.

Demographics and description of the care situation

Demographics and care situation

The iMTA Valuation of Informal Care Questionnaire (iVICQ) is a questionnaire which facilitates an accurate description of providing informal care and its effects on informal caregivers [62]. We used the sections of the background information of patients and caregivers, the informal care situation and questions to economically validate informal care as a directory for our questions regarding these subjects.

Questions to assess the working mechanism of the support program

The support program aims to improve feelings of control over caregiving and reduce psychological distress. Therefore, at the end of every online module, the caregiver is asked two questions about “feelings of control over executing caregiving tasks” and “the level of distress they experience”, at that moment on a VAS scale [63].

Monitoring adherence to the intervention modules

In order to assess the use of and the adherence to the online modules we will collect log data of the participants such as the frequency of logging in, the duration of logging in, which parts of the modules are downloaded and which functions are used.

Data management

All personal data will be coded, removed from the data for analysis and stored separately. Only designated research staff will have access to the keys linking the data with the personal information. The research team will have access to the final dataset. Data management and monitoring of the trial will be performed by qualified personnel according to standard operation procedures of the Brain Center Rudolf Magnus, University Medical Center Utrecht.

Analyses

Statistical analyses

Descriptive statistics will be used to report demographic variables, clinical outcomes, and the use of the different modules. Group differences in primary and secondary outcomes will be compared with linear mixed model analyses, in which the mediator and covariates will be included. Statistical analyses will be performed primarily according to intention-to-treat and secondarily according to per-protocol principles. The intention to treat analyses will include data of all included caregivers, regardless of their adherence to the intervention or their missing data. In the per-protocol analyses we will only include caregivers who completed at least 4 modules (66.7%) and the T2 measurement. All hypotheses will be tested 2-sided, with a critical value of 0.05. Effect sizes on the primary outcome variable (HADS total) will be calculated with Cohen’s D using the means and pooled standard deviations of the two groups.

Interview analyses

Interviews will be transcribed and analyzed thematically [64]. The texts will be broken down into fragments based on content and fragments will be labeled with a code using NVIVO 10 [65, 66]. Once the coding of all interviews has been completed, codes will be sorted according to similarities and overarching themes and subthemes will be identified.

Discussion

To our knowledge, this will be the first study to evaluate a blended support program for caregivers of ALS and PMA patients. The program is aimed at enhancing feelings of control over caregiving tasks using ACT principles. Previous research on ALS caregiving revealed increasing levels of psychological distress in caregivers, a lack of existing interventions and an urgent need for support [7, 21]. ACT interventions have proven to reduce psychological distress in other caregiver populations [27, 28] and are valuable in contexts with circumstances that cannot be changed [23, 24].

A strength of this intervention is the blended approach: face-to-face support in combination with online support. Due to the many hours ALS and PMA caregivers spend on providing care, they often experience a lack of personal time [17], which reduces the opportunity to access traditional forms of support. Therefore, the blended approach may provide support in a more time-efficient manner, as caregivers can access information and exercises any time at home via an online platform.

Although previous research has provided information on factors associated with psychological distress and burden [e.g. 15], the underlying process is still unclear due to a gap in research on personal factors related to the caregiver. The use of a theoretical framework is considered as another strength since it helps to gain insight into whether the demand-control theory is applicable to the caregiver situation in ALS and PMA [18] and will provide knowledge on the influence of factors such as control and mastery in relation to psychological distress and burden. This will provide information to understand how, when and for whom the intervention will be effective.

Further, caregivers and healthcare professionals were involved in both the development process of the intervention and the design of the study. Due to their involvement, we were able to develop an intervention that meets the needs and wishes of caregivers and includes the most important themes according to professionals. Caregivers and professionals will also be involved in the next steps of the research such as the recruitment and the dissemination of the results of the study. Previous studies indicated that engaging the target group increases study enrollment and may enhance the uptake and the acceptance of interventions [67, 68].

Another strength of the study design is the mixed method approach; questionnaires and interviews will be used to evaluate the program which enables a throughout evaluation and may lead to further improvement of the support program.

The support program may also have some weaknesses. First of all, the online part of the support program might be an obstacle for some caregivers due to a lack of information and communication technology literacy. These caregivers might prefer to receive traditional face-to-face support.

Another limitation might be that the intervention is only focused on partners, which means that primary caregivers who have another type of relationship with the patient are excluded while they might be in need of support. Once the intervention has proven to be effective, it might be worthwhile developing an adapted version for primary caregivers with other relationships to the patient.

A limitation of the study design might also be contamination with care as usual. In the last couple of years, the value and the importance of the social environment of patients has been emphasized. This has led to a stronger focus on caregivers in standard care; care facilities are encouraged to involve caregivers in their care plans. Due to this recent shift, the support for caregivers might have improved and it may be more difficult to demonstrate a significant difference when we compare care as usual with our support program. However, if caregivers perceive the support program as being more user-friendly than care as usual, this will encourage its use in standard care.

To conclude, this study will provide insight into the effects of a blended support program for informal caregivers of patients with ALS and PMA by targeting feelings of control over caregiving tasks using ACT principles. The program could potentially benefit caregivers, and might affect patients’ wellbeing indirectly.

Dissemination plan

Results of this study will be published in international, peer-reviewed journals and presented at relevant conferences/congresses, both national as well as international. Results will furthermore be communicated through national publications and published on relevant websites.

Abbreviations

ACT: 

Acceptance and Commitment Therapy

ALS: 

Amyotrophic Lateral Sclerosis

ALS-FRS-R: 

Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised

ALS-FTD-Q: 

Amyotrophic Lateral Sclerosis-Frontotemporal Dementia-Questionnaire

CarerQoL: 

Care Related- Quality of Life

CSQ-8: 

8-item Client Satisfaction Questionnaire

HADS: 

Hospital Anxiety and Depression Scale

iVICQ: 

institute for Medical Technology Assessment Valuation of Informal Care Questionnaire

MPSS: 

Multidimensional Scale of Perceived Social Support

MQOL: 

McGill Quality of Life Questionnaire

PMA: 

Progressive Muscular Atrophy

SPBS: 

Self Perceived Burden Scale

SPIRIT: 

Standard Protocol Items Recommendations for Interventional Trials

SRB: 

Self-Rated Burden scale

TIDieR: 

Template for Intervention Description and Replication

ZBI: 

Zarit Burden Interview

Declarations

Acknowledgments

We kindly acknowledge the following people for their valued contributions to the development of the intervention: the caregivers and professionals who provided feedback and the patient association ALS Patients Connected. We also would like to thank Dr. Stavros Nikolakopoulos for his statistical advice.

Funding

The authors are grateful to the Netherlands ALS Foundation for funding of this study. The Netherlands ALS Foundation did not influence study design and will not play an influential role in data analysis or interpretation.

Availability of data and materials

The datasets generated and analyzed during the current study will be available from the author upon reasonable request.

Current status

Recruitment began on August 10, 2017.

Authors’ contributions

JW, AB, LHB, JMAVM and CDS participated in the design of this study protocol. CHCD developed the original version of the intervention for partners of patients with cancer. JW and CDS adapted the program to the needs of ALS and PMA caregivers. RK and CHCD provided feedback on the content of the intervention and on the study protocol. All authors read and approved the final manuscript.

Ethics approval and consent to participate

This study has been approved by the Medical Ethics Committee of the University Medical Center Utrecht under file number 16–273-D. Participation is voluntary and all respondents will provide written informed consent before inclusion. Any amendments to the protocol will be submitted to and approved by the Medical Ethics Committee of the University Medical Center Utrecht.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Center of Excellence in Rehabilitation Medicine, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht University and De Hoogstraat Rehabilitation, Utrecht, the Netherlands
(2)
Department of Rehabilitation, Academic Medical Center Amsterdam, Amsterdam, The Netherlands
(3)
Department of Psychology, Health and Technology, University of Twente, Enschede, The Netherlands
(4)
Patient Association ALS Patients Connected, Rotterdam, The Netherlands
(5)
Department of Neurology, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht, The Netherlands
(6)
Department of Rehabilitation, Physical Therapy Science & Sports, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

References

  1. Kiernan MC, Vucic S, Cheah BC, Turner MR, Eisen A, Hardiman O, et al. Amyotrophic lateral sclerosis. Lancet. 2011;377(9769):942–55.View ArticlePubMedGoogle Scholar
  2. Liewluck T, Saperstein DS. Progressive muscular atrophy. Neurol Clin. 2015;33(4):761–73.View ArticlePubMedGoogle Scholar
  3. Raaphorst J, Beeldman E, De Visser M, De Haan RJ, Schmand B. A systematic review of behavioural changes in motor neuron disease. Amyotroph Lateral Scler. 2012;13(6):493–501.View ArticlePubMedGoogle Scholar
  4. Chio A, Gauthier A, Vignola A, Calvo A, Ghiglione P, Cavallo E, et al. Caregiver time use in ALS. Neurology. 2006;67(5):902–4.View ArticlePubMedGoogle Scholar
  5. Kim WK, Liu X, Sandner J, Pasmantier M, Andrews J, Rowland LP, et al. Study of 962 patients indicates progressive muscular atrophy is a form of ALS. Neurology. 2009;73(20):1686–92.View ArticlePubMedPubMed CentralGoogle Scholar
  6. Creemers H, Veldink JH, Grupstra H, Nollet F, Beelen A, van den Berg LH. Cluster RCT of case management on patients' quality of life and caregiver strain in ALS. Neurology. 2014;82(1):23–31.View ArticlePubMedGoogle Scholar
  7. Goldstein LH, Atkins L, Landau S, Brown R, Leigh PN. Predictors of psychological distress in carers of people with amyotrophic lateral sclerosis: a longitudinal study. Psychol Med. 2006;36(6):865–75.View ArticlePubMedGoogle Scholar
  8. Gauthier A, Vignola A, Calvo A, Cavallo E, Moglia C, Sellitti L, et al. A longitudinal study on quality of life and depression in ALS patient-caregiver couples. Neurology. 2007;68(12):923–6.View ArticlePubMedGoogle Scholar
  9. Rabkin JG, Wagner GJ, Del Bene M. Resilience and distress among amyotrophic lateral sclerosis patients and caregivers. Psychosom Med. 2000;62(2):271–9.View ArticlePubMedGoogle Scholar
  10. Moore MK. Dying at home: a way of maintaining control for the person with ALS/M N D. Palliative Med. 1993;7(Suppl 2):65–8.View ArticleGoogle Scholar
  11. Chio A, Vignola A, Mastro E, Giudici AD, Iazzolino B, Calvo A, et al. Neurobehavioral symptoms in ALS are negatively related to caregivers' burden and quality of life. Eur J Neurol. 2010;17(10):1298–303.View ArticlePubMedGoogle Scholar
  12. Murphy V, Felgoise SH, Walsh SM, Simmons Z. Problem solving skills predict quality of life and psychological morbidity in ALS caregivers. Amyotroph Lateral Scler. 2009;10(3):147–53.View ArticlePubMedGoogle Scholar
  13. Pagnini F, Rossi G, Lunetta C, Banfi P, Castelnuovo G, Corbo M, et al. Burden, depression, and anxiety in caregivers of people with amyotrophic lateral sclerosis. Psychol Health Med. 2010;15(6):685–93.View ArticlePubMedGoogle Scholar
  14. Pagnini F, Phillips D, Bosma CM, Reece A, Langer E. Mindfulness as a protective factor for the burden of caregivers of amyotrophic lateral sclerosis patients. J Clin Psychol. 2016;72(1):101–11.View ArticlePubMedGoogle Scholar
  15. Creemers H, de Moree S, Veldink JH, Nollet F, van den Berg LH, Beelen A. Factors related to caregiver strain in ALS: a longitudinal study. J Neurol Neurosur PS. 2015;0:1–7.Google Scholar
  16. Weisser FB, Bristowe K, Jackson D. Experiences of burden, needs, rewards and resilience in family caregivers of people living with motor Neurone disease/amyotrophic lateral sclerosis: a secondary thematic analysis of qualitative interviews. Palliat Med. 2015;29(8):737–45.View ArticlePubMedGoogle Scholar
  17. de Wit J, Schröder, CD, el Mecky, J, Beelen, A , van den Berg, LH & Visser-Meily, JMA. Support needs of caregivers of patients with amyotrophic lateral sclerosis: a qualitative study. Palliat Support Care. 2018:0:1–7.Google Scholar
  18. Molloy GJ, Johnston DW, Johnston M, Gao C, Witham MD, Struthers AD, et al. Using the demand-control model of job strain to predict caregiver burden and caregiver satisfaction in the informal caregivers of heart failure patients. Br J Health Psychol. 2008;13(3):401–17.View ArticlePubMedGoogle Scholar
  19. Orbell S, Gillies B. What’s stressful about caring? J Appl Soc Psychol. 1993;23:272–90.View ArticleGoogle Scholar
  20. van Groenestijn AC, Schroder CD, Visser-Meily JM, Reenen ET, Veldink JH, van den Berg LH. Cognitive behavioural therapy and quality of life in psychologically distressed patients with amyotrophic lateral sclerosis and their caregivers: results of a prematurely stopped randomized controlled trial. Amyotroph Lateral Scler Frontotemporal Degener. 2015;16(5–6):309–15.View ArticlePubMedGoogle Scholar
  21. Aoun SM, Bentley B, Funk L, Toye C, Grande G, Stajduhar KJ. A 10-year literature review of family caregiving for motor neurone disease: moving from caregiver burden studies to palliative care interventions. Palliat Med. 2013;27(5):437–46.View ArticlePubMedGoogle Scholar
  22. Hayes SC, Strosahl K, Wilson KG. Acceptance and commitment therapy: an experiential approach to behavior change. New York: Guilford; 1999.Google Scholar
  23. Davis EL, Deane FP, Lyons GC. Acceptance and valued living as critical appraisal and coping strengths for caregivers dealing with terminal illness and bereavement. Palliat Support Care. 2015;13(2):359–68.View ArticlePubMedGoogle Scholar
  24. Graham CD, Weinman J, Sadjadi R, Chalder T, Petty R, Hanna MG, et al. A multicentre postal survey investigating the contribution of illness perceptions, coping and optimism to quality of life and mood in adults with muscle disease. Clin Rehabil. 2014;28(5):508–19.View ArticlePubMedGoogle Scholar
  25. Arch JJ, Craske MG. Acceptance and commitment therapy and cognitive behavioral therapy for anxiety disorders: different treatments, similar mechanisms? Clin Psychol-Sci Pr. 2008;15(4):263–79.View ArticleGoogle Scholar
  26. A-Tjak JG, Davis ML, Morina N, Powers MB, Smits JA, Emmelkamp PM. A meta-analysis of the efficacy of acceptance and commitment therapy for clinically relevant mental and physical health problems. Psychother Psychosom. 2015;84(1):30–6.View ArticlePubMedGoogle Scholar
  27. Losada A, Marquez-Gonzalez M, Romero-Moreno R, Mausbach BT, Lopez J, Fernandez-Fernandez V, et al. Cognitive-behavioral therapy (CBT) versus acceptance and commitment therapy (ACT) for dementia family caregivers with significant depressive symptoms: results of a randomized clinical trial. J Consult Clin Psychol. 2015;83(4):760–72.View ArticlePubMedGoogle Scholar
  28. Burke K, Muscara F, McCarthy M, Dimovski A, Hearps S, Anderson V, et al. Adapting acceptance and commitment therapy for parents of children with life-threatening illness: pilot study. Fam Syst Health. 2014;32(1):122–7.View ArticlePubMedGoogle Scholar
  29. Buhrman M, Skoglund A, Husell J, Bergstrom K, Gordh T, Hursti T, et al. Guided internet-delivered acceptance and commitment therapy for chronic pain patients: a randomized controlled trial. Behav Res Ther. 2013;51(6):307–15.View ArticlePubMedGoogle Scholar
  30. Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gotzsche PC, Krleza-Jeric K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7.View ArticlePubMedPubMed CentralGoogle Scholar
  31. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.View ArticlePubMedGoogle Scholar
  32. Vanhoutte EK, Faber CG, Merkies ISJ. Statistische significantie of klinische relevantie? Ned Tijdschr Genees. 2010;154:A2516.Google Scholar
  33. van Exel NJ, Scholte op Reimer WJ, Brouwer WB, van den Berg B, Koopmanschap MA, van den Bos GA. Instruments for assessing the burden of informal caregiving for stroke patients in clinical practice: a comparison of CSI, CRA, SCQ and self-rated burden. Clin Rehabil. 2004;18(2):203–14.View ArticlePubMedGoogle Scholar
  34. Mioshi E, Lillo P, Kiernan M, Hodges J. Activities of daily living in motor neuron disease: role of behavioural and motor changes. J Clin Neurosci. 2012;19(4):552–6.View ArticlePubMedGoogle Scholar
  35. Cedarbaum JM, Stambler N, Malta E, Fuller C, Hilt D, Thurmond B, et al. The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function. J Neurol Sci. 1999;169(1–2):13–21.View ArticlePubMedGoogle Scholar
  36. Raaphorst J, Beeldman E, Schmand B, Berkhout J, Linssen WH, van den Berg LH, et al. The ALS-FTD-Q: a new screening tool for behavioral disturbances in ALS. Neurology. 2012;79(13):1377–83.View ArticlePubMedGoogle Scholar
  37. Kohle N, Drossaert CH, Schreurs KM, Hagedoorn M, Verdonck-de Leeuw IM, Bohlmeijer ET. A web-based self-help intervention for partners of cancer patients based on acceptance and commitment therapy: a protocol of a randomized controlled trial. BMC Public Health. 2015;15:303.View ArticlePubMedPubMed CentralGoogle Scholar
  38. Kohle N, Drossaert CH, Jaran J, Schreurs KM, Leeuw IM, Bohlmeijer ET. User-experiences with a web-based self-help intervention for partners of cancer patients based on acceptance and commitment therapy and self-compassion: a qualitative study. BMC Public Health. 2017;17(1):225.View ArticlePubMedPubMed CentralGoogle Scholar
  39. Hayes SC, Luoma JB, Bond FW, Masuda A, Lillis J. Acceptance and commitment therapy: model, processes and outcomes. Behav Res Ther. 2006;44(1):1–25.View ArticlePubMedGoogle Scholar
  40. Fletcher L, Hayes SC. Relational frame theory, acceptance and commitment therapy, and a functional analytic definition of mindfulness. J Ration-Emot Cogn-B. 2005;23(4):315–36.Google Scholar
  41. Allen NB, Chambers R, Knight W. Melbourne academic mindfulness interest G. Mindfulness-based psychotherapies: a review of conceptual foundations, empirical evidence and practical considerations. Aust N Z J Psychiatry. 2006;40(4):285–94.View ArticlePubMedGoogle Scholar
  42. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67(6):361–70.View ArticlePubMedGoogle Scholar
  43. Spinhoven P, Ormel J, Sloekers PP, Kempen GI, Speckens AE, Van Hemert AM. A validation study of the hospital anxiety and depression scale (HADS) in different groups of Dutch subjects. Psychol Med. 1997;27(2):363–70.View ArticlePubMedGoogle Scholar
  44. Spinhoven P, van Peski-Oosterbaan AS, Van der Does AJ, Willems LN, Sterk PJ. Association of anxiety with perception of histamine induced bronchoconstriction in patients with asthma. Thorax. 1997;52(2):149–52.View ArticlePubMedPubMed CentralGoogle Scholar
  45. Zarit SH, Reever KE, Bach-Peterson J. Relatives of the impaired elderly: correlates of feelings of burden. Gerontologist. 1980;20(6):649–55.View ArticlePubMedGoogle Scholar
  46. Bedard M, Molloy DW, Squire L, Dubois S, Lever JA, O'Donnell M. The Zarit burden interview: a new short version and screening version. Gerontologist. 2001;41(5):652–7.View ArticlePubMedGoogle Scholar
  47. Higginson IJ, Gao W, Jackson D, Murray J, Harding R. Short-form Zarit caregiver burden interviews were valid in advanced conditions. J Clin Epidemiol. 2010;63(5):535–42.View ArticlePubMedGoogle Scholar
  48. Brouwer WB, van Exel NJ, van Gorp B, Redekop WK. The CarerQol instrument: a new instrument to measure care-related quality of life of informal caregivers for use in economic evaluations. Qual Life Res. 2006;15(6):1005–21.View ArticlePubMedGoogle Scholar
  49. Hoefman RJ, van Exel NJ, Looren de Jong S, Redekop WK, Brouwer WB. A new test of the construct validity of the CarerQol instrument: measuring the impact of informal care giving. Qual Life Res. 2011;20(6):875–87.View ArticlePubMedPubMed CentralGoogle Scholar
  50. Hoefman RJ, van Exel J, Brouwer WB. Measuring the impact of caregiving on informal carers: a construct validation study of the CarerQol instrument. Health Qual Life Outcomes. 2013;11:173.View ArticlePubMedPubMed CentralGoogle Scholar
  51. Cohen SR, Mount BM, Strobel MG, Bui F. The McGill quality of life questionnaire: a measure of quality of life appropriate for people with advanced disease. A preliminary study of validity and acceptability. Palliat Med. 1995;9(3):207–19.View ArticlePubMedGoogle Scholar
  52. Steffen AM, McKibbin C, Zeiss AM, Gallagher-Thompson D, Bandura A. The revised scale for caregiving self-efficacy: reliability and validity studies. J Gerontol B Psychol Sci Soc Sci. 2002;57(1):P74–86.View ArticlePubMedGoogle Scholar
  53. Karasek R, Brisson C, Kawakami N, Houtman I, Bongers P, Amick B. The job content questionnaire (JCQ): an instrument for internationally comparative assessments of psychosocial job characteristics. J Occup Health Psychol. 1998;3(4):322–55.View ArticlePubMedGoogle Scholar
  54. Zimet GD, Dahlem NW, Zimet SG, Farley GK. The multidimensional scale of perceived social support. J Pers Assess. 1988;52(1):30–41.View ArticleGoogle Scholar
  55. Canty-Mitchell J, Zimet GD. Psychometric properties of the multidimensional scale of perceived social support in urban adolescents. Am J Community Psychol. 2000;28(3):391–400.View ArticlePubMedGoogle Scholar
  56. Zimet GD, Powell SS, Farley GK, Werkman S, Berkoff KA. Psychometric characteristics of the multidimensional scale of perceived social support. J Pers Assess. 1990;55(3–4):610–7.View ArticlePubMedGoogle Scholar
  57. Lange A. Handleiding Interactionele Probleem Oplossings Vragenlijst (IPOV) [manual for interactional problem- solving inventory (IPSI)]. Deventer: Van Loghum Slaterus; 1983.Google Scholar
  58. Meijer AM, van den Wittenboer GL. Contribution of infants' sleep and crying to marital relationship of first-time parent couples in the 1st year after childbirth. J Fam Psychol. 2007;21(1):49–57.View ArticlePubMedGoogle Scholar
  59. Cousineau N, McDowell I, Hotz S, Hebert P. Measuring chronic patients' feelings of being a burden to their caregivers: development and preliminary validation of a scale. Med Care. 2003;41(1):110–8.View ArticlePubMedGoogle Scholar
  60. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Eval Program Plann. 1979;2(3):197–207.View ArticlePubMedGoogle Scholar
  61. de Brey H. A cross-national validation of the client satisfaction questionnaire: the Dutch experience. Eval Program Plann. 1983;6(3–4):395–400.View ArticlePubMedGoogle Scholar
  62. Hoefman R, Van Exel N, Brouwer W. iMTA valuation of informal care questionnaire (iVICQ). Version 1.0. In: Rotterdam: iBMG/iMTA; 2011.Google Scholar
  63. Roth AJ, Kornblith AB, Batel-Copel L, Peabody E, Scher HI, Holland JC. Rapid screening for psychologic distress in men with prostate carcinoma: a pilot study. Cancer. 1998;82(10):1904–8.View ArticlePubMedGoogle Scholar
  64. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.View ArticleGoogle Scholar
  65. Roche JC, Rojas-Garcia R, Scott KM, Scotton W, Ellis CE, Burman R, et al. A proposed staging system for amyotrophic lateral sclerosis. Brain. 2012;135(3):847–52.View ArticlePubMedPubMed CentralGoogle Scholar
  66. NVivo Qualitative Data Analysis Software. QSR International Pty Ltd; 2012.Google Scholar
  67. Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89.View ArticlePubMedPubMed CentralGoogle Scholar
  68. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2018

Advertisement