Questionnaire development
Beginning in April 2015 CanSMART expert members collaborated to design an online survey to capture the self-management tasks, needs and support preferences of Canadians with a chronic illness. The CanSMART team was composed of 15 individuals from across Canada, including chronic disease researchers, clinical psychologists, family physicians, nurses, rehabilitation science practitioners, epidemiologists, and patient research partners. These stakeholders had come together as part of the Canadian Institutes of Health Research’s Strategy for Patient Oriented Research. Four successive versions of the survey design were circulated electronically for comments to the CanSMART team. Feedback was incorporated in turn, with each survey version discussed and further revised during team meetings until agreement among team members was reached.
Questions on self-management tasks were inspired by the structure of the PAST [16]. The PAST was developed in the Netherlands and asks respondents to indicate how often each of 19 listed self-management tasks are completed and how often support is needed with the tasks. Patient research partners first reviewed the wording and content of PAST items to assess their relevance to Canadians with chronic conditions. Additional input from other team members led to further changes: coping with pain and limitations was separated into 2 questions; 5 individual lifestyle questions on diet, exercise and health habits were collapsed into a single question; and new items were added to address asking for or needing help with household chores, self-care (e.g., bathing, dressing), and employment issues. The PAST items on interactions with healthcare providers (an important aspect of self-management) were excluded from the CanSMART questionnaire because corresponding questions from the Patient Assessment of Chronic Illness Care (PACIC) [17] were included in the larger survey [18]. Frequency of tasks was reported on a 4-point ordinal response scale: 1 (never), 2 (sometimes), 3 (mostly), or 4 (always), with a “not applicable” response option [16]. The final CanSMART self-management tasks questionnaire comprised 12 items (Additional file 1: Appendix 1, see also Table 2 but note that item on employment is not listed in Table). The team reviewed all materials with an eye to using plain language.
Recruitment of study sample
A cross-sectional survey was launched in English and French in August 2015 across Canada [9]. We recruited a convenience sample by asking CanSMART team members and representatives of national and regional patient and disease-specific advocacy groups to circulate the survey link via email, newsletters, social media and relevant websites. Advocacy groups were identified by team members and through web searches. In total, 30 advocacy groups were contacted to share study information and the survey link. Interested individuals aged 18 and over, with at least one chronic physical and/or emotional condition were invited to participate. The invitation specified, “To be in this study, you must be 18 years of age, living in Canada, and have one or more chronic physical and/or emotional illnesses”. Age, sex, level of education, employment status, province of residence and type of chronic conditions were included in the brief demographics section appearing at the end of the survey. The online survey did not require any identifying information, and no incentive or compensation was provided. The welcome page informed participants that the survey was voluntary, anonymous, and that completion of the survey was interpreted as informed consent. The study protocol was approved by the Research Ethics Committee of St. Mary’s Hospital Center, Montreal. The survey was closed in February 2016.
Psychometric analysis
We based the scale development on standard psychometric methodology [19], applied to the ordinal items on self-management task frequency (Additional file 1: Appendix A). The item #12 on managing work-related limitations was excluded, because 57% of the sample had “missing” responses. The study sample was restricted to the subset of patients who had completed the remaining 11 items. The first step in our psychometric analysis was to identify valid subscales by using both Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA). Ideally, we would need to work with two large independently collected data sets, and perform scale development on one, using EFA, and validation on the other, using CFA. However, in clinical settings such an approach may not be possible, due to the inherent difficulties of human data collection. Using guidelines recommended by Anderson [20], we therefore randomly split the sample into two equal size subsamples called ‘exploratory’ and ‘confirmatory’. We applied EFA to the former and CFA with maximum likelihood (ML) to the latter [21, 22]. Pearson’s Chi-square and T-tests were used to compare patient characteristics and frequency of self-management tasks in the two subsamples. From the EFA we determined the dimensionality of the factor space, which we refer to as ‘p’; the dimensionality was determined by parallel analysis [22, 23]. To interpret the dimensions found, we listed the ‘important’ items for each dimension, i.e., those with factor loadings (after ‘varimax’) larger than 0.4 [23].
The ‘important’ items for each dimension were grouped to represent one structural equation model (SEM). To take into account variables which we considered meaningful but do not appear ‘important’, we defined informally a number of alternative models, i.e. we consider several ways to assign the non-important variables to any dimension, including removal. We then evaluated these models by CFA. In CFA, several statistics are available to assess the fit of a model. Following Kline [24], we relied on the following four: Chi-Square, Root Mean Square Error of Approximation (RMSEA), Bentler’s Comparative Fit Index (CFI), and the Standardized Root Mean Square Residual (SRMR). Values of Chi-Square close to zero indicate an acceptable model fit; in practice, it is usually required that RMSEA ≤ 0.10, CFI ≥ 0.90, and SRMR ≤ 0.10. Recent references suggest stricter criteria: RMSEA ≤ 0.06, CFI ≥ 0.95, and SRMR ≤ 0.08), but these stricter criteria are only valid for larger sample sizes than ours (n > 500) [25]. In large sample sizes (> 300), since CFA requires (approximate) multivariate normality of the items, ordinal items such as ours might be analyzed by replacing the Pearson correlation matrix with the polychoric correlation matrix [25]; this was not appropriate given our smaller sample size.
The second step in this psychometric analysis was to perform Rasch analysis [26] on the selected model. Likelihood Ratio (LR) tests [27] were performed to determine whether the Partial Credit Model (PCM) [28] or the Rating Scale Model (RSM) [29] would be used. According to Rasch analysis, threshold points on a response scale should be correctly ordered for each item; e.g. the respondents would consider endorsing 'always' to represent greater need for the latent trait rather than 'mostly'. In contrast, ‘disordered thresholds’ occur when respondents have difficulty consistently discriminating between response categories. The Category Characteristic Curve (CCC) [30], which displays the probability of a respondent endorsing a particular response category based on their level of need for the item (intensity), was obtained to examine the performance of the response scale. A disorder threshold was detected in the CCC plot when the line of a corresponding answer crossed other lines (answer) at the same latent trait value; if an item had a disordered threshold, the CCC informed response re-categorisation. To determine item fit, ‘infit’ and ‘outfit’ statistics were calculated, whereby values between 0.7 and 1.3 are acceptable [31]. Finally, unidimensionality was assessed by the Martin-Löf Test [32].
Once validity and ‘scoring properties’ were confirmed, we computed a subscale score for each dimension by simply averaging the important items for the dimension. Internal consistency was assessed using Cronbach’s alpha for each subscale; acceptable reliability can be achieved with a Cronbach Alpha of 0.7 but a threshold of 0.6 is also considered acceptable for short scales (< 10 items) [33].
Finally, we studied the variation of the subscales (computed after Rasch analysis) across subgroups defined by patient characteristics (including diseases). To determine whether these associations were independent we conducted multivariable linear regression analyses [34]. All the analyses were performed with SAS version 9.4 and R. [27] (eRm package).