Summary of the PACE trial

PACE was a large randomised trial whose primary aim was to assess the effectiveness of GET and CBT as treatments for CFS (early publications refer to it as a “randomised controlled trial”, but “randomised trial” is more appropriate, given that several nuisance variables were not fully controlled across trial arms, e.g., contact hours). Participants were 641 adults with mild-to-moderate CFS defined by the Oxford criteria [10]: the principal symptom must be fatigue, which must have had a definite onset, resulted in significant disability, and have persisted for at least six months. Participants also had to score 65 or less on the Short-Form Health Survey Physical Function subscale [11]. Also, they had to report experiencing at least six of the 11 fatigue items on the Chalder Fatigue Questionnaire (CFQ [12]), as “more than” or “much more than” than prior to illness.

Participants were randomised into four groups. All were offered at least three medical consultations. The first group, which we will call Control, received no further treatment (the trial publications use the term Specialised Medical Care). The other groups received up to 15 therapy sessions over 36 weeks. One group received CBT, one GET, and the fourth group received a novel treatment, Adaptive Pacing Therapy. Both the CBT and the GET interventions were built upon a behavioural/deconditioning model of CFS. This model proposes that there is no major ongoing disease process underlying CFS - only deconditioning due to recent inactivity, and its various consequences. When patients attempt to increase their activity, they experience normal fatigue, stiffness and other symptoms, which they misinterpret as signs of continuing disease. The patients then become more focused on their symptoms, and fearful of further activity, creating a self-perpetuating cycle [2]. The GET programme was designed to help CFS patients overcome this purported fear of exercise and intense symptom-focusing through graded exposure to exercise, and thereby also reverse any deconditioning that had occurred. Participants were asked to choose an aerobic activity they enjoyed, and to gradually increase the duration and intensity of that activity under the supervision of a therapist. The CBT programme had similar aims, but addressed the fear of activity, maladaptive illness beliefs and symptom focusing using a combination of CBT and practical activities ([2], p. 825). Participants were encouraged to view their symptoms as arising from anxiety, intense symptom focusing and/or deconditioning. The sessions addressed fears about exercise and other “unhelpful cognitions” that may perpetuate symptoms, and encouraged participants to try gradually increasing their activity ([2], p. 825).

Adaptive Pacing Therapy, in which patients were advised not to exceed a certain level of activity, was created specifically for the trial. Results for this trial arm did not differ significantly from those for the Control arm for any of the outcomes considered in this article. Consequently, we will not discuss them further here.

Primary outcomes

In 2011, the first major publication from the trial reported results based on this new primary outcome [2]. It was found that, following treatment, scores on both these continuous measures improved in all groups, but significantly more so in the CBT and GET groups than in the other groups. In the 2011 publication, rates of overall improvement were also reported; however, these were not based on the protocol-specified definition, but rather on a very different, and much more generous, one: Definition B in Table 1. By this new definition, 59% of CBT participants and 61% of GET participants were classed as having improved overall [2]. However, 45% of Control participants did so too. Results for the original protocol-specified definition of improvement - Definition A in Table 1 – do not appear in any peer reviewed publications from the trial (which number in the double digits [15]).

Rates of recovery

An important secondary outcome specified in the trial protocol was the proportion of patients who met the specified definition of recovery at the end of the trial [1]. The definition of recovery presented there considered each participant’s scores on two key self-rated measures (fatigue, physical function), one further measure of overall self-rated improvement and finally, whether the participant still met various CFS case definitions. The complete definition of recovery is given in Table 1 (Definition A). However, results for this outcome never appeared in published reports. Instead, a 2013 paper reported recovery rates based on a much more generous definition of recovery (Definition B in Table 1) [4]. According to these new criteria, 22% of patients in each of the CBT and GET groups qualified as recovered, but only 7% in the Control group. The difference in recovery rates between the CBT/GET groups and the Control group was statistically significant. The PACE investigators have not specified when the decision to change the definition of recovery was made, except to say it was “before the analysis occurred” [16]; the change does not appear in any documentation prior to the final publication, and there is no published evidence that it was approved by the trial steering committee.

Other outcome measures

A number of other secondary outcome measures were collected at 52 weeks, including several additional subjective outcomes, and also four objectively scored measures, which are described further below. During the course of the trial, data for a range of adverse events and outcomes were also collected; these are also described briefly below.

The adverse events measures collected during the trial included: serious adverse events (death, hospitalisation, etc.); serious deterioration (a broader category that included a serious adverse event, sustained decrease in self-reported physical function or overall health, or withdrawal due to worsening); and non-serious adverse events. Serious adverse events were significantly more prevalent in the GET group (8%) than in the Control group (4%); there were no other statistically significant group differences.

Long-term follow-up

A mail survey was conducted at least two years after randomisation (median 31 months [7]:). Survey response rates were 72%, 74% and 79% for the Control, CBT and GET groups respectively. Participants were again asked to complete the trial’s primary fatigue and physical rating scales, and several other questionnaires. A 2015 paper reported the results for the fatigue and physical function measures, again treating them as separate, continuous variables [7]. Analyses of these measures, based on an available case approach, failed to yield any significant effects of treatment group. However, the investigators did not view this negative result as a cause for concern at all. They argued that many patients in the Control and Adaptive therapy trial arms had received some CBT or GET after the conclusion of the main trial, and this could explain why they had since improved to the level of the other patients.

Current analyses

The primary objective of our reanalyses was to examine how the trial outcomes would have looked if the investigators had adhered to their published protocol. Specifically, we were interested in analysing results for the primary outcome set out in that document: overall improvement rates. We also calculated recovery rates based on the definition outlined in the protocol. Results from this latter analysis have been published elsewhere [17, 18], but here we present more complete details of our method and findings. Finally, we explored the published data on long-term outcomes to examine whether they had been contaminated by patients’ post-trial therapy experiences, as the PACE researchers hypothesised.

Recovery rates

Using the protocol-specified definition of recovery, and applying an intention-to-treat approach, the rates of recovery were 7%, 4% and 3% for the CBT, GET and Control groups respectively. Applying an available-case approach, these rates were 8%, 5%, and 3% respectively. In neither instance was there a statistically significant effect of treatment on recovery rates (p values were 0.14 and 0.10, respectively, for the intent-to-treat and available case approaches).

Long-term outcomes

Out of those who responded to the long-term follow-up, 43% of the Control participants had received no further CBT or GET after the completion of the trial. This was also the case for 74% and 75% of the respondents from the CBT and GET arms respectively. Considered together, this subset of participants was perhaps slightly less severely affected than the remaining patients: they scored slightly better on the primary physical function and fatigue scales at 52 weeks than those who opted for further treatment (physical function: 61.3 vs. 48.1; fatigue: 23.9 vs. 25.9). However, at 52 weeks, the pattern of scores across treatment arms was the same as for the sample as a whole: the CBT and GET participants in our subset rated their fatigue as slightly lower and their physical function slightly higher at 52 weeks than the Control participants. In this respect, our subsample may be considered reasonably representative of the sample as a whole.

Table 2 provides arithmetic means for the two major self-report outcome measures for this subset of patients (i.e., those who received no further treatment). The pattern of results presented here mirrors that obtained for the entire cohort: the small group differences apparent on these measures at 52 weeks are no longer evident at long-term follow-up. A one-way analysis of variance revealed that there were no statistically reliable effects of treatment group on either outcome measure (Physical function: F(3,291) = 0.70, ns; Fatigue F(3,291) = 0.17, ns). If we repeat the analyses, adding in those cases who received some additional therapy sessions, but less than the minimum 10 considered by the investigators to be an “adequate” dose ([7], p.1071), the outcome does not change (Physical function: F(3,384) = 1.85, ns; Fatigue F(3,384) = 0.86, ns). Consequently, the disappearance of group differences at long-term follow-up cannot be attributed to the effects of additional post-trial therapy.

Discussion of new results

Our reanalyses of the trial data based on the published protocol generated some troubling findings. First, scores on the protocol-specified primary outcome measure — improvement in self-reported fatigue and physical function – were numerically higher for the CBT and the GET groups than for the Control group. However, these differences did not pass the threshold for statistical significance after correcting for the number of planned comparisons specified in the trial protocol. Using a more lenient correction (assuming only five planned comparisons), outcomes are only marginally more positive: the comparison between GET and Control just reaches this threshold, but the comparison between CBT and Control does not. Of course, our analyses did not incorporate a number of important stratification variables that were unavailable in the FOIA dataset. However, it appears unlikely that their inclusion would substantially alter the result, and our analyses remain the closest approximation to the originally specified one that has ever been published. Our findings suggest that, had the investigators stuck to their original primary outcome measure, the outcomes would have appeared much less impressive.

Improvement rates for self-rated fatigue and physical function considered individually did yield some statistically significant findings, which suggests that the interventions were somewhat specific in the way they altered patients’ illness perceptions. Self-rated physical function scores showed greater improvement in the GET group than in the Control group — but not self-rated fatigue scores – which suggests GET had a modest effect on patients’ perceptions of their physical function, but did not do much to alter symptom perceptions. Conversely, self-rated fatigue showed greater improvement in the CBT group than in Controls – but not physical function – which suggests CBT elicits modest reductions in symptom-focusing, but does not do much to improve patients’ confidence in their physical capacities.

Second, when recovery rates were calculated using the definition specified in the published protocol, these were extremely low across the board, and not significantly greater in the CBT or GET groups than in the Control group. Neither an intent-to-treat nor an available case analysis yielded a significant benefit for these therapies over conventional medical care. Again, we were unable to incorporate a number of stratification variables into this analysis, but it is unlikely that the result would be different had we done so.

With respect to long-term outcomes, the investigators’ original analysis did not reveal any significant effects of treatment allocation on self-reported fatigue and physical function at long-term follow-up [7]. They suggested this null effect may have been due to the confounding effects of post-trial therapy. Our informal re-examination of the long-term follow-up results provide no support for this suggestion. We found that even when patients who received post-trial CBT or GET are excluded, there is still no evidence of any long-term treatment-related benefits – not even a trend in the hypothesised direction. Of course, our analyses were informal. Ideally, we would have replicated the analysis reported in [7] for this patient subset, which included all the covariates listed in that analysis ([7], p. 1068), such as fatigue and physical function scores at 52 weeks, time of follow-up, trial centre and disease caseness. This was not possible on the data available. However, until better evidence becomes available, there is no reason to believe that post-trial therapy can offer a viable explanation for the absence of treatment effects at long-term follow-up.

One major problem for the PACE trial is that it was originally designed around a highly optimistic view of the therapeutic benefits of CBT and GET. Drawing on results from previous, smaller trials, the PACE investigators estimated that CBT would be likely to yield an improvement rate some six times greater than medical care alone, and GET would yield a rate five times greater [20]. These expectations formed the basis of the power calculations for the trial. But unfortunately, the improvement rates for CBT and GET participants - when compared with Control participants - fell markedly short of those expectations. So it is perhaps not surprising that an analysis of the binary improvement data alone was insufficient to detect any statistically reliable effects. In this context, it would have been perfectly acceptable first to report the protocol-specified primary outcome analysis, and then to explore the data using methods that are more sensitive to smaller effects – for example, analysis of the individual, continuous outcome measures. However, instead, the researchers chose to omit the former analysis altogether, and report only the latter. They then reported improvement rates based on an entirely new, and much more generous, definition of improvement. In sum, the analyses that were the least complimentary to CBT and GET never appeared in the published reports; the analyses that showed these interventions in a more favourable light were the only ones to be published.

As we have already pointed out, the timing of the change to the primary outcome – several months after trial completion - was highly problematic. There was also insufficient independent justification for making the change. For reasons that are never made clear, investigators had suddenly taken the view that “…a composite measure would be hard to interpret, and would not allow us to answer properly our primary questions of efficacy (i.e. comparing treatment effectiveness at reducing fatigue and disability).” ([13], p. 25). Certainly, the separate analysis of the two continuous measures provides useful additional information, but this does not justify abandoning the originally planned outcome. Further, the protocol already included measures of specific improvement rates in self-rated fatigue and physical function, and it is not clear why these were abandoned in favour of the new measure.

Turning now to the recovery rates, the late changes to the definition of recovery made it much easier for a patient to qualify as recovered. These changes were quite substantial. For example, the minimum physical function score required to qualify as recovered was reduced from 85 to 60, which is close to the mean score for patients with Class II congestive heart failure (57/100 [21]), and lower than the score required for trial entry (65/100). Also, on the fatigue criterion, a patient could now count as “recovered” despite reporting continuing fatigue on as many as seven out of the 11 fatigue questionnaire items, a level that substantially overlaps with that required for trial entry. Again, these changes operated to favour the study hypotheses. They enabled the researchers to make the claim that CBT and GET were significantly more likely to lead to recovery than conventional medical care (the original recovery definition would have yielded a null result), and to declare that at least “a fifth” of participants recover with CBT and GET [4, 22]. Neither claim could have been made if the original definition of recovery had been used.

Again, the timing of the change to the recovery definition – over a year after the trial was completed - is highly problematic. Also, an adequate justification for the change is yet to be provided. In their 2013 publication on recovery rates, the researchers argued that the normal ranges for some key scores were wider than previously thought, which would justify classing more participants as “recovered” on these measures [4]. However, we have recently shown that when the chronically ill and the very old were excluded from the relevant reference samples, and where correct statistics were applied to determine appropriate cut-off values, the normal ranges are, if anything, narrower than previously believed [17]. Consequently, this argument does not stand up to scrutiny (see [17] for further details).

Several other arguments have been presented in defence of these changes [23, 24]. One was that since there is no agreed definition of recovery, the new modified one is just as good as the original (the original definition “simply makes different assumptions” [24], p. 289). This argument fails to explain why the definition was changed in the first place. If both definitions are indeed equally good, then the one to be preferred is surely the one that was specified in advance, before any of the results were known. Another argument was that the recovery rates obtained with the modified definition were numerically similar to those found in some previous trials of CBT for CFS [23]. However, these other trials used entirely different definitions of recovery, so are not relevant here. One final argument was that the original definition of recovery was simply “too stringent to capture clinically meaningful recovery” [23]. However, the only supporting evidence for this statement comes from the disappointing recovery rates in the PACE trial itself; no independent justification is offered. Clearly, a strong concept like recovery must be operationalised carefully. Physicians and lay people understand this term to mean a return to good health [25], and any definition must preserve this core meaning. If anything, the original protocol-specified definition was rather generous, and may have identified some individuals that had not recovered in the plain English sense of the word. For example, on the primary physical function measure (the SF36), it was possible to score in the bottom decile for working age individuals with no long-term illness or disability, and still count as recovered on that criterion [17]. The definition also did not require evidence of an ability to return to work or other premorbid activities, even though these are very important components of what recovery means to patients. There was certainly no justification for further loosening that definition. In sum, none of the trial investigators’ arguments adequately justified the late changes to the recovery definition. More detailed discussion of these issues can be found elsewhere [26].

Turning now to long-term follow-up, the original publication of the long-term follow-up data reported no significant differences amongst treatment groups at this time point [7]. However, the authors dismissed their own finding, arguing that many participants received additional post-trial therapy which might have operated to obscure group differences. Instead, they based their main conclusion on comparisons between time points. For example, the first line of the Discussion reads: “The main finding of this long-term follow-up study of the PACE trial participants is that the beneficial effects of the rehabilitative CBT and GET therapies on fatigue and physical functioning observed at the final 1 year outcome of the trial were maintained at long-term follow-up 2·5 years from randomisation.” ([7] p. 1072, Italics added). This conclusion is repeated in the Abstract. The decision to lead with this conclusion again operated to show the findings in a more positive light than would have been possible based on their own primary between-groups analysis. The informal analyses we presented here provide no support for the investigators’ claim that post-trial therapy contaminated the long-term outcome data. Of course, our analyses did not include important potentially confounding variables that might differ amongst trial arms, and such a comprehensive analysis might possibly produce a different result. However, until there is positive evidence to suggest that this is the case, the conclusion we must draw is that PACE’s treatment effects are not sustained over the long term, not even on self-report measures. CBT and GET have no long-term benefits at all. Patients do just as well with some good basic medical care.

Overall evaluation of the trial

Some notable strengths of the PACE study included the large sample size (determined a priori using power analysis [1]), the random allocation of patients to treatment arms, the use of a well-formulated protocol to minimise drop-outs, and the reporting of the full CONSORT trial profile (including detailed information about missing data). The incorporation of an active comparison group - Adaptive Pacing Therapy - also provided a useful secondary control for factors such as overall therapy time and patient-therapist alliance. It is worth pointing out that results for this group were not significantly different from those for the Control group on any of the measures considered in this paper. Other strengths were that each therapy group received a substantial dose of therapy, and standardised manuals ensured comparability of treatments across centres and therapists. Finally, a wide range of outcomes was measured, including several objective measures, as well as various adverse events measures.

However, despite these strengths, the design, analysis and reporting of the results introduced some significant biases. We have already discussed some of the biases that were introduced at the analysis and reporting stage. Several key results that showed CBT and GET in less than favourable light were omitted and replaced with new ones that appeared more favourable to the treatments. These changes were made at a late stage in the trial, and we have argued here that none had sufficient independent justification. In reality, the effects of CBT and GET were very modest - and not statistically reliable overall if we apply procedures very close to those specified in the original published protocol.

Another source of bias arose from the trial’s heavy reliance on self-reports from participants who were aware of their treatment allocation. Clearly, in a behavioural intervention trial, full blinding is not possible. Nevertheless, it is the researchers’ responsibility to consider the possible effects of lack of blinding on outcomes, and to ensure such factors are insufficient to account for any apparent benefits. A trial that is not blinded, self-reported outcomes in particular can produce highly inflated estimates of treatment-related benefits [27, 28]. A recent meta-analysis of clinical trials for a range of disorders found that when patients were not blinded to their treatment allocation, their self-reported improvement on the treatment of interest was inflated by 0.56 standard deviations, on average, when compared to a corresponding blinded phase of the same trial [29]. In contrast, observer-rated measures of improvement were not significantly affected by blinding. Given this discrepancy in the effects of blinding on subjective and objective measures, it appears unlikely that these effects reflect genuine health benefits. Amore plausible explanation is that they are expectation-related artefacts – for example, they reflect the operation of attentional biases that favour the reporting of events consistent with one’s expectations [30], or recall/confirmation biases that enhance recollection for expectation-consistent events [31].

The PACE investigators have argued that expectancy effects alone cannot account for the positive self-reported improvements, because at the start of treatment, patients’ expectations of improvement were not greater in the CBT and GET groups than in the other groups [2, 23]. However, they fail to point out that CBT and GET participants were primed during treatment to expect improvement. The manual given to CBT participants at the start of treatment proclaimed CBT to be “a powerful and safe treatment which has been shown to be effective in... CFS/ME” ([32], p. 123). The GET participants’ manual described GET as “one of the most effective therapy strategies currently known” ([33], p. 28). Both interventions emphasised that faithful adherence to the programme could lead to a full recovery. Such messages — from an authoritative source — are likely to have substantially raised patients’ expectations of improvement. Importantly, no such statements were given to the other treatment groups. When we add to this the fact that the CBT programme, and to a lesser extent GET, was designed to reduce “symptom focusing”, which may have further influenced self-report behaviour in the absence of genuine improvement [27, 34], these findings start to look very worrying indeed.

A further cause for concern in the PACE trial was that the two primary self-report measures appear to behave in different ways depending upon the intervention. Our analysis based of the protocol-specified outcomes indicated that GET produces modest enhancements in patients’ perceived physical function, but has little effect on symptom perception. Conversely, CBT improved symptom perception – specifically, self-rated fatigue scores – but had little effect on perceived physical function. If these interventions were operating to create a genuine underlying change in illness status, we would expect change on one measure to be accompanied by change on the other.

Given the high risk of participant response bias in this study, it was therefore crucial to demonstrate accompanying improvement on more objective measures. However, only one such measure showed a treatment effect. On the six-minute walking test, the originally-reported available case analysis found that GET participants walked reliably farther than Control participants at the primary, 52-week endpoint. However, after an entire year, this group walked an average of just 67 m farther than baseline, and around 30 m farther than Controls. To put this in context, a sample of Class II chronic heart failure patients with similar baseline walking distances increased their distance by an average of 141 m after only three weeks of a gentle graded exercise programme [35].

No other objective measures yielded significant treatment effects. Most notably, treatment did not affect aerobic fitness, measured using a step test. If GET had genuinely improved participants’ physical function and levels of activity, these improvements should have been clearly evident on fitness measures taken a full year after trial commencement. Treatment also did not affect time lost from work [3]. There was ample opportunity for improvement here: during the six months preceding the trial, 83% of participants were either in work or would have worked if able (based on the number reporting lost work days). This suggests they could have immediately increased their hours if their health had permitted. Finally, the percentage of participants receiving government benefits or income protection actually increased over the treatment period for all groups [3]. It is concerning that these negative findings were not even published until years after the primary results had been reported, so these inconsistencies are not immediately apparent to the reader. For example, the crucial fitness results were not published until four years after the primary outcomes. The investigators dismissed most of these measures as unimportant or unreliable; they did not consider them valuable as a means of estimating the degree of bias inherent in their self-report outcomes.

The absence of evidence for treatment-related recovery is an additional, serious concern for the trial. CBT and GET were not seen as adjunct treatments that might relieve a little distress. Rather, they were seen as capable of reversing the very behaviours and cognitions responsible for CFS. The behavioural-deconditioning model, on which the treatments were based, assumes that there is no underlying disease process in CFS, and that patients’ concerns about exercise are merely “fearful cognitions” that need addressing ([36], p. 47–8). Participants in some trial arms were even told that “there is nothing to stop your body from gaining strength and fitness” ([32], p. 31). If this model of CFS were correct, and if the treatments were operating as hypothesised, then some participants that duly followed the programme should have returned to the levels of health and physical function, that they enjoyed prior to illness onset. Therefore, the rates of recovery in the CBT and GET groups should have been significantly and reliably higher than in the Control group, irrespective of the method used to define recovery. This was not the case.

The failure of CBT and GET to “reverse” CFS is perhaps not so surprising when we consider recent exercise physiology studies. CFS patients have shown various physical abnormalities when tested 24 h after exertion (reduced VO₂max and/or anaerobic thresholds; for a review, see [37]). These abnormalities are not seen in sedentary, healthy adults or even in patients with cardiovascular disease, and therefore cannot be attributed to deconditioning alone. Such findings call into question the core assumption of the behavioural/deconditioning model that there is no ongoing disease process. If there is a rational basis for patients’ concerns over exercise, encouraging them to push through symptoms may be harmful, and recasting patients’ concerns as dysfunctional may cause additional, psychological harm.

Turning now to safety issues, there were few group differences in the incidence of adverse events, and the researchers concluded that both CBT and GET were safe for people with CFS. This finding – particularly that relating to GET - contrasts markedly with findings from informal surveys conducted by patient organisations [38, 39]. In these surveys, between 33% and 79% of respondents report worsened health as a result of having participated in some form of graded exercise programme (weighted average across 11 different surveys: 54% [39]). Of course, in such surveys, participant self-selection may operate to enhance the reporting rates for adverse outcomes. However, this finding is so consistent, and the number of participants surveyed is so large (upwards of 10,000 cases), that it cannot be entirely dismissed. One likely reason for the discrepancy between PACE’s findings and those of patient surveys is the conservative approach used in PACE’s GET programme. Patients were encouraged to increase activity only if it provoked no more than mild symptoms [40]. Unfortunately, compliance with the activity recommendations was not directly assessed: actigraphy data were collected only at trial commencement [1] and never reported. This is a significant omission, since there is evidence that graded exercise therapies are not always successful in actually increasing CFS patients’ activity levels [41]. Even those who comply with exercise goals may reduce other activities to compensate [42]. The lack of improvement in fitness levels in PACE’s GET group does suggest that participants may not have substantially increased their activity levels, even over the course of an entire year. Also, even though the majority of GET participants chose walking as their primary activity [2], this group demonstrated an average increase in walking speed of only 10% after an entire year (increases of 50% or more have been observed in other patient populations [35]). Given these features, it is inappropriate to generalise the safety findings from PACE to graded activity programmes more widely, especially as they are currently implemented in clinical settings.