An analysis of studies into counseling therapies for depression – such as cognitive-behavioural therapy – has found that the effect of such approaches has been overestimated because studies that show a strong effect of the treatments are getting published over studies with more modest results.

In 117 studies, “talking therapies” had an average effect of 0.67 on symptoms of depression, with zero being “no effect,” whereas once publication bias was taken into account this effect dropped to 0.42.

Publication bias is “the tendency for increased publication rates among studies that show a statistically significant effect of treatment.” Think about it – patients, researchers and editors are going to be more interested in studies showing that a treatment has a considerable effect on a condition than those showing that the treatment has only a negligible effect. As a result, those studies showing a strong effect either way – that a treatment is really good or really useless – are more likely to get published and the effect of a therapy gets overestimated.

Publication bias is known to affect studies on the effects of antidepressant medications: trials showing that a drug works are more likely to get published than those showing that it doesn’t work. This is due in no small part to the pharmaceutical companies that fund drug research wanting to promote the most positive outcome possible for the medications that they sell.

Now a new analysis by Cuijpers et al. has found a similar effect among studies about psychotherapy for depression – only the most dramatic findings are getting out there and as such the effects of “talking therapies” are being “talked up.”

The authors looked at published studies on psychological treatments for depression – either approaches where verbal communication between a therapist and a client was the core element or in which a patient independently worked through a treatment book with some kind of personal support from a therapist.

They found a total of 117 studies that compared 175 treatment conditions with a control condition. When they calculated the effect of treatment in each study and analysed all these numbers together, the mean treatment effect for psychotherapy compared with control was 0.67, with zero being “no effect.”

The authors then undertook several types of statistical tests to determine whether the studies in their analysis were subject to publication bias.

The mean effect dropped to 0.42 once all the data had been adjusted for publication bias, and the various tests all pointed strongly to bias. The authors point out that effect sizes of 0.80 can be assumed to be large, effect sizes of 0.50 are moderate, and effect sizes of 0.20 are small.

Interestingly, no indication of publication bias was found for studies examining interpersonal psychotherapy, which targets how the patient interacts with other people, or for studies examining psychotherapy for women with postpartum depression.

The authors then looked specifically at the much vaunted approach cognitive-behavioural therapy, where therapist focuses on the impact a patient’s present dysfunctional thoughts have on current behaviour and future functioning. The overall effect size of the 89 comparisons between cognitive-behavioural therapy and a control condition was 0.69, but this value was reduced to 0.49 after adjustment for publication bias.

The authors conclude that research on psychotherapy for adult depression does not seem to be any freer from publication bias than research on medication treatment. As they say, “Pharmaceutical companies have clear financial reasons to inflate research findings, and psychological investigators have both personal and professional reasons for doing the same.”

The implications for this analysis are pretty messy – the research suggests that one of the two most important treatments for adult depression is not as effective as assumed, bad news given that “talking therapies” are generally thought to work better than pharmaceutical approaches, the other top therapy.
———————————————————————————————
Cuijpers P at el. (2010) Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. The British Journal of Psychiatry 196 (3): 173-178. DOI: 10.1192/bjp.bp.109.066001