Talking therapies for depression are overrated thanks to publication bias

DepressionAn analysis of studies into counseling therapies for depression – such as cognitive-behavioural therapy – has found that the effect of such approaches has been overestimated because studies that show a strong effect of the treatments are getting published over studies with more modest results.

In 117 studies, “talking therapies” had an average effect of 0.67 on symptoms of depression, with zero being “no effect,” whereas once publication bias was taken into account this effect dropped to 0.42.

Publication bias is “the tendency for increased publication rates among studies that show a statistically significant effect of treatment.” Think about it – patients, researchers and editors are going to be more interested in studies showing that a treatment has a considerable effect on a condition than those showing that the treatment has only a negligible effect. As a result, those studies showing a strong effect either way – that a treatment is really good or really useless – are more likely to get published and the effect of a therapy gets overestimated.

Publication bias is known to affect studies on the effects of antidepressant medications: trials showing that a drug works are more likely to get published than those showing that it doesn’t work. This is due in no small part to the pharmaceutical companies that fund drug research wanting to promote the most positive outcome possible for the medications that they sell.

Now a new analysis by Cuijpers et al. has found a similar effect among studies about psychotherapy for depression – only the most dramatic findings are getting out there and as such the effects of “talking therapies” are being “talked up.”

The authors looked at published studies on psychological treatments for depression – either approaches where verbal communication between a therapist and a client was the core element or in which a patient independently worked through a treatment book with some kind of personal support from a therapist.

They found a total of 117 studies that compared 175 treatment conditions with a control condition. When they calculated the effect of treatment in each study and analysed all these numbers together, the mean treatment effect for psychotherapy compared with control was 0.67, with zero being “no effect.”

The authors then undertook several types of statistical tests to determine whether the studies in their analysis were subject to publication bias.

The mean effect dropped to 0.42 once all the data had been adjusted for publication bias, and the various tests all pointed strongly to bias. The authors point out that effect sizes of 0.80 can be assumed to be large, effect sizes of 0.50 are moderate, and effect sizes of 0.20 are small.

Interestingly, no indication of publication bias was found for studies examining interpersonal psychotherapy, which targets how the patient interacts with other people, or for studies examining psychotherapy for women with postpartum depression.

The authors then looked specifically at the much vaunted approach cognitive-behavioural therapy, where therapist focuses on the impact a patient’s present dysfunctional thoughts have on current behaviour and future functioning. The overall effect size of the 89 comparisons between cognitive-behavioural therapy and a control condition was 0.69, but this value was reduced to 0.49 after adjustment for publication bias.

The authors conclude that research on psychotherapy for adult depression does not seem to be any freer from publication bias than research on medication treatment. As they say, “Pharmaceutical companies have clear financial reasons to inflate research findings, and psychological investigators have both personal and professional reasons for doing the same.”

The implications for this analysis are pretty messy – the research suggests that one of the two most important treatments for adult depression is not as effective as assumed, bad news given that “talking therapies” are generally thought to work better than pharmaceutical approaches, the other top therapy.
———————————————————————————————
Cuijpers P at el. (2010) Efficacy of cognitive-behavioural therapy and other psychological treatments for adult depression: meta-analytic study of publication bias. The British Journal of Psychiatry 196 (3): 173-178. DOI: 10.1192/bjp.bp.109.066001

You may also like

5 Comments

  1. I am under the impression that counsiling and therapy and totally overrated, so your life totally sucks on so many different levels you can’t hardly count them all. And seeing some counsiler for 50 min/week is supposed to make it all better. I tried it and it was a total wasted of time. Both mine and hers, how can she possibly make it possible to solve or at the very least make my life manageable, when every part of my life is in the shitter.

    Just try this or just try that and see what happens, does she not think that I have not tried these techniques in the past. The real miracle is that I am still here, I have been depressed for probably about 25 years, and you mean to tell me a few minutes will start to solve all my problems. Death seems to be about the only way out of my depression, however being a Christian it rubs against the grain of my faith, and I’m not too sure that is the answer.

    One thing I am sure of is that therapy did not work for me, medication did for a while,however even that starts to mess with your head, which just makes things worse, so you are stuck in this endless cycle of meds, medical marajuana, phycho-threapy and nothing works. So, I am afraid that I am stuck in this endless cycle of depression, and am waiting to die…

  2. Just to be clear, 0.42 is a lot better than nothing, and I have no problem with CBT et al being used for the appropriate patients and problems. Just as long as its advocates don’t over state its benefits, overall they are modest at best.

  3. Thanks for the feedback guys. I’ll take out the sentence “talking therapies reduced the symptoms of depression on average by 67% compared with a control approach, whereas once publication bias was taken into account psychotherapy was only 42% better than control,” as I don’t want to mislead anyone.

    Regarding effect sizes, it seems that even the lower effect size of 0.42 for psychotherapy is still better than what is being shown for antidepressants – 0.31, 0.37 and 0.15 you say. The bottom line seems to be: “psychotherapy has quite a modest effect on depression and it’s less than we thought, but it’s still better than pharmacological options and a valid choice for a hard to treat disorder”.

  4. 0.42 is down the low end of the range for Cohen’s d. It is a very modest effect size and certainly nothing to get excited about, especially when you consider that the therapeutic outcomes measures used in these trials are usually limited and subjective.

    Talking therapies may score somewhat better than pharmacological ones, but given the low standard they are being compared with, that ain’t saying a lot. The real world difference between .42 and .31 is not much, and there is no practical difference between .42 and .37.

  5. Hi Helen,

    A minor point is that effect size is a not a measure of percentage better than control. In this case, as the authors use Cohen’s d, it is the difference between the groups measured in standard deviations of the sample. It’s also worth saying that an effect size of .42 is smaller we’d expected from the hype, but it is still quite respectable in comparison to antidepressant studies: e.g. Kirsh et al. reported an overall effect size of .31 using combined published and unpublished data and Turner et al. reported an effect size of .37 for published trials and .15 for unpublished trials.

Leave a Reply