5

A model of StrongMinds - Oxford Prioritisation Project

By Lovisa Tenberg and Konstantin Sietzy

Cross-posted to the Oxford Prioritisation Project blog. We're centralising all discussion on the Effective Altruism forum. To discuss this post, please comment here.

Summary: In 2016 James Snowden of the Centre for Effective Altruism built a quantitative model estimating the impact of StrongMinds. In order to measure our uncertainty about the estimate, we built a detailed, annotated translation of the model in Guesstimate (a “spreadsheet for things that aren’t certain”) which can be found here. This post is acts as an appendix to the Guesstimate model.

The intervention

Improving mental health (MH) in lower and middle income countries (LMIC) as a highly effective cause has received increasing attention in the Effective Altruism community over the past year (GWWC has written on it here and here; Harvard EA has written on it here; and Givewell staff members donated to it in 2016). It is vast in scale and mostly neglected in LMICs, but tractability remains somewhat of a black box.

MH is vast in scale, amounting for 7.4% of the Global Burden of Disease, but 37% of the Global Burden of Non-Communicable Disease. Importantly, the outlook is worsening: DALYs due to mental illness grew by 38% between 1990 and 2010, and are expected to continue on this trajectory.

MH is neglected both in-country and in the international community. According to Victoria de Menil writing for the Centre of Global Development, “one third of LMIC do not have a designated budget for mental health… and among those that do, the average expenditure on mental health in low-income countries is 0.5% of the total health budget.” She contrasts the treatment gap in Nigeria – 90% across all diagnosable mental disorders – with an average of 32% for schizophrenia to 56-57% for depression and anxiety disorders in European countries. The international donor community affords MH improvement disproportionately little attention given its scale: only 0.7% of international donor funding is directed towards it, and hardly any NGOs work directly on MH, especially not at the level of widely scalable interventions.

On tractability, the evidence is out. Most existing scholarship on the effectiveness of MH interventions exists in a high-income country context, and has little focus on the cost-effectiveness of interventions. Yet it is increasingly on the map of global health professionals and national governments, having been included in the January 2016 Sustainable Development Goals after high-profile campaigning. In addition, King’s College and the London School of Hygiene and Tropical Medicine are conducting 40+ trials into various components of the effectiveness of MH interventions as part of their Centre for Global Mental Health.

The organisation

StrongMinds treats women with depression in Uganda. The organisation recruits community mental health facilitators to treat groups of 10-12 women using Interpersonal Group Therapy (IPT-G) over a course of 8-12 weeks. StrongMinds is committed to impact measurement and scalability: internal figures suggest that 80% of patients are depression-free at the end of treatment, with the effect being stable in follow-ups. They also claim a 67% decrease in unemployment. StrongMinds aims to scale to 100,000 patients treated by 2019 and 2m by 2025. StrongMinds has been lauded as a highly transparent conversation in private conversations with GWWC staff members, and is led by pedigreed staff in the global health sector (their CEO is an ex-employee of PSI).

The model

The model is based on a 2016 model created by James Snowden for CEA. It is mainly based on StrongMinds organisational data and internal impact assessment results. A detailed, annotated translation of the model in Guesstimate can be found here.

Key inputs are i) StrongMinds data on expenses and patients treated; ii) converted DALY weights of 1 point reductions on the PHQ-9 depression scale used to measure StrongMinds impact; iii) trajectory of treatment effect of IPT-G for depression over time, adapted from research by Rebecca Reay et al. (2012). The model outputs data for both the year 2016, and a 2019 estimate ("2019E"), based on StrongMinds projections of their expenses and number of patients treated.

StrongMinds per-patient costs were assessed using YTD mid-2016 cost and patient treated metrics, arriving at a cost per patient of $206. StrongMinds hopes to scale to having treated 100,000 patients by 2019 at a total cost of $6.6m, reducing their per-patient cost over time by roughly 2/3rds to $66.

StrongMinds measures impact on the 27-point, linear PHQ-9 scale. To convert PHQ-9 impact to DALYs averted, Global Burden of Disease DALY-weighting of most severe depression (0.658) was divided by PHQ-9 points-weighting of most severe depression (27) to render 0.024 DALYs averted per PHQ-9 point reduced.

As StrongMinds’ impact assessment only measures absolute effect, the model proxies counterfactual impact by relying on Reay et al. who compare the trajectory of IPT-G patients’ depression with an untreated control group. The model extrapolates Reay’s comparison over a ten year span to arrive at a total multiplier of original impact over time that is applied to the baseline treatment effect.

Uncertainties

The model suffers from three key uncertainties. Firstly, there is an established case that preference-based measurements such as DALYs may underrate the badness of depression. Secondly, the trajectory of treatment effect vs a counterfactual is proxied through research on postpartum depression only. Other types of depression may not exhibit a declining counterfactual over time. Simultaneously, other types of depression may be less amenable to IPT-G. Finally, data (and the 2019E) is solely based on StrongMinds internal assessments, suggesting caution.

Comments (13)

Comment author: MichaelPlant 15 May 2017 12:37:17AM 3 points [-]

Great to see this here. Thanks Konstantin and Lovisa. A couple of thoughts.

It would be good if you'd put your headline results in the post and what, if anything, you think further flows from your conclusions (i.e. you now consider StrongMinds is more/less effective than something else, such as AMF).

Can you provide a link to the 0.658 DALY rating for depression? I can never remember how much of that is "years of life lost" and how much is "years lived with disability". There are two parts that make up DALYS and I think people should present them seperately; there are different views you can take on the badness of deaths. This is helpful to those of an Epicurean persuasion such as myself who are more concerned with making people happy than just keeping people alive (to riff off Narveson 1967).

Comment author: ThomasSittler 23 May 2017 11:07:16AM *  0 points [-]

Thanks for the comment.

Regarding epicureanism: More generally, breaking everything down into years of life lost to death, years of suffering, and years of potential future life enabled, would be a good idea for a future improvement of the models. It would enable us to see how the models work for people with different values. By the way anyone can make a copy of our models and adapt them! :)

Regarding your (separate) view that depression is much worse than consensus DALY weights account for, I think this is best thought of in relative terms. You want to make sure the HEWALY weights across all four models are consistent with your values and empirical beliefs, rather than just the DALY weight on depression in the StrongMinds model.

Comment author: MichaelPlant 25 May 2017 06:02:03PM 0 points [-]

I agree on your second point that you'd want to adjust all the models, I was just hoping you could give me a reference. My thought is that depression removing 0.65 of someone's happiness for a year (i.e. going from 8/10 to a 2.5/10) seems about right on the life satisfaction scores. This means that everything else should have a much lower comparative weight, rather than making depression worse than death. For instance, maybe blindness really has a weight of 0.1 rather than 0.5 as I believe it does at present.

Comment author: Peter_Hurford  (EA Profile) 14 May 2017 08:55:32PM 1 point [-]

Wouldn't ~$660/DALY be likely, approximately less cost-effective than GiveDirectly?

Comment author: ThomasSittler 23 May 2017 11:00:07AM *  0 points [-]

Wouldn't ~$660/DALY be likely, approximately less cost-effective than GiveDirectly?

That's indeed what our current model says. I have some more comments at the bottom of this post.

Comment author: MichaelPlant 15 May 2017 12:24:50AM 0 points [-]

Peter, do you have any figures Give Directly? Also, what is the measure of cost-effectiveness you're thinking of? Here's GiveWell's spreadsheet which, AFAICT, is in terms of "cost per life saved equivalent" which I'm not sure how to compare to DALYs or anything else (in fact, even after some searching, I'm still not sure what "cost per life saved equivalent" even is).

Comment author: Halstead 16 May 2017 02:05:30PM *  3 points [-]

Michael, the definition is here - https://docs.google.com/spreadsheets/d/1KiWfiAGX_QZhRbC9xkzf3I8IqsXC5kkr-nwY_feVlcM/edit#gid=1034883018

On the results tab, if you hover over the "cost per life saved equivalent" box, it says "A life saved equivalent is based on the "DALYs per death of a young child averted" input each individual uses. What a life saved equivalent represents will therefore vary from person to person. "

I agree this is too hard to find and it would be good if this were fixed. I'd also like to see the assumptions made about this figure more clearly spelled out

Comment author: Benito 15 May 2017 12:32:39AM 0 points [-]

I had been assuming "cost per life saved equivalent" meant somewhere in the range of 50-100 QALYs, the rough length of a human life.

(The "equivalent" thing would be about it being spread over many people - it's not the case that you literally gave one person a whole life, but you caused an equivalent amount of "good living" to happen.)

Comment author: MichaelPlant 15 May 2017 09:25:03PM 1 point [-]

Yeah, that seems plausible, but I'd like GW to set it out and argue for it, rather than for me/us to have to try and guess to work it out. I've searched

http://www.givewell.org/how-we-work/our-criteria/cost-effectiveness and http://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models

And still can't find it.

Comment author: michaelchen 16 May 2017 12:24:51AM *  2 points [-]

From http://blog.givewell.org/2016/12/12/amf-population-ethics/

  • "According to the median GiveWell staff member, averting the death of a child under 5 averts about 8 DALYs (“Bed Nets”, B57)"
  • "each 5-or-over death prevented gets a weight of 4 “young life equivalent” units (“Bed Nets”, B62)"
  • "averting 1 DALY is equivalent to increasing ln(consumption) by one unit for three years (“Bed Nets”, B72)"

I think this "young life equivalent" is the same as what GiveWell calls in other places the "life equivalent."

Comment author: Benito 16 May 2017 09:57:21AM 0 points [-]

Well I don't understand that at all, and it seems to contradict my guess.

I thought DALYs had a more rigorous conversion than "we took our median estimate" and I thought a life was a full life, not just preventing death one time. Strike me wrong on this count.

Comment author: Owen_Cotton-Barratt 16 May 2017 10:37:59AM 2 points [-]

DALYs do use a more defensible analysis; GiveWell aren't using DALYs. This has some good and some bad aspects (related to the discussion in this post, although in this case the downside of defensibility is more that it doesn't let you incorporate considerations that aren't fully grounded).

The problem with just using DALYs is that on many views they overweigh infant mortality (here's my view on some of the issues, but the position that they overweigh infant mortality is far from original). With an internal agreement that they significantly overweigh infant mortality, it becomes untenable to just continue using DALYs, even absent a fully rigorous alternative. Hence falling back on more ad hoc but somewhat robust methods like asking people to consider it and using a median.

[I'm just interpreting GW decision-making from publicly available information; this might easily turn out to be a misrepresentation.]

Comment author: Peter_Hurford  (EA Profile) 16 May 2017 04:51:37PM *  0 points [-]

Back in 2007, GiveWell approximately defined a life saved as saving someone who had a 50% chance of reaching age 60, which would very roughly imply ~30 DALYs per life saved. This analysis is also likely out of date with GiveWell's modern views.

I agree I'd like to see more discussion of this issue.