4

Cost-effectiveness analysis: drug liberalization holds promise as a mental health intervention

Cross-posted to the Enthea site.

In a nutshell:

  • If it polls well, a California ballot initiative that increases medical access to psychedelic drugs is a promising intervention in terms of DALYs averted.
  • We estimate the cost-per-DALY-averted to range from $52,000/DALY to $442,000/DALY, with a best-guess estimate of $119,000/DALY. Full analysis available here.
  • This figure is an estimate of the minimum expected benefit from a ballot initiative, not the total expected benefit. Many of the outcomes that matter most are not included in the analysis (e.g. pushing US federal drug policy away from a crime model & towards a public health model), as they are speculative and difficult to model.


The model is quite rough; we'd be very grateful for comments and critiques of it.


Major caveats & considerations:

  • The model assumes that a ballot initiative polls well. Our current understanding is that if polling looks promising, then there's a high chance of the initiative succeeding. If polling doesn't look promising, there's minimal chance of success.
    • Because of the binary distribution of outcomes, we didn't include a linear discount for “probability that initiative polls well.” Instead, the model is to be read as if the initiative polls well. If the polling isn't promising, a ballot initiative isn't worth pursuing at this time.
    • An important takeaway here: polling on drug liberalization is important for assessing the tractability of an initiative. We plan to write more about this in the future.
  • The model does not include many of the benefits could plausibly result from increasing access to psychedelics, including:
    • (for psychedelic users) Alleviation of mild depression and anxiety
    • (for psychedelic users) Anti-addiction aid for behavioral addictions (gambling, online gaming, social media, etc.)
    • (for psychedelic users) Improved psychological openness
    • (for psychedelic users, speculative) Improved relationships between friends, co-workers, and significant others
    • (for psychedelic users, speculative) Improved self-efficacy
    • (societal) Reduced sentences for drug offenders
    • (societal) Increased personal liberty (which appears to correlate strongly with self-reported life satisfaction)
    • (societal) Pushing US federal drug policy towards a public health model, i.e. away from a crime model
  • Most of the modeled cost comes from costs of treatment, not costs of the ballot initiative.
    • Costs of treatment are included to avoid confusions around charitable leverage.
    • Costs of treatment are speculative, as psychedelic treatments aren't currently carried out. We estimated the cost-of-treatment by assuming that each patient would be treated under a protocol similar to the protocols of the pilot studies for psilocybin and MDMA (psychotherapy sessions before and after the dose, sitter present during the dose).


Q & A:

  • What intervention is being modeled, exactly?
    • The analysis models the costs & benefits of a ballot initiative that increases access to psychedelics such that sufferers of depression, PTSD, alcoholism, and tobacco addiction can receive psychedelic treatment for these conditions in California.
    • The analysis is intentionally agnostic re: the policy specifics of such an initiative. We are still learning about the strategic landscape here, and are currently unsure what specific initiative text would be best.
  • Why model mental health impacts if the largest benefits would come from other impacts?
    • Minimum expected benefit: the mental health benefits from psychedelics are one of the least speculative benefits of liberalizing US drug policy. Our model gives the expected benefit a ballot initiative would achieve at minimum, if all other impacts were net zero. We plan to write more about other plausible benefits soon.
    • Easier to model: other impacts are complicated by flow-through effects, etc. It would be very difficult to build a believable quantitative analysis of these impacts.
  • Is a ballot initiative cost-effective on the grounds of mental health benefits alone?
    • It depends on what you mean by cost-effective.
      • Public health interventions are generally considered “cost-effective” if cost-per-DALY is 1-3x gross national income per capita. GNI per capita in the US is $57,500, so a “cost-effective” intervention in the US is between $57,500 and $172,500 per DALY. By this criterion, the mental health effects of a ballot initiative are cost-effective under best-guess and optimistic assumptions in the model (and not cost-effective under pessimistic assumptions).
      • GiveWell top charities achieve much lower cost-per-DALYs with interventions in the developing world (on the order of $100s or $1,000s per DALY). However, these results rest on philosophical assumptions about population ethics (for AMF, discussion here and here) and empirical interpretations of the interventions' effects on happiness and income (for GiveDirectly and deworming interventions, some discussion here).
        • We don't want to take a position on these issues here; we raise them only to point out that arriving at apples-to-apples comparisons of heterogenous interventions is very complicated, and that there are reasonable assumptions under which GiveWell top charities are not extraordinarily cost-effective compared to other interventions.
        • Mental health interventions have the benefit of directly increasing life satisfaction (which physical health and economic development interventions can only do indirectly).
        • Further, our analysis is not attempting to model the full impacts of a drug liberalization initiative. Rather, it's attempting to model the minimum benefit we should expect from such an initiative, even if all the speculative benefits turned out to have no impact.
  • Why doesn't the model account for adverse effects of psychedelics, like bad trips?
    • Contrary to common belief, psychedelics are quite safe, and very nontoxic – see the “Background and safety” section here, as well as the “Safety of psychedelics” section of Nichols 2016.
    • Carbonaro et al. 2016 surveyed 2,000 psychedelic users who self-reported having a “challenging experience” with psychedelics. Despite the difficulty of the experience, 84% reported benefiting from the bad trip. Given this result, it's not obvious that bad trips are pathological.
  • One plausible strategy for increasing access to psychedelics is to decriminalize all drugs. Wouldn't all-drug decriminalization cause a major uptick in use of addictive drugs?
    • It's definitely possible, though Portugal decriminalized all drugs in 2001 and didn't see an uptick in drug use.
    • Assessing this impact is complicated – all-drug decriminalization would increase access both to addictive drugs (e.g. opioids) and psychedelic anti-addiction treatments for these drugs; it's unclear what the net impact of this would be.



Thanks to Chris Smith and an anonymous collaborator for their help building the model and thinking through relevant considerations.

Comments (14)

Comment author: Michael_S 14 September 2017 10:52:08PM *  5 points [-]

Hey; I made some comments on this on the doc, but I thought it was worth bringing them to the main thread and expanding.

First of all, I'm really happy to sea other EAs looking at ballot measures. They're a potentially very high EV method of passing policy/raising funding. They're particularly high value per dollar when spending on advertising is limited/nothing since the increased probability of passage from getting a relatively popular measure on the ballot is far more than the increased probability from spending the same amount advertising for it.

Also, am I correct in interpreting that you assume 100% chance of passage in your model conditional on good polling? Polling can help, but ballot measure polling does have a lot of error (in both directions). So even a popular measure in polling is hardly guarantee of passage (http://themonkeycage.org/2011/10/when-can-you-trust-polling-about-ballot-measures/).

Finally, in your EV estimates, you seem to be focus on the individual treatment cost of the intervention, which overwhelms the cost of the ballot measure. I don't think this is getting at the right question when it comes to running a ballot measure. I believe the gains from the ballot measure should be the estimated sum of the utility gains from people being able to purchase the drugs multiplied by the probability of passage; the costs should be how much it would cost to run the campaign. On the doc, you made the point that Givewell doesn't include leverage on other funding in their estimates, but when it comes to ballot measures, leverage is exactly what you're trying to produce, so I think an estimate is important.

Comment author: Milan_Griffes 15 September 2017 01:54:09AM *  1 point [-]

Thanks for the comments!

am I correct in interpreting that you assume 100% chance of passage in your model conditional on good polling?

No, the best-guess input is an 80% chance of passage, conditional on good polling and sufficient funding (see row 81). What "good" means here is a little underspecified – an initiative that polls at 70% favorability would have a much higher probability of passing than one that polls at 56%.

you seem to be focus on the individual treatment cost of the intervention, which overwhelms the cost of the ballot measure.

Right. You could think of this analysis as trying to model whether psychedelic treatments for mental health conditions would be cost-effective if they were available today. For example, consider a promising intervention that would entirely cure someone's depression for a year, but costs $10,000,000 per treatment. We probably wouldn't want to run a ballot initiative to increase access to such a intervention, as it wouldn't be cost-effective even if it were easily accessible.

Comment author: Michael_S 15 September 2017 02:48:50AM 0 points [-]

Cool; had missed that row. Yeah, if it polls, 70% the chance of passage might be close to 80%. Conditional upon that level of support, your estimate seems reasonable to me (assuming the ballot summary language would not be far more complex than the polled language).

Yeah, I agree that it being an effective treatment is a necessary precursor to it being a good ballot law to pass by ballot initiative and part of the EV calculation for spending money on the ballot measure itself.

Comment author: Peter_Hurford  (EA Profile) 14 September 2017 11:45:00PM 1 point [-]

I believe the gains from the ballot measure should be the estimated sum of the utility gains from people being able to purchase the drugs multiplied by the probability of passage; the costs should be how much it would cost to run the campaign. On the doc, you made the point that Givewell doesn't include leverage on other funding in their estimates, but when it comes to ballot measures, leverage is exactly what you're trying to produce, so I think an estimate is important.

One potential way of thinking about this is that the ballot measure in itself does not accomplish much, it just "unlocks" the ability for people to more cheaply help themselves. This could be modeled as the costs of the ballot measure + the costs of people helping themselves over a stream of X years, put against the benefits of people helping themselves over X years. I would use 5 for X, assuming that a lot can change in 5 years and maybe drug legalization would happen anyway, but I think a higher value for X could also be justified.

This kind of (costs of unlocking + costs of what is unlocked over time) vs. benefits of what is unlocked over time is also how I model the cost-benefit of developing a new medicine (like a vaccine), since the medicine is useless unless it is actually given to people, which costs additional money.

Comment author: Michael_S 14 September 2017 11:52:15PM 1 point [-]

That seems similar to Milan_Griffes' approach. However, when we're comparing ballot measures to other opportunities, I think the relevant cost to EA would be the cost to launch the campaign. That's what EAs would actually be spending money on and what could be spent on other interventions.

We don't have to assume away the additional costs of getting the medicine, but that can be factored into the benefit (ie. the net benefit is the gains they would get from the medicine - the gains they lose from giving up the funds to purchase the drugs)

Comment author: ThomasSittler 15 September 2017 08:36:42AM 1 point [-]

the model is to be read as if the initiative polls well

Have you thought about some cheap ways to get more information on how this is likely to poll (even poor quality info) ?

Comment author: Milan_Griffes 15 September 2017 02:09:27PM 0 points [-]

Yes. Two projects that seem promising here: (1) a systematic review of recent public opinion polls on psychedelics, (2) running Google Surveys on possible ballot initiative texts: https://www.google.com/analytics/surveys/

Comment author: MichaelPlant 15 September 2017 08:53:03AM 0 points [-]

Hello Miles and thanks for all this, good to see it's getting discussed.

As you probably saw, I produced a cost-effectiveness model of drug policy campaigning in the final post of my (rather long) series on the subject. In that, I suggest it's plausible drug policy reform, again just by allowing the use of psychedelics to treat mental health, could be in the range of $166/HALYs (Happiness adjusted life years), which would make it some 300 times more cost effective than you suggest.

It would be really helpful if you could say where and why you disagree with my model, given that, if you think drug policy reform is a promising intervention, my analysis suggest it's much more promising that yours does! For simplicity, let's assume HALYs and DALYs are just different types of apples, then I'd want to know why you think the structure of my model is wrong.

Comment author: Milan_Griffes 15 September 2017 02:15:37PM *  2 points [-]

I haven't engaged closely with your model, but here are some differences that immediately stand out:

  • Your analysis models the a change that impacts the entire UK, whereas ours models a change that impacts California.
  • Your model assumes that everyone in the UK who might benefit from treatment would seek treatment.
  • Your model assumes that everyone who receives treatment would benefit from treatment.
  • Your model doesn't include a replicability adjustment, to discount effect sizes due to the limited amount of evidence.
  • As far as I can tell, your model doesn't include costs of treatment, only costs of rescheduling.
Comment author: MichaelPlant 16 September 2017 01:21:38PM 1 point [-]

Hello Milan!

Your model assumes that everyone in the UK who might benefit from treatment would seek treatment. Your model assumes that everyone who receives treatment would benefit from treatment.

FWIW, in my model I don't assume either of those things. I assume an average counterfactual effect (counter to no rescheduling) of 0.1 HALYs for the 10m in the UK affected by depression or anxiety, not that they all get treatment or everyone benefits from the treatment (to be fair, I specify this in an edit of 14/08/2017 and you might have read it beforehand).

I don't mention replicability, but then I am assuming the rescheduling only brings a slight improvement (in the latter, more optimistic estimate I discuss whether this might be higher than 0.1 HALYs). I also mention the confusing possibility that treating some people with psychedelics might free up health care resources for other treatments.

I don't include costs of treatment, as I'm assuming this is an EA-funded campaign where our job, and what the money goes to, is changing the law and then allowing normal health care distribution to occur in the new scenario (i.e. in the US = insurer pays, in UK = govt pays).

Hence, looking at your model, I'm not sure why you include the costs of treatment, unless you think EA funders are going to be paying for those too. Even if you do think this, we should really want to have two seperate models, one for "cost of changing the law, assuming health practices then change aoccrdingly" and another for "cost effectiveness to EA funders to provide psychedelic therapy if it's available". As an aside, your model is really thorough, and I'm grateful to you for having put it together, good stuff!

This may also sound picky, but what we want to know (1) what is the most suitable model is for any give intervention, so if we're disagreeing with each other, we want to know why we're disagreeing, not just that we're disagreeing. Hence I was asking where and why you disagreed with my model.

You might reply your model is separate (campaign lobbying in UK vs ballot iniative and treatment funding in the US(?)) but, we also want to know (2) whether some new intervention is more cost-effective than all other current interventions an EA could fund (on one or more moral theory). If it's not more cost-effective then, all things considered, it would be bad to fund it. That's why I also asked if, and why, you think your drug policy reform strategy is more cost-effective than the one I proposed.

As it stands, we are perhaps comparing apples and oranges: you seem to have bundled treatment in with a policy change, and assumed this policy change will almost certainly occur depending on the polling numbers. I've just looked at policy change and estimated how much we could spend on it to change public/policy opinion and it still be more effective than AMF, assuming AMF is the current most cost-effective intervention. Hence we may need to get on the same page on this first.

Comment author: Milan_Griffes 16 September 2017 05:29:55PM *  0 points [-]

FWIW, in my model I don't assume either of those things. I assume an average counterfactual effect (counter to no rescheduling) of 0.1 HALYs for the 10m in the UK affected by depression or anxiety, not that they all get treatment or everyone benefits from the treatment (to be fair, I specify this in an edit of 14/08/2017 and you might have read it beforehand).

I see, thanks for clarifying. I think an average counterfactual effect of 0.1 HALY is very large (using the assumptions from our model, it implies a 1.20 HALY per treatment improvement in people who try and respond to the treatment: 0.1 average HALY / (0.57 people who seek treatment * 0.44 treatment-seekers who would try psilocybin treatment * 0.33 treatment-takers who respond to treatment).

With a DALY weight for major depression of 0.65, this implies that 1 psilocybin treatment alleviates major depression for 2 years, which is very optimistic. How are you deriving the 0.1 figure?

I don't mention replicability, but then I am assuming the rescheduling only brings a slight improvement

As above, I don't think the assumed improvement is slight. We should definitely include a replicability adjustment as these effects are demonstrated in small-N pilot studies.

I'm not sure why you include the costs of treatment, unless you think EA funders are going to be paying for those too

From my comment further up the thread:

"You could think of this analysis as trying to model whether psychedelic treatments for mental health conditions would be cost-effective if they were available today. For example, consider a promising intervention that would entirely cure someone's depression for a year, but costs $10,000,000 per treatment. We probably wouldn't want to run a ballot initiative to increase access to such a intervention, as it wouldn't be cost-effective even if it were easily accessible."

My understanding is that most public health cost-effectiveness modeling includes all costs of treatment, regardless of who's paying.

That's why I also asked if, and why, you think your drug policy reform strategy is more cost-effective than the one I proposed.

I haven't yet thought enough about what strategy makes the most sense. Our model is designed to be largely strategy-agnostic, as most of the costs are costs-of-treatment.

assumed this policy change will almost certainly occur depending on the polling numbers.

Sort of. I think a lot of the tractability question here hinges on what the polling looks like, which is what we're planning to look into next.

Comment author: MichaelPlant 24 September 2017 04:54:33PM 0 points [-]

I think we're talking past each other on exactly which counterfactuals we have in mind.

There seem to be a couple of bits:

Counterfactual A is: how much better magic mushrooms (MM) is than conventional treatment for people who undergo conventional treatment. This should be multiplied by the number of years before the rescheduling would otherwise have occured.

An additional counterfactual B is: assuming counterfactual A happens and is cheaper than current treatment, that should free up resources for treating the mentally ill who didn't get MM treatment. Should also use the same timescale as A.

I'm now lost on exactly what you're modelling. My model lumps A and B together and assumed a 0.1 HALY increase average across those with depression or anxiety in the UK.

Moving on

My understanding is that most public health cost-effectiveness modeling includes all costs of treatment, regardless of who's paying.

I think this is the wrong way to think about it from an EA perspective. Imagine I'm a rich funder. I will pay for the ballot iniative, but I won't be pay for the health treatments. hence when i do my cost-effectiveness analysis for the ballot, my cost is the ballot expenditure only, the benefit is the counterfactual happiness increase that rules from the new treatments occurring, presuming normal health stuff happens, i.e. doctors upgrade to the new treatments.

As the funder who wants to do the most good, I'm comparing the cost effectivess of this ballot to other things I could fund, like bednets. I'm not funding the treatments themselves, so that's misleading. If I were a government, maybe I'd think about it the way you propose, but then governments dont fund ballot initiative, so that would also be misleading.

It could be the case that, one psychedlics are used in treatment, I could then, as a rich funder, think about paying for those vs paying for bednets. As I said before, that is also an important question. hence we want to split these apart for greater accuracy.

Comment author: ThomasSittler 15 September 2017 08:34:52AM 0 points [-]

The well-being improvement estimates seem to come from small pilot studies with no control group, showing very large impacts. I don't have enough background to guess how large these impacts are relative to other known treatments or placebo. The smoking impacts come from Johnson et al. 2017 (N = 15), the depression impacts come from Carhart-Harris et al. 2016 (N = 12).

Comment author: Milan_Griffes 15 September 2017 02:26:27PM *  0 points [-]

It's true that these effects all come from small-N pilot studies. Each effect size is discounted substantially by a replicability adjustment (best-guess input is an 80% discount).

Most of the studies considered didn't have a control group, though the PTSD study (Mithoefer et al. 2010) did. We included a placebo-effect adjustment for that study.

Interestingly, the depression study participants (Carhart-Harris et al. 2016) had all failed to respond to other depression treatments, so it's plausible that placebo effects were less strong in this population.

Similarly, most (all?) of the smoking study participants (Johnson et al. 2014) were longtime smokers who had made multiple unsuccessful attempts to quit in the past. Plausible that placebo effects were less strong in the population as well.