Comment author: MichaelPlant 13 October 2017 12:02:51PM 4 points [-]

A further thought. If something like OxPrio were to run again, I think it should specify, from the outset, that it won't give money to any charity already considered a mainstream by the EA world (I admit this is vague and would need some tightening). My thinking is (1) this invites participant to do new research, rather than just replicate that done by others. This is both useful for the EA world and also more interesting for the particpants and the audience. I was hoping OxPrio would come up with something more radical that giving money to 80k. (2) I encourages people to question the received wisdom about what the best charities/orgs are without risking looking stupid for choosing something different.

Comment author: MichaelPlant 13 October 2017 10:57:38AM *  9 points [-]

Hello Tom, thanks very much for this write up. Three comments:

I very much admire your ability to self-criticise, but I think you're being overly harsh on yourself. It didn't turn out as well as you hoped, but you couldn't have known that in advance, which was the point. I think this is a good example of what is sometimes called 'hits-based charity': EAs trying new things with a high expected value but a low probability of success. I also hesitate to call this a failure because, as you noted, quite a few lessons were learnt. I think your (only?) substantial mistake was in having too high expectations about what a part-time student group could achieve. Perhaps you took "EAs", who are typically smart, consciousness and driven as your reference group, rather than "student club/society" which no one really expects to be very productive or world-changing.

On reflection, I wonder if OxPrio fell into a sort of research no-man's land. It was too detailed for student, average EAs to engage with, but maybe not in depth enough to attract critical commentary and engagement from full-time researchers, such as those in CEA or GiveWell, whose research you were, to some extent, replicating. I'm not sure who you thought the target audience of your research was.

I think a contributing factor to not having much local, Oxford university engagement is that you'd selected a team. Presumably the people who would be most interested in OxPrio's research applied. I imagine many of the people who applied, but you rejected from the team, then decided that, as a standard psychological reflex, that they didn't want to be involved further (disclaimer: I applied and was rejected, but ended up being really curious about what was OxPrio were doing anyway). Hence the process of selecting alienated much of your intended audience. I don't have suggestion for what would have been better, I just think this is worth factoring in.

Comment author: ClaireZabel 03 October 2017 08:14:28PM 7 points [-]

Personally, I downvoted because I guessed that the post was likely to be of interest to sufficiently few people that it felt somewhat spammy. If I imagine everyone posting with that level of selectivity I would guess the Forum would become a worse place, so it's the type of behavior I think should probably be discouraged.

I'm not very confident about that, though.

Comment author: MichaelPlant 04 October 2017 11:39:06AM 1 point [-]

As a reply, I feel like uncharitably downvoting things because you don't think it matches the interests of large numbers of forums users is the sort of behaviour that should probably be discouraged.

Unless you think the poster is being insincere, I think it's unhelpful and unfriendly to downvote what they say and thus discourage them. It contributes to the feeling EA is an unwelcoming place. If we only talk about causes people already find plausible then we will never improve the causes we work on, and it will drive away people interested in other causes.

I agree that posting about conferences is maybe not very exciting, but it's not obvious this should be posted somewhere else. Facebook would be the alternative, but that not ideal for reasons not worth discussing here.

I upvoted the original post for balance.

Comment author: ThomasSittler 30 September 2017 09:00:14PM *  3 points [-]

Internally we value the average CEA staff hour at ~$75

This is $75 roughly in "EA money" (i.e. OpenPhil's last dollar), yes? It's significantly lower than I thought. However, I suspect that this intuition was biased (upward), because I more often think in terms of "non-EA money". In non-EA money, CEA time would have a much higher nominal value. But if you think EA money can be used to buy good outcomes very cost-effectively (even at the margin) then $75 could make sense.

Comment author: MichaelPlant 01 October 2017 12:50:13AM 1 point [-]

All this was hard to follow.

Comment author: aspencer 26 September 2017 02:42:49PM 2 points [-]

Categorizing quality of life based on personal testimony is a challenging task. The reasons you listed show many specific problems, and more generally, human judgement is fickle and error-prone. For instance, Thinking Fast and Slow claims that we are loss-averse and that we overweight the cost of losing something. I wonder, then, how the responses of perceived quality of life differ between people who were born with particular illnesses (like blindness) and people that suffered from it later in life.

The inherent fallacies in human judgement cause me to wonder if it can ever be a reliable source to quantify the effect of illnesses. At the risk of being hyper-pragmatic, perhaps we should attempt to quantify the effect of illnesses by only considering the degree to which the illness impacts a person's ability to provide useful social function.

Of course, this approach also has many inherent issues. For one, meaningfully quantifying this would be incredibly challenging if not infeasible. It would also likely weight the value of the rich much higher than the value of the poor.

Comment author: MichaelPlant 26 September 2017 07:51:09PM 2 points [-]

If you don't think you can quantify QoL by self-reports, I'm not sure how you're going to be able to quantify useful social functions instead!

FWIW, measuring happiness turns out to be basically fine. You might like this article on the topic which discusses it: http://journals.sagepub.com/doi/10.1111/j.1745-6916.2007.00030.x

Comment author: Milan_Griffes 16 September 2017 05:29:55PM *  0 points [-]

FWIW, in my model I don't assume either of those things. I assume an average counterfactual effect (counter to no rescheduling) of 0.1 HALYs for the 10m in the UK affected by depression or anxiety, not that they all get treatment or everyone benefits from the treatment (to be fair, I specify this in an edit of 14/08/2017 and you might have read it beforehand).

I see, thanks for clarifying. I think an average counterfactual effect of 0.1 HALY is very large (using the assumptions from our model, it implies a 1.20 HALY per treatment improvement in people who try and respond to the treatment: 0.1 average HALY / (0.57 people who seek treatment * 0.44 treatment-seekers who would try psilocybin treatment * 0.33 treatment-takers who respond to treatment).

With a DALY weight for major depression of 0.65, this implies that 1 psilocybin treatment alleviates major depression for 2 years, which is very optimistic. How are you deriving the 0.1 figure?

I don't mention replicability, but then I am assuming the rescheduling only brings a slight improvement

As above, I don't think the assumed improvement is slight. We should definitely include a replicability adjustment as these effects are demonstrated in small-N pilot studies.

I'm not sure why you include the costs of treatment, unless you think EA funders are going to be paying for those too

From my comment further up the thread:

"You could think of this analysis as trying to model whether psychedelic treatments for mental health conditions would be cost-effective if they were available today. For example, consider a promising intervention that would entirely cure someone's depression for a year, but costs $10,000,000 per treatment. We probably wouldn't want to run a ballot initiative to increase access to such a intervention, as it wouldn't be cost-effective even if it were easily accessible."

My understanding is that most public health cost-effectiveness modeling includes all costs of treatment, regardless of who's paying.

That's why I also asked if, and why, you think your drug policy reform strategy is more cost-effective than the one I proposed.

I haven't yet thought enough about what strategy makes the most sense. Our model is designed to be largely strategy-agnostic, as most of the costs are costs-of-treatment.

assumed this policy change will almost certainly occur depending on the polling numbers.

Sort of. I think a lot of the tractability question here hinges on what the polling looks like, which is what we're planning to look into next.

Comment author: MichaelPlant 24 September 2017 04:54:33PM 0 points [-]

I think we're talking past each other on exactly which counterfactuals we have in mind.

There seem to be a couple of bits:

Counterfactual A is: how much better magic mushrooms (MM) is than conventional treatment for people who undergo conventional treatment. This should be multiplied by the number of years before the rescheduling would otherwise have occured.

An additional counterfactual B is: assuming counterfactual A happens and is cheaper than current treatment, that should free up resources for treating the mentally ill who didn't get MM treatment. Should also use the same timescale as A.

I'm now lost on exactly what you're modelling. My model lumps A and B together and assumed a 0.1 HALY increase average across those with depression or anxiety in the UK.

Moving on

My understanding is that most public health cost-effectiveness modeling includes all costs of treatment, regardless of who's paying.

I think this is the wrong way to think about it from an EA perspective. Imagine I'm a rich funder. I will pay for the ballot iniative, but I won't be pay for the health treatments. hence when i do my cost-effectiveness analysis for the ballot, my cost is the ballot expenditure only, the benefit is the counterfactual happiness increase that rules from the new treatments occurring, presuming normal health stuff happens, i.e. doctors upgrade to the new treatments.

As the funder who wants to do the most good, I'm comparing the cost effectivess of this ballot to other things I could fund, like bednets. I'm not funding the treatments themselves, so that's misleading. If I were a government, maybe I'd think about it the way you propose, but then governments dont fund ballot initiative, so that would also be misleading.

It could be the case that, one psychedlics are used in treatment, I could then, as a rich funder, think about paying for those vs paying for bednets. As I said before, that is also an important question. hence we want to split these apart for greater accuracy.

Comment author: MichaelPlant 24 September 2017 04:28:32PM 0 points [-]

Could you make it clearer what you want us, as forum readers, to be doing?

Comment author: MichaelPlant 24 September 2017 04:26:46PM 4 points [-]

Thanks for posting this, it's a really thorough write up of the issue.

I wrote a bit about this about a year ago where I argued effective altruism is overlooking happiness and I'm pleased to see you reached the same conclusions (I also found the the Dolan 2011 paper very persuasive)! I think your analysis was 1. much more substantial than mine and 2. didn't hide the information in an additional document people had to go find (on reflection, that was a mistake on my part). Where I think this criticism of DALYs leads us, in our quest to do the most good, is towards mental health as a substantial cause area and away from physical health.

As a separate point: this post does raise the more general worry about the effectiveness with which information gets shared in EA circles. I'd looked into this before and there's some duplication of effort here: if I'd found a way to make my research better known, the author might have researched something else instead. To be clear, I mean this in no way as a criticism of the author, I think it's just unfortunate. It's not the first time I've come across this phenomenon in the EA world either and I may make a post on the general problem soon.

Comment author: David_Althaus 24 September 2017 10:50:12AM 3 points [-]

Great post!

Nitpick:

For instance, the worst possible health state would be represented by “11111”.

I think "11111" usually refers to full health. (cf. the "EQ-5D Value Sets: Inventory, Comparative Review and User Guide" by Szende, Oppe & Devlin, 2007).

As part of a bigger project on descriptive (population) ethics, I've been working on a literature review of health economics. It also contains a section on the EQ-5D and its weaknesses. Here some excerpts:

Problem II: Impossible health states Another problem is that many health states, such as e.g. 22123 are psychologically impossible or at least very implausible. E.g. if you have “no problems with performing your usual activities (work, study, housework, family or leisure activities, etc.) ”, you can’t, simultaneously, suffer from “extreme depression”. This is immediately obvious to anyone who ever suffered from severe depression.

I’d guess that almost as much as 20% of all EQ-5D health states are psychologically impossible. This indicates that the whole system is suboptimal.

Problem III: Using “immediate death” Another problem is that subjects are often asked to choose between “immediate death” vs. the alternative scenario. However, this means that the subject is unable to say goodbye to their loved ones, or get their affairs in order. Arguably, the difference between dying immediately and dying in e.g. 3 months can make an enormous difference."

(Incorporating the TTO lead-time approach can easily overcome this problem.)

Anway, you write:

First, DALYs are biased towards physical health. The instruments used for eliciting them and affective forecasting errors cause mental health to be underrepresented.

I couldn't agree more.

IMHO, another big problem is the evaluation of states worse than death (SWD) (and states of severe mental illness such as depression arguably belong in this category). For example, most studies don't even allow for SWD assessments. Furthermore, most researchers transform negative evaluations, limiting them to a lower bound of -1. Assuming that people with a history of mental illness more often evaluate health states indicating severe mental illness as highly negative (i.e. give utilities as lower than -1), then this ex-post transformation causes their judgments to have less influence than the judgments of uninformed people who underestimate the severity of mental illness.

I discuss this problem, as well as other problems, in much greater detail in my doc.

I plan on publishing the doc within the next months, but if you're interested I'm happy to send you a link to the current version.

Comment author: MichaelPlant 24 September 2017 04:10:32PM 1 point [-]

Some nitpicks in turn!

I’d guess that almost as much as 20% of all EQ-5D health states are psychologically impossible. This indicates that the whole system is suboptimal.

I don't think this follows. If these states are impossible (I don't disagree) then they'll never come in real life, so it won't matter what people say in the TTOs. As long as people make sensible judgements about the health states that actually occur, it doesn't matter what they say in impossible ones. I think you should push the fact they don't make sensible judgements in general - affective forecasting stuff, etc.

IMHO, another big problem is the evaluation of states worse than death (SWD) (and states of severe mental illness such as depression arguably belong in this category). For example, most studies don't even allow for SWD assessments. Furthermore, most researchers transform negative evaluations, limiting them to a lower bound of -1. Assuming that people with a history of mental illness more often evaluate health states indicating severe mental illness as highly negative (i.e. give utilities as lower than -1), then this ex-post transformation causes their judgments to have less influence than the judgments of uninformed people who underestimate the severity of mental illness.

Curious. Hmm. IIRC, DALYs and QALYs don't have a neutral point: 1 is healthy, 0 is dead, but it's not specified where between 0 and 1 is neutral. Is neutral 0.5? 0? Unless you know where neutral is you can't specify the minimum point on the scale, because it doesn't make sense.

Assuming that people with a history of mental illness more often evaluate health states indicating severe mental illness as highly negative (i.e. give utilities as lower than -1)

What would -1 mean here? DALYs and QALYs aren't well-being scales and can't straightforwardly be interpreted as such.

Comment author: astupple 20 September 2017 02:08:19AM 2 points [-]

While I completely see what you're saying, at the risk of sounding obtuse, I think the opposite of your opener may be true.

"People who do things are not, in general, idea constrained"

The contrary of this statement may be the fundamental point of EA (or at least a variant of it): People who do things in general (outside of EA) tend to act on bad ideas. In fact, EA is more about the ideas underlying what we do than it is about the doing itself. Millions of affluent people are doing things (going to school, work, upgrading their cars and homes, giving to charity), without examining the underlying ideas. EA's success is its ability to convert doers to adopt its ideas. It's creating a pool of doers who use EA ideas instead of conventional wisdom.

Perhaps there are two classes of doers, those already in the EA community who "get it," and those outside who are just plugging away at life. When I think of filling talent gaps, I think that can be filled by (A) EA community members developing skills, and (B) recruiting skilled people to join the community. Group A probably doesn't need good ideas because they've already accepted the ideas of our favorite thinkers etc. The marginal benefit of even better ideas is small. Instead, group A is better off if it simply gets down to the hard work of growing talent. But group B is laboring under bad ideas, and for many, it might not take much at all to get them to substitute bad ideas for EA-ideas. My guess is that, to grow talent, it is easier to convert doers from group B than to optimize doers in group A (which is certainly not to say group A shouldn't do the hard work of optimizing their talent).

There is an odd circularity here- I think I just argued myself out of my original stance. I seem to have just concluded that we shouldn't focus on the ideas of the EA community (which was my original intention) and instead should focus on methods of recruiting.

Maybe I'm arguing that we should develop recruiting ideas?

Also- any suggestions for good formal discussions of the philosophy and sociology of ideas (beyond the slightly nauseating pop business literature)? "Where Good Ideas Come From" by Steven Johnson is excellent, but not philosophically rigorous.

Comment author: MichaelPlant 21 September 2017 02:34:58PM 1 point [-]

You don't need to have argued yourself out of the position. Here's the thought: ideas are important. Evidence in this direction is EA coming along and showing people their previous ideas were bad. Continuing in the same line, unless we think we have all the best ideas already - which would be frighteningly arrogant - that suggests continuing to developing our ideas would be very useful. Hence working on ideas is still very important for those who, as you said, already "get it".

Gworly is right that people aren't lacking ideas. You (astupple) were right that they often lacking good ideas.

Further, on this:

It doesn't take long, relative to the effort that will be invested to do something, to read and think enough to decide what to do, so it seems more likely to me that on the margin we need more desire to do than more curation of ideas about what to do

This is statement lots of philosophers, including those within EA, would disagree with. Indeed, the whole point of 80k is that your life is a long time and it's fitting to spend a non-trivial period reflecting on how to do good.

View more: Next