Comment author: Bernadette_Young 15 May 2017 09:03:15PM *  7 points [-]

The mention of the specific errors found in DCP2 estimates of de-worming efficacy, seem to be functioning here as guilt by association. I can't see any reason they should be extrapolated to all other calculations in different chapters of a >1000 page document. The figure from DCP2 for trachoma treatment directly references the primary source, so it's highly unlikely to be vulnerable to any spreadsheet errors.

The table Toby cites and you reference here (Table 50.1 from DCP2) says "trichiasis surgery". This means surgical treatment for a late stage of trachoma. Trichiasis is not synonymous with trachoma, but a late and severe complication of trachoma infection, by which stage eyelashes are causing corneal friction. It doesn't 'sometimes' lead to blindness, though that is true of trachoma infections when the whole spectrum is considered. Trichiasis frequently causes corneal damage leading to visual impairment and blindness. You are right to point out that not every person with trichiasis will develop blindness, and a "Number Needed to Treat" is needed to correct the estimate from $20 per case of blindness prevented. However we don't have good epidemiological data to say whether that number is 1, 2, 10 or more. Looking at the literature it's likely to be closer to 2 than 10. The uncertainty factor encoded in Peter Singer's use of $100 per person would allow for a number needed to treat of 5.

In this case the term "cure" is appropriate - as trichiasis is the condition being treated by surgery. At one point Toby's essay talks about curing blindness as well as curing trachoma. Strictly speaking trichiasis surgery is tertiary prevention (treatment of a condition which has already caused damage to prevent further damage.), but the error is not so egregious as to elicit the scorn of the hypothetical doctor you quote below. (Source: I am a medical doctor specialising in infectious diseases, I think the WHO fact sheet you link to is overly simplifying matters when it states "blindness caused by trachoma is irreversible").

[Edited to add DOI: I'm married to Toby Ord]

Comment author: saulius  (EA Profile) 15 May 2017 11:58:21PM *  4 points [-]

Thank you very much for writing this. Ironically, I did not do enough fact-checking before making public claims. Now I am not even sure I was right to say that everyone should frequently check facts in this manner because it takes a lot of time and it's easy to make mistakes, especially when it's not the field of expertise for most of us.

Trichiasis surgery then does seem to be absurdly effective in preventing blindness and pain. I am puzzled why GiveWell hasn't looked into it more. Well, they explain it here. The same uncertainty about "Number Needed to Treat".

I want to ask if you don't mind:

  • When literature says that surgery costs ~$20-60 or $7.14, is that for both eyes?
  • Do you think that it's fair to say that it costs say $100 to prevent trachoma-induced blindness? Or is there too much uncertainty to use such number when introducing EA?
Comment author: Julia_Wise 15 May 2017 02:31:36PM 3 points [-]

Thanks for researching and writing this up! We've been discussing the topic a lot at CEA/Giving What We Can over the last few days. I think this points to the importance of flagging publication dates (as GiveWell does, indicating that the research on a certain page was current as of a given date but isn't necessarily accurate anymore). Fact-checking, updating, or just information flagging as older and possibly inaccurate was on our to-do list for materials on the Giving What We Can site, which go back as much as 10 years and sometimes no longer represent our best understanding. I now think it needs to be higher priority than I did.

For individuals rather than organizations, I'm unsure about the best way to handle things like this, which will surely come up again. If someone publishes a paper or blog post, how often are they obliged to update it with corrected figures? I'm thinking of a popular post which used PSI's figure of around $800 to save a child's life. In 2010 when it was written that seemed like a reasonable estimate, but it doesn't now. Is the author responsible for updating the figure everywhere the post was published and re-published? (That's a strong disincentive for ever writing anything that includes a cost-effectiveness estimate, since they're always changing.) Does everyone who quoted it or referred to it need to go back each year and include a new estimate? My guess is it's good practice, particularly when we notice people creating new material that cites old figures, to give them a friendly note with a link to newer sources, with the understanding that this stuff is genuinely confusing and hard to stay on top of.

Comment author: saulius  (EA Profile) 15 May 2017 03:15:16PM 0 points [-]

It's obviously impossible to enforce everyone to update figures all the time. If there is an old publication date, everyone probably understands that it could be outdated. I just think that the date should be always featured prominently. E.g. in this page it could be better. I think that flagging pages the way GiveWell does is a great idea. But featured pages that have no date should probably be checked or updated quite often. I mean pages like "top charities", "what we can achieve" and "myths about aid" in GWWC's case.

Comment author: PeterSinger 13 May 2017 11:47:33PM 7 points [-]

These are good points and I'm suitably chastened for not being sufficiently thorough in checking Toby Ord's claims,
I'm pleased to see that GiveWell is again investigating treating blindness: http://blog.givewell.org/2017/05/11/update-on-our-views-on-cataract-surgery/. In this very recent post, they say: "We believe there is evidence that cataract surgeries substantially improve vision. Very roughly, we estimate that the cost-effectiveness of cataract surgery is ~$1,000 per severe visual impairment reversed.[1]"
The footnote reads: "This estimate is on the higher end of the range we calculated, because it assumes additional costs due to demand generation activities, or identifying patients who would not otherwise have known about surgery. We use this figure because we expect that GiveWell is more likely to recommend an organization that can demonstrate, through its demand generation activities, that it is causing additional surgeries to happen. The $1,000 figure also reflects our sense that cost-effectiveness in general tends to worsen (become more expensive) as we spend more time building our model of any intervention. Finally, it is a round figure that communicates our uncertainty about this estimate overall. But it's reasonable to say that until they complete this investigation, which will be years rather than months, it may be better to avoid using the example of preventing or curing blindness." So the options seem to be either not using the example of blindness at all, or using this rough figure of $1000, with suitable disclaimers. It still leads to 40 cases of severe visual impairment reversed v. 1 case of providing a blind person with a guide dog.

Comment author: saulius  (EA Profile) 14 May 2017 10:57:35AM 2 points [-]

agree :)

Comment author: PeterSinger 12 May 2017 11:23:20PM 13 points [-]

Regrettably, I misspoke in my TED talk when I referred to "curing" blindness from trachoma. I should have said "preventing." (I used to talk about curing blindness by performing cataract surgery, and that may be the cause of the slip.) But there is a source for the figure I cited, and it is not GiveWell. I give the details in The Most Good You Can Do", in an endnote on p. 194, but to save you all looking it up, here it is:

"I owe this comparison to Toby Ord, “The moral imperative towards cost-effectiveness,” http://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/moral_imperative.pdf. Ord suggests a figure of $20 for preventing blindness; I have been more conservative. Ord explains his estimate of the cost of providing a guide dog as follows: “Guide Dogs of America estimate $19,000 for the training of the dog. When the cost of training the recipient to use the dog is included, the cost doubles to $38,000. Other guide dog providers give similar estimates, for example Seeing Eye estimates a total of $50,000 per person/dog partnership, while Guiding Eyes for the Blind estimates a total of $40,000.” His figure for the cost of preventing blindness by treating trachoma comes from Joseph Cook et al., “Loss of vision and hearing,” in Dean Jamison et al., eds., Disease Control Priorities in Developing Countries, 2d ed. (Oxford: Oxford University Press, 2006), 954. The figure Cook et al. give is $7.14 per surgery, with a 77 percent cure rate. I thank Brian Doolan of the Fred Hollows Foundation for discussion of his organization’s claim that it can restore sight for $25. GiveWell suggests a figure of $100 for surgeries that prevent one to thirty years of blindness and another one to thirty years of low vision but cautions that the sources of these figures are not clear enough to justify a high level of confidence."

Now, maybe there is some more recent research casting doubt on this figure, but note that the numbers I use allow that the figure may be $100 (typically, when I speak on this, I give a range, saying that for the cost of training one guide dog, we may be able to prevent somewhere between 400 - 1600 cases of blindness. Probably it isn't necessary even to do that. The point would be just as strong if it were 400, or even 40.

Comment author: saulius  (EA Profile) 13 May 2017 02:01:41PM *  4 points [-]

EDIT: this comment contains some mistakes

To begin with, I want to say that my goal is not to put blame on anyone but to change how we speak and act in the future.

His figure for the cost of preventing blindness by treating trachoma comes from Joseph Cook et al., “Loss of vision and hearing,” in Dean Jamison et al., eds., Disease Control Priorities in Developing Countries, 2d ed. (Oxford: Oxford University Press, 2006), 954. The figure Cook et al. give is $7.14 per surgery, with a 77 percent cure rate.

I am looking at this table from the cited source (Loss of Vision and Hearing, DCP2). It’s 77% cure rate for trachoma that sometimes develops into blindness. Not 77% cure rate for blindness. At least that’s how I interpret it, I can’t be sure because the cited source of the figure in the DCP2’s table doesn’t even mention trachoma! From what I’ve read, sometimes recurrences happen so 77% cure rate from trachoma is much much more plausible. I'm afraid Toby Ord made the mistake of implying that curing trachoma = preventing blindness.

What is more, Toby Ord used the same DCP2 report that GiveWell used and GiveWell found major errors in it. To sum up very briefly:

Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. [...] The estimates on deworming are the only DCP2 figures we’ve gotten enough information on to examine in-depth.

Regarding Fred Hollows Foundation, please see GiveWell’s page about them and this blog post. In my eyes these discredit organization’s claim that it restores sight for $25.

In conclusion, without further research we have no basis for the claim that trachoma surgeries can prevent 400, or even 40 cases of blindness for $40,000. We simply don't know. I wish we did, I want to help those people in the video.

I think one thing that is happening is that we are too eager to believe any figures we find if they support an opinion we already hold. That severely worsens already existing problem of optimizer’s curse.


I also want to add that preventing 400 blindness cases for $40,000 (i.e. one case for $100) to me sounds much more effective than top GiveWell's charities. GiveWell seem to agree, see citations from this page

Based on very rough guesses at major inputs, we estimate that cataract programs may cost $112-$1,250 per severe visual impairment reversed [...] Based on prior experience with cost-effectiveness analyses, we expect our estimate of cost per severe visual impairment reversed to increase with further evaluation. [...] Our rough estimate of the cost-effectiveness of cataract surgery suggests that it may be competitive with our priority programs; however, we retain a high degree of uncertainty.

We tell the trachoma example and then advertise GiveWell, showing that GiveWell’s top and standout charities are not even related to blindness and no one in EA ever talks about blindness. So people probably assume that GiveWell’s recommended charities are much more effective than surgery that cures blindness for $100 but they are not.

Because GiveWell’s estimates for cataract surgeries are based on guesses, I think we shouldn’t use those figures in introductory EA talks as well. We can tell the disclaimers but the person who hears the example might skip them when retelling the thought experiment (out of desire to sound more convincing). And then the same will happen.

Comment author: BenHoffman 13 May 2017 02:22:40AM 1 point [-]

On the ableism point, my best guess is that the right response is to figure out the substance of the criticism. If we disagree, we should admit that openly, and forgo the support of people who do not in fact agree with us. If we agree, then we should account for the criticism and adjust both our beliefs and statements. Directly optimizing on avoiding adverse perceptions seems like it would lead to a distorted picture of what we are about.

Comment author: saulius  (EA Profile) 13 May 2017 10:31:53AM *  1 point [-]

The article Vollmer cites says:

Singer’s idea about the relative value of guide dogs sets up a false dichotomy, assuming that you can fund guide dogs or fund medical prevention. In fact, you can do both.

In this case that seems to be the substance of the criticism. You can't anticipate every counter-argument one could make when talking to bigger audiences, but this one is pretty common. It might be necessary to say

if I have to decide where to donate my $ 100...

Not sure it would help, it could be that such arguments trigger bad emotions for other reasons and the counter-arguments we hear are just rationalizations of those emotions. It does feel like a minefield.

Therefore, when comparing any 2 charities while introducing someone (especially an audience) to EA, we must phrase it carefully and sensitively. BTW, I think there is something to learn from way Singer phrased it in the TED talk:

Take, for example, providing a guide dog for a blind person. That's a good thing to do, right? Well, right, it is a good thing to do, but you have to think what else you could do with the resources. It costs about 40,000 dollars...

Comment author: TruePath 11 May 2017 07:37:31AM 0 points [-]

This feels like nitpicking that gives the impression of undermining Singer's original claim when in reality the figures support them. I have no reason to believe Singer was claiming that of all possible charitable donations trauchoma is the most effective, merely to give the most stunningly large difference in cost effectiveness between charitable donations used for comparable ends (both about blindness so no hard comparisons across kinds of suffering/disability).

I agree that within the EA community and when presenting EA analysis of cost-effectiveness it is important to be upfront with the full complexity of the figures. However, <b>Singer's purpose at TED isn't to carefully pick the most cost effective donations but to force people to confront the fact that cost effectiveness matters.</b>. While those of us already in EA might find a statement like "We prevent 1 year of blindness for every 3 surgeries done which on average cost..." perfectly compelling the audience members who aren't yet persuaded simply tune out. After all it's just more math talk and they are interested in emotional impact. The only way to convince them is to ignore getting the numbers perfectly right and focus on the emotional impact of choosing to help a blind person in the US get a dog rather than many people in poor countries avoid blindness.

Now it's important that we don't simplify in misleading ways but even with the qualifications here it is obvious that it still costs orders of magnitude more to train a dog than prevent blindness via this surgery. Moreover, once one factors in considerations like pain, the imperfect replacement for eyes provided by a dog, etc.. the original numbers are probably too favorable to dog training as far as relative cost effectiveness goes.

This isn't to say that your point here isn't important regarding people inside EA making estimates or givewell analysis or the like. I'm just pointing out that it's important to distinguish the kind of thing being done at a TED talk like this from that being done by givewell. So long as when people leave the TED talk their research leaves the big picture in place (dogs >>>> trauchoma surgery) it's a victory.

Comment author: saulius  (EA Profile) 11 May 2017 08:44:16AM *  16 points [-]

I think there is truth in what you said. But I also have disagreements:

"The only way to convince them is to ignore getting the numbers perfectly right and focus on the emotional impact"

That's a dangerous line of reasoning. If we can't make a point with honest numbers, we shouldn't make the point at all. We might fail to notice when we are wrong when we use bogus numbers to prove whatever opinion we already hold.

What is more, many people who become EAs after hearing such TED talks already think in numbers. They continue in believing the same numbers afterwards and are more likely to dismiss other cause areas because of it. I myself once mocked a co-worker for taking an effort to recycle when the same effort could do so much more impact for people in Africa. That's wrong in any case, but I was probably wrong in my reasoning too because of numbers.

Also, I'm afraid that some doctor will stand up during an EA presentation and say

You kids pretend to be visionaries, but in reality you don't have the slightest idea what you are talking about. Firstly, it's impossible to cure trachoma induced blindness. Secondly [...] You should go back to play in your sandboxes instead of preaching adults how to solve real world problems

Also, I'm afraid that the doctor might be partially right

30

Fact checking comparison between trachoma surgeries and guide dogs

In a 2013 TED talk Peter Singer claims “It costs about 40,000 dollars to train a guide dog and train the recipient so that the guide dog can be an effective help to a blind person. It costs somewhere between 20 and 50 dollars to cure a blind person in... Read More
Comment author: Peter_Hurford  (EA Profile) 27 August 2016 01:11:43AM 3 points [-]

I imagine there's also some selection bias -- those doing ETG often have jobs that are harder to leave to go to a conference.

Comment author: saulius  (EA Profile) 27 August 2016 12:08:53PM 1 point [-]

Also more hardcore EAs are more likely to come to the conference. But these percentages don't mean much to me because people can't be easily categorized into EAs and non-EAs. There are varying degrees of EAness.

Comment author: John_Maxwell_IV 26 August 2016 11:24:51AM 1 point [-]

Only 10% of attendees at EAG were earning to give as their long-term plan for impact.

I'd be interested to learn more about this number. I assume some attendees were uncertain about their long-term plan? If so, some of these people may end up earning to give?

Also what do 80K's target percentages vs actual percentages look like for other EA career options?

Comment author: saulius  (EA Profile) 27 August 2016 11:59:12AM *  3 points [-]

In EA survey (https://eahub.org/sites/effectivealtruismhub.com/files/SurveyReport2015.pdf page 18) there was a question "What broad career path are you planning to follow?". Results: Direct charity / nonprofit work: 190; Earning to give: 512; Research: 362; None of these: 375; Didn't answer: 913.

Comment author: RomeoStevens 27 August 2016 05:05:54AM *  3 points [-]

WRT "crazy", I mean things that might not pass initial sniff tests (absurdity heuristic), things that are outside or far away in reference class space and thus are hard to reason about via analogy, things that make taboo tradeoffs and are thus bad to talk about publicly for brand reasons, or just plain audaciousness. Maybe there are more cues for thinking about these, haven't tried to apply tools to it yet.

Comment author: saulius  (EA Profile) 27 August 2016 11:48:10AM 1 point [-]

Crazy to EAs or crazy to general population? If it's the latter, AI-safety research qualifies. If it's the former, EAF's wild animal suffering research might still qualify. If you disagree, tell an example of a crazy idea.

View more: Next