Comment author: Peter_Hurford  (EA Profile) 19 September 2017 11:21:34PM *  4 points [-]

I can see how the non-response rate looks alarming and I definitely owe some context for that.

One thing we tried this year was a separate donations only survey, where people only reported their donations and a few other questions. Race was not on this slimmer survey. 554 did not answer this race question because they were never asked it.

Another source of apparent non-response is the fact that we asked people Yes or No for four different races (White, Black, Asian, and Hispanic). It looks like some people checked "Yes" for one race but did not explicitly check "No" for the others. This accounts for another 120 people.

Combining these first two reasons, there are only 67 people genuinely ignoring the race question. You then have to account for survey fatigue, where people answer some of the questions at the beginning of the survey, but then get bored, busy, or distracted and quit the survey without answering the rest of the questions. Given that race was at the bottom of the seventh page of the survey, this could be acute. I couldn't find anyone who neglected to answer the race question but did answer a question after the race question, so it looks like these three factors may fully account for all non-response.

Comment author: Bernadette_Young 20 September 2017 01:11:29PM 4 points [-]

That's still a very important point that doesn't seem to have been made in the analysis here: the demographic questions were not included in the questions put to all respondents. Since there are good reasons to think that people taking the "full" and "donations only" survey will differ systematically (e.g. more likely to have been involved with EA for longer). If the non responses are not random that's an important caveat on all these findings and very much limits any comparisons that can be done over time. I can't seem to see it discussed in the post?

Comment author: saulius  (EA Profile) 15 May 2017 11:58:21PM *  4 points [-]

Thank you very much for writing this. Ironically, I did not do enough fact-checking before making public claims. Now I am not even sure I was right to say that everyone should frequently check facts in this manner because it takes a lot of time and it's easy to make mistakes, especially when it's not the field of expertise for most of us.

Trichiasis surgery then does seem to be absurdly effective in preventing blindness and pain. I am puzzled why GiveWell hasn't looked into it more. Well, they explain it here. The same uncertainty about "Number Needed to Treat".

I want to ask if you don't mind:

  • When literature says that surgery costs ~$20-60 or $7.14, is that for both eyes?
  • Do you think that it's fair to say that it costs say $100 to prevent trachoma-induced blindness? Or is there too much uncertainty to use such number when introducing EA?
Comment author: Bernadette_Young 16 May 2017 09:22:18AM *  3 points [-]

Thanks for responding!

I think it's laudable to investigate the basis for claims as you've done. It's fair to say evidence appraisal and communication really is a specialist area in its own right, and outside our ares of expertise it's common to make errors in doing so. And while we all like evidence confirms what we think, other biases may be at play. I think some people in effective altruism also put a high value on identifying and admitting mistakes, so we might also be quick to jump on a contrary assessment even if it has some errors of its own.

I think your broader point about communicating the areas and extent of uncertainty is important, but the solution to how we do that when communicating in different domains is not simple. For example, you can look at how NICE investigates the efficacy of clinical interventions. They have to distill 1000's of pages of evidence into a decision, and even the 'summary' of that can be 100s of pages long. At the front of that will be an 'executive summary' which can't possibly capture all the ares of uncertainty and imperfect evidence, but usually represents their best assessment because ultimately they have to make concrete recommendations.

Another approach is that seen in the Cochrane Systematic Reviews. These take a very careful approach to criticising the methodology of all studies included in their analysis. A running joke though its that every Cochrane review reaches the same conclusion: "More Evidence is Needed". This is precise and careful, but often lacks any practical conclusion.

Re your 2 questions:

It's $7.14 for 1 eye (in 2001) with 77% success, according to this source: In Toby Ord's essay he uses this to derive the "less than $20 per person" figure (7.14 *2 /(0.77) = $18.5 ) So that's both eyes (in 2001 terms).

My main area of uncertainty on that figure is around number needed to treat. I've spoken to a colleague who is an ophthalmologist and has treated trichiasis in Ghana. Her response was "trachoma with trichiasis always causes blindness". But in the absence of solid epidemiology to back it up, I think it's wise to allow for NNT being higher than 1. I would be comfortable with saying that for about $100 we can prevent trachoma-induced blindness, in order to contrast that with things that we consider a reasonable buy in other contexts. (I haven't assessed any orgs to know if there are orgs who do it for that little: they may for instance do surgeries on a wider range of conditions with varying DALYs gained per dollar spent).

Comment author: saulius  (EA Profile) 13 May 2017 02:01:41PM *  5 points [-]

EDIT: this comment contains some mistakes

To begin with, I want to say that my goal is not to put blame on anyone but to change how we speak and act in the future.

His figure for the cost of preventing blindness by treating trachoma comes from Joseph Cook et al., “Loss of vision and hearing,” in Dean Jamison et al., eds., Disease Control Priorities in Developing Countries, 2d ed. (Oxford: Oxford University Press, 2006), 954. The figure Cook et al. give is $7.14 per surgery, with a 77 percent cure rate.

I am looking at this table from the cited source (Loss of Vision and Hearing, DCP2). It’s 77% cure rate for trachoma that sometimes develops into blindness. Not 77% cure rate for blindness. At least that’s how I interpret it, I can’t be sure because the cited source of the figure in the DCP2’s table doesn’t even mention trachoma! From what I’ve read, sometimes recurrences happen so 77% cure rate from trachoma is much much more plausible. I'm afraid Toby Ord made the mistake of implying that curing trachoma = preventing blindness.

What is more, Toby Ord used the same DCP2 report that GiveWell used and GiveWell found major errors in it. To sum up very briefly:

Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. [...] The estimates on deworming are the only DCP2 figures we’ve gotten enough information on to examine in-depth.

Regarding Fred Hollows Foundation, please see GiveWell’s page about them and this blog post. In my eyes these discredit organization’s claim that it restores sight for $25.

In conclusion, without further research we have no basis for the claim that trachoma surgeries can prevent 400, or even 40 cases of blindness for $40,000. We simply don't know. I wish we did, I want to help those people in the video.

I think one thing that is happening is that we are too eager to believe any figures we find if they support an opinion we already hold. That severely worsens already existing problem of optimizer’s curse.

I also want to add that preventing 400 blindness cases for $40,000 (i.e. one case for $100) to me sounds much more effective than top GiveWell's charities. GiveWell seem to agree, see citations from this page

Based on very rough guesses at major inputs, we estimate that cataract programs may cost $112-$1,250 per severe visual impairment reversed [...] Based on prior experience with cost-effectiveness analyses, we expect our estimate of cost per severe visual impairment reversed to increase with further evaluation. [...] Our rough estimate of the cost-effectiveness of cataract surgery suggests that it may be competitive with our priority programs; however, we retain a high degree of uncertainty.

We tell the trachoma example and then advertise GiveWell, showing that GiveWell’s top and standout charities are not even related to blindness and no one in EA ever talks about blindness. So people probably assume that GiveWell’s recommended charities are much more effective than surgery that cures blindness for $100 but they are not.

Because GiveWell’s estimates for cataract surgeries are based on guesses, I think we shouldn’t use those figures in introductory EA talks as well. We can tell the disclaimers but the person who hears the example might skip them when retelling the thought experiment (out of desire to sound more convincing). And then the same will happen.

Comment author: Bernadette_Young 15 May 2017 09:03:15PM *  7 points [-]

The mention of the specific errors found in DCP2 estimates of de-worming efficacy, seem to be functioning here as guilt by association. I can't see any reason they should be extrapolated to all other calculations in different chapters of a >1000 page document. The figure from DCP2 for trachoma treatment directly references the primary source, so it's highly unlikely to be vulnerable to any spreadsheet errors.

The table Toby cites and you reference here (Table 50.1 from DCP2) says "trichiasis surgery". This means surgical treatment for a late stage of trachoma. Trichiasis is not synonymous with trachoma, but a late and severe complication of trachoma infection, by which stage eyelashes are causing corneal friction. It doesn't 'sometimes' lead to blindness, though that is true of trachoma infections when the whole spectrum is considered. Trichiasis frequently causes corneal damage leading to visual impairment and blindness. You are right to point out that not every person with trichiasis will develop blindness, and a "Number Needed to Treat" is needed to correct the estimate from $20 per case of blindness prevented. However we don't have good epidemiological data to say whether that number is 1, 2, 10 or more. Looking at the literature it's likely to be closer to 2 than 10. The uncertainty factor encoded in Peter Singer's use of $100 per person would allow for a number needed to treat of 5.

In this case the term "cure" is appropriate - as trichiasis is the condition being treated by surgery. At one point Toby's essay talks about curing blindness as well as curing trachoma. Strictly speaking trichiasis surgery is tertiary prevention (treatment of a condition which has already caused damage to prevent further damage.), but the error is not so egregious as to elicit the scorn of the hypothetical doctor you quote below. (Source: I am a medical doctor specialising in infectious diseases, I think the WHO fact sheet you link to is overly simplifying matters when it states "blindness caused by trachoma is irreversible").

[Edited to add DOI: I'm married to Toby Ord]

Comment author: Bernadette_Young 20 April 2017 08:24:17PM *  8 points [-]

I'm pleased to see the update on GWWC recommendations; it was perturbing to have such different messages being communicated in different channels.

However I'm really disappointed to hear the Giving What We Can trust will disappear - not least because it means I no longer have a means to leave a legacy to effective charities in my will (which I'll now need to change). Previously the GWWC trust meant I had a means of leaving money, hedging against changes in the landscape of what's effective, run by an org whose philosophy I agree with and whose decisions I had a good track record of trusting. EA funds requires I either specify organisations (which I can do myself in a will, but might not be the best picks at a relevant time), or trust a single individual in whom I don't have the same confidence. Also if a legacy is likely to be a substantial amount of money I am more risk averse about where it goes.

Comment author: Alex_Barry 06 February 2017 09:17:15PM *  1 point [-]

Yes I mention the issues associated with college-based randomization in section 3.1.

Good point about not collecting identifying data, it should just be possible to ask for whatever information was used to decide who to leaflet, such as first letter of last name, which should avoid this issue.

Comment author: Bernadette_Young 06 February 2017 09:45:29PM 1 point [-]

Ah sorry, I missed that bit!

Comment author: Bernadette_Young 06 February 2017 08:34:49PM 0 points [-]

Ethics approval would probably depend on not collecting identifying data like name, so it would be important to build that into your design. College name would work, but pseudo-randomising by leafleting some colleges would introduce significant confounding, because colleges frequently differ in their make up and culture.

Comment author: georgie_mallett 18 November 2016 05:21:33PM 1 point [-]

Ah sorry Bernadette I misunderstood your first question!

I think 'pin down an explanation' was probably too strong on my part, because I definitely don't think it'd be conclusive and I do hope that we have some more qualitative research into this.

We do have professionals working on the survey this year (is that what you meant by professional involvement?) and I've sent your comment to them. They're far better placed to analyze this than me!

Comment author: Bernadette_Young 20 November 2016 01:08:16PM 0 points [-]

Thanks Georgie - I see where we were misunderstanding each other! That's great - research like this is quite hard to get right, and I think it's an excellent plan to have people with experience and knowledge about the design and execution as well as analysis involved. (My background is medical research as well as clinical medicine, and a depressing amount of research - including randomised clinical trials - is never able to answer the important question because of fundamental design choices. Unfortunately knowing this fact isn't enough to avoid the pitfalls. It's great that EA is interested in data, but it's vital we generate and analyse good data well.)

Comment author: georgie_mallett 16 November 2016 07:35:19PM 0 points [-]

Me too! We're in the process of creating the survey now and will be distributing it in January. This is one thing we're going to address, and if you have suggestions about specific questions, we'd be interested in hearing them.

Comment author: Bernadette_Young 17 November 2016 10:00:56AM 1 point [-]

Unless you have a specific hypothesis that you are testing, I think the survey is the wrong methodology to answer this question. If you actually want to explore the reasons why (and expect there will not be a single answer) then you need qualitative research.

If you do pursue questions on this topic in a survey format, it is likely you will get misleading answers unless you have the resources to very rigorously test and refine your question methodology. Since you will essentially be asking people if they are not doing something they have said is good to do, there will be all sorts of biases as play, and it will be very difficult to write questions that function the way you expect them to. To the best of my knowledge question testing didn't happen at all with the first survey, I don't know if any happened with the second.

I appreciate the survey uses a vast amount of people's resources, and is done for good reasons. I hate sounding like a doom-monger, but there are pitfalls here and significant limitations on surveys as a research method. I think the EA community risks falling into a trap on this topic, thinking dubious data is better than none, when actually false data can literally costs lives. As previously, I would strongly suggest getting professional involvement.

Comment author: Bernadette_Young 16 November 2016 04:42:16PM *  0 points [-]

The median EA donation ($330) was pretty low. There could be various reasons for this, but we can only really pin down an explanation when .impact conduct the next EA Survey. I

According to the reports, the first survey of 2014 (ie reported in 2015) found a median donation of $450 in 2013, with 766 people reporting their donations.

The next survey of 2015 (ie reported 2106) found a mediant donation of $330 in 2014, with 1341 people reporting their donations.

Repeating the survey has gathered more data and actually produced a lower estimate. I'm interested how the third survey will help understand this better?

Comment author: Gleb_T  (EA Profile) 25 October 2016 02:44:29AM -3 points [-]

Note – I will make separate responses as my original comment was too long for the system to handle. This is part one of my comments.

Some of you will be tempted to just downvote this comment because I wrote it. I want you to think about whether that’s the best thing to do for the sake of transparency. If this post gets significant downvotes and is invisible, I’ll be happy to post it as a separate EA Forum post. If that’s what you want, please go ahead and downvote.

I’m very proud of and happy with the work that Intentional Insights does to promote rational thinking, wise decision-making, and effective giving to a broad audience. To be clear, we focus on spreading rational thinking in all areas of life, not only charitable giving, with the goal of raising the sanity waterline and ameliorating x-risk. We place articles in major venues, appear on radio and television, and spread our content through a wide variety of other channels. It is not an exaggeration to say we have reached millions of people through our work. Now, we don’t have a large resource base. We have a miniscule budget of just over 40K, mostly provided by my wife and I. It’s thanks to our broad network of volunteers of over 50 people that we can make this difference. A few of these volunteers also provide some contract work, and I’m really happy they can do so. Thanks to all the folks who helped make this happen!

Let’s go on to the content of the post. I appreciate the constructive part of the criticism of the authors of this post, and think some of points are quite correct.

1) I do think we made some mistakes with our social media, especially on Facebook, and we are working to address that.

2) We have instituted a Conflict of Interest policy to provide clear guidance to anyone in an official position with InIn to disclose their affiliations when making public statements about the organization.

3) Unfortunately, the person I asked to update our social media impact after Jacy Reese thoughtfully pointed out the “shares” vs. “likes” issue forgot to update the EA Impact document, although she did update the others. Thanks for bringing it to our attention, and it’s now fixed.

4) While I was careful to avoid explicitly soliciting upvotes, my actions were intended to bring information about opportunities to upvote to supporters of Intentional Insights. I should have been clear about that, and I noted that later in the FB post.

5) I am at heart a trusting person. I trusted the figures from TLYCS, and why shouldn’t I? They are the experts on their figures. I’m glad that this situation led to a revision of the figures, as I want to know the actual impact that we are making, and not have a false and inflated belief about our impact.

In part two, I will describe what aspects of the post I disagreed with.

P. S. Based on past experience, I learned that back and forth online about this will not be productive, so I did not plan to engage with, and if someone wants to learn more about my perspective, they are welcome to contact me privately by my email.

Comment author: Bernadette_Young 25 October 2016 03:13:50PM 17 points [-]

I have down-voted this comment because I think as a community we should strongly disapprove of this sort of threat

"If this post gets significant downvotes and is invisible, I’ll be happy to post it as a separate EA Forum post. If that’s what you want, please go ahead and downvote."

The criticisms have been raised in an exceptionally transparent manner: Jeff made a public post on Facebook, and Gleb was tagged in to participate. Within that thread the plans to make this document were explained and even linked to: anybody (Gleb included) could read and contribute to that document while it was under construction.

This statement - that all criticism in the form of down-voting is likely to be driven by personal animosity or an attempt to hide negative feedback - is both baseless and appears to be an attempt to ward off all criticism. While I feel that Gleb is currently in a very difficult position, this framing of the conversation makes engagement impossible, hence downvoting.

View more: Next