Comment author: david_reinstein 20 May 2017 06:29:50PM 3 points [-]

I like this article and I agree with the argument in principle, but I'd like to see a bit more information presented about how the elasticity parameter is estimated.

In other words, what data has been used to compute this parameter? Experiments where people make choices among different lotteries? Implicit choices where people make tradeoffs involving risk? Stated preferences over comparisons of societal distributions of wealth?

Comment author: Toby_Ord 22 May 2017 02:46:21PM 4 points [-]

I think it is mainly from individuals' explicit preferences over hypothetical gambles for income streams. e.g. if you are indifferent between a sure salary of $50,000 PA and a 50-50 gamble between a salary of $25,000 or one of $100,000, then that fits logarithmic utility (eta = 1). Note that while people's intuitions about such cases are far from perfect (e.g. they will have status quo bias) this methodology is actually very similar to that of QALYs/DALYs. But I imagine all methods you mention are used. Also other methods such as happiness surveys give results in the same ballpark. If asking about ideal societal distribution, then that is actually a somewhat different question as there could be additional moral reasons in favour of equality or priority to the worst off on top of diminishing marginal utility effects. Eta is typically intended to set aside such issues, though there are other tests to measure those.

Comment author: Toby_Ord 21 January 2016 11:45:26AM 15 points [-]

The terms 'softcore EAs' and 'hardcore EAs' are simply terrible. I strongly urge people to use other words to talk about these groups.

Comment author: Toby_Ord 29 July 2015 07:44:54AM 1 point [-]

Thanks Stefan, this is a very good point.

Comment author: Gregory_Lewis 17 March 2015 04:10:09PM 9 points [-]

Thank you for doing this survey and analysis. I regret that the feedback from me was primarily critical, and that this reply will follow in a similar vein. But I don’t believe the data from this survey is interpretable in most cases, and I think that the main value of this work is as a cautionary example.

A biased analogy

Suppose you wanted to survey the population of Christians at Oxford: maybe you wanted to know their demographics, the mix of denominations, their beliefs on ‘hot button’ bioethical topics, and things like that.

Suppose you did it by going around the local churches and asking the priests to spread the word to their congregants. The local catholic church is very excited, and the priest promises to mention at the end of his sermon; you can’t get through to the Anglican vicar, but the secretary promises she’ll mention it in the next newsletter; the evangelical pastor politely declines.

You get the results, and you find that Christians in Oxford are overwhelmingly catholic, that they are primarily White and Hispanic, and tend conservative on most bioethical issues, and are particularly opposed to abortion and many forms of contraception.

Surveys and Sampling

Of course, you shouldn’t think that, because this sort of survey is shot through with sampling bias. You’d expect Catholics are far more likely to respond to the survey than evangelicals, so instead of getting a balanced picture of the ‘Christians in Oxford’ population, you get a picture of a ‘primarily Catholics in Oxford with some others’ – and predictably the ethnicity data and the bioethical beliefs are skewed.

I hope EA is non-denominational (or failing that, ecumenical), but there is a substructure to the EA population – folks who hang around LessWrong tend to be different from those who hang around Giving What We Can, for example. Further they likely differ in ways the survey is interested in: their gender, their giving, what causes they support, and so on. To survey of ‘The Effective Altruism Movement’, the EAs who cluster in both need to be represented proportionately (ditto all the other subgroups).

The original plan (as I understand) was to obviate the sampling concerns by just sampling the entire population. This was highly over-confident (when has a voluntary survey captured 90%+ of a target population?) and the consequences of its failure to become a de facto ‘EA census’ significant. The blanket advertising of the survey was taken up by some sources more than others: LessWrong put in on their main page, whilst Giving What We Can didn’t email it around – for example. Analogous to the Catholics and the Pentecostals, you would anticipate LWers to be significantly over-sampled versus folks in GWWC (or, indeed, versus many other groups, as I’d guess LW’s ‘reach’ to its membership via its main page is much better than many other groups). Consequently results like the proportion of EAs who care about AI/x-risk, where most EAs live, or what got them involved in EA you would predict to be slanted towards what LWers care about, where LWers live (bay area), or how LWers got involved in EA (LW!).

If the subgroups didn’t differ, we could breathe a sigh of relief. Alas, not so: the subgroups identified by URL significantly differ across a variety of demographic information, and their absolute size (often 10-20%) makes the difference practically as well as statistically significant – I’d guess if you compared ‘where you heard about EA’ against URL, you’d see an even bigger difference. It may understate the case – if one moved from 3 groups (LW, EA FB, contacts) to 2 (LW, non-LW), one may see more differences, and the missing variable issues and smaller subgroup size mean the point estimates for (e.g.) what proportion of LWers care about X-risk is not that reliable.

Convenience sampling is always dicey, as unlike probabilistic sampling any error in parameter estimate due to bias will not expectedly diminish as you increase the sample size. However, the sampling strategy in this case is particularly undesirable as the likely bias runs pretty much parallel to the things you are interested in: you might hope that (for example) the population of the EA facebook might not be too slanted in terms of cause selection compared to the ‘real’ EA population – not a group like GWWC, LW, CFAR, etc.

What makes it particularly problematic is that it is very hard estimate the ‘size’ of this bias: I wouldn’t be surprised if this survey only oversampled LWers by 5-10%, but I wouldn’t be that surprised if it oversampled LWers by a factor of 3 either. The problem is that any ‘surprise’ I get from the survey mostly goes to adjusting my expectation of how biased it is. Suppose I think ‘EA’ is 50% male and I expect the survey to overestimate the %age male by 15%. Suppose the survey said EA was 90% male. I am going to be much more uncertain about the degree of over-representation than I am about what I think the ‘true EA male fraction’ is. So the update will be to something like 52% male and the survey overestimating by 28%. To the extent I am not an ideal epistemic agent, feeding me difficult to interpret data might make my estimates worse, not better.

To find fault is easy; to plan well, difficult

Science rewards caution and planning; many problems found in analysis could only have been fixed in design, and post-hoc cleaning of data is seldom feasible and still seldomer easy. Further planning could have made the results more interpretable. Survey design has a variety of jargon like “population definition”, “sampling frame”. More careful discussion of what the target population was and how they were going to be reached could have flagged the sampling bias worry sooner, likewise how likely a ‘saturation’ strategy was to succeed. As it was most of the discussion seemed to be focused on grabbing as many people as possible.

Similarly, ‘baking in’ the intended analysis plan with the survey itself would have helped to make sure the data could be analysed in the manner intended (my understanding – correct me if I’m wrong! – is that the planning of exactly what analysis would be done happened after the survey was in the wild). In view of the sampling worries, the analysis was planned to avoid giving aggregate measures sensitive to sampling bias, but instead explore relationships between groups via regression (e.g. what factors predict amount given to charity). However, my understanding is this pre-registered plan had to be abandoned as the data was not amenable. Losing the pre-registered plan for a new one which shares no common elements is regrettable (especially as the new results are very vulnerable to sampling bias), and a bit of a red flag.

On getting better data, and on using data better

Given the above, I think the survey offers extremely unreliable data. I’m not sure I agree with the authors it is ‘better than nothing’, or better than our intuitions - given most of us are imperfect cognizers, it might lead us more astray to the ‘true nature’ of the EA community. I am pretty confident it is not worth the collective time and energy it has taken: it probably took a couple of hundred hours of the EA community’s time to fill in the surveys, leave alone the significant work from the team in terms of design, analysis, etc.

Although some things could not have been helped, I think many things could have, and there were better approaches ex ante:

1) It is always hard to calibrate one’s lack of knowledge about something. But googling things like ‘survey design’, ‘sampling’, and similar are fruitful – if nothing else, they suggest that ‘doing a survey’ is not always straightforward and easy, and put one on guard for hidden pitfalls. This sort of screening should be particularly encouraged if one isn’t a domain expert: many things in medicine concord with common sense, but some things do not, likewise statistics and analysis, and no doubt likewise many other matters I know even less about.

2) Clever and sensible the EA community generally is, it may not always be sufficient to ask for feedback on a survey idea and then interpreting the lack of response as a tacit green light. Sometimes ‘We need expertise and will not start until we have engaged some’, although more cautious, is also more better. I’d anticipate this concern will grow in significance as EAs tackle things ‘further afield’ from their background and training.

3) You did get a relative domain expert raise the sampling concerns to you within a few hours of going live. Laudable though it was that you were responsive to this criticism and (for example) tracked URL data to get a better handle on sampling concerns, invited your critics to review prior drafts and analysis, and mention the methodological concerns prominently, it took a little too long to get there. There also seemed a fair about of over-confidence and defensiveness – not only from some members of the survey team, but from others who thought that, although they hadn’t considered X before and didn’t know a huge amount about X, that on the basis of summary reflection X wasn’t such a big deal. Calling a pause very early may have been feasible, and may have salvaged the survey from the problems above.

This all comes across as disheartening. I was disheartened too: effective altruism intends to put a strong emphasis on being quantitative, getting robust data, and so forth. Yet when we try to practice what we preach, our efforts leave much to be desired (this survey is not the only – or the worst – example). In the same way good outcomes are not guaranteed by good intentions, good information is not guaranteed by good will and hard work. In some ways we are trailblazers in looking hard at the first problem, but for the second we have the benefit of the bitter experience of the scientists and statisticians who have gone before us. Let us avoid recapitulating their mistakes.

Comment author: Toby_Ord 17 March 2015 08:30:56PM 3 points [-]

Thanks for sharing such detailed thoughts on this Greg. It is so useful to have people with significant domain expertise in the community who take the time to carefully explain their concerns.

Comment author: Toby_Ord 20 November 2014 11:12:36AM *  4 points [-]

You might be interested in:

http://en.wikipedia.org/wiki/Health_Impact_Fund

http://en.wikipedia.org/wiki/Social_impact_bond

Which are practical prize type solutions along similar lines.

Comment author: RyanCarey 16 November 2014 01:40:36PM 8 points [-]

On the face of it, it seems likely that removing one of your kidneys will decrease your life expectancy somewhat. For patients admitted to hospital, a substantial fraction, something like a quarter, have problems with mild or moderate impaired kidney function. When you donate your kidney, your remaining kidney compensates, but only to the point that it gives you 70% of your original kidney function. So on priors, we should favour the hypothesis that kidney donation increases your change of getting kidney failure.

Then, we take the evidence into account. The evidence, some of which you point out, has identified time after time that kidney donors have comparable risk of renal failure as the general population, despite being screened for diabetes, severe hypertension and kidney failure. This suggests again that donating a kidney increases your risk of kidney failure.

Then, let's look at more studies, mentioned in your footnote 25: Mjoen finds a large mortality risk from kidney donation. Kaplan effectively criticises Mjoen in relation to mortality rates, but concedes that there is likely to be a real effect on renal failure. Kaplan goes on to speculate that this risk of renal failure might arise from hereditary factors (from being related to the donor), rather than from donation itself. Muzaale, which you cite in relation to the claim that 'kidney donation has not been shown to measurably decrease long-term life expectancy.', shows that risk of renal failure is elevated, even for unrelated donors. By comparing related donors, unrelated donors and nondonors, one can see that the decrease in kidney failure is mostly correlated with donation, rather than heredity.

You cite Ibrahim in relation to the same point. To quote the article: "An increased incidence rate of ESRD in donors compared with non-donor controls is now also corroborated in a recently presented abstract on almost 100,000 living kidney donors from the United States.6 In that study, the incidence rate of ESRD was eightfold higher in donors (comparable to the 11-fold increase in the incidence rate in this Norwegian study). Thus, there are now at least two studies describing an approximately tenfold increase in the incidence of ESRD after donation, which is a serious concern."

All of this evidence strongly supports the common sense view that giving up a kidney will mildly increase the risk of end-stage kidney failure.

End-stage renal failure can often cut short a person's lifespan by many years. Beyond being bad from a selfish point of view, this could also decrease your physical capacity to help others. Kidney failure mostly afflicts people in their post-retirement years. However, this is not always the case, nor is it always the case that people cease useful altruistic activities when they retire. The question from an altruistic point of view is how big this opportunity cost has to be to outweigh the benefits of kidney donation.

It depends how many other ways there are of saving lives. By donating one's kidneys, one saves about 15 years of life. How much would it cost to save 15 years of life through a GiveWell donation? Maybe $2k. So if you care about individuals equally regardless of the country that they're from, then this is about how much one should be willing to pay to save a life. Is it reliable to apply GiveWell's estimates here? Maybe not. Are there other opportunities that will have even better impact? Plausibly. So let's use $2k as a target, ableit an approximate one.

How long would it take to create $2k of value? That's generally 1-2 weeks of work. So if kidney donation makes you lose more than 1-2 weeks of life, and those weeks constitute funds that you would donate, or voluntary contributions that you would make, then it's a net negative activity for an effective altruist.

Kidney donation makes you face a 1/4000 chance of death in the operation. If you've 40 years to live, that's 3 days of expected lifespan lost already. When you are admitted to hospital, you will run a substantial risk of having a recovery that is longer than your funded leave from work. So that's another expected week or so gone. Then, if you run a 1% chance of end-stage renal failure, that might rob you a few more weeks of time, although that time isn't taken from the prime of your life.

All things considered, it seems like if you care about people equally, irrespective of the medical condition they're suffering from and the country they're born in, and if you're prepared to donate your time or earnings, then you will actually be causing net harm by performing a random kidney donation.

Comment author: Toby_Ord 19 November 2014 03:18:08PM 5 points [-]

I'm inclined to agree with Ryan's argument here. One way I look at it is that I wouldn't donate a kidney in order to get $2,000 (whether that was to be spent on myself or donated to effective charities), or equivalently, that I am prepared to pay $2,000 to keep my second kidney. This means that, for me at least, donating is dominated by extra donations.

I am surprised that this comes out as close as it does though. If we didn't have quite so effective charities, kidney donation would be a great option.

Comment author: Toby_Ord 19 November 2014 02:48:22PM 3 points [-]

Thanks Tom, this looks great. I'd broaden it out to include Giving What We Can's recommended charities.

Comment author: Toby_Ord 31 October 2014 03:53:06PM 8 points [-]

Thanks for writing this up! I should note that Giving What We Can also has a good way of doing this: you can leave the money to the Giving What We Can Trust. By default this will be allocated between the charities Giving What We Can recommends, but you can also specify other charities (in any country) that are global poverty related. Indeed, one can also make one's annual donations via the Giving What We Can Trust, allowing UK taxpayers to get Gift Aid on the entire donation, even if part or all of it goes to a charity abroad. I do this and I also find that it simplifies the giving process (I don't have to look up the donation pages for the six or so charities that I split my donations between). Since I've pretty much explained everything about the trust, I'll also mention that money donated into it legally cannot be used to support Giving What We Can itself.

Comment author: Gregory_Lewis 23 October 2014 09:49:36PM 25 points [-]

I'm strongly against changing the pledge, as well as the underlying motivation to more tightly incorporate GWWC with (cause-neutral) EA

It seems part of the GWWC 'brand equity' is that basically all morally serious people agree that giving large amounts to help those in desperate poverty is a good thing to do (although some EAs would add that there might be something else that can do much better). All other causes EAs commonly endorse (i.e. animal welfare, x-risk) have much poorer common sense credentials.

GWWC is a valuable intersection between Effective Altruism and the wider world. In one direction, GWWC packages some EA ideals (moral commitment, looking at evidence and data carefully, some moves towards cause agnosticism) in a manner that not-too-inferentially distant from most people, and a common route for people getting 'more EA'. Looking the other way, the wider appeal of global poverty over the currently-coalescing 'Tenets of Effective Altruism' can attract those not 'fully on board', and this group of more liminal 'EA' people can be an anchor against the EA community drifting further towards becoming an insular, epistemically overconfident and morally arrogant monoculture.

Insofar as the changes to the pledge (and the ethos motivating them) pushes GWWC towards being an avowedly 'EA organisation'; insofar as their membership have less to do with commonsensically good things like 'stopping extreme poverty'; and insofar as it risks pushing away those inclined to ally with the 'EA movement', these changes threaten the goods GWWC brings to the EA movement. I also fear knock-on effects to GWWC itself and the millions of pounds it encourages and directs to good causes (is there a plan to survey the current members of GWWC about this change in the pledge?)

I agree that the 'EA community' should have a cause-neutral way to make a morally serious declaration of commitment, and especially to convert what I hope to be wildly successful outreach efforts into concrete action to make the world better. One could do worse than modelling this community on GWWC, given how successful it has been.

But I don't think GWWC can be this community whilst maintaining its focus on global poverty. I don't see how subtle changes to the wording of the pledge to make it in principle 'animal rights friendly' help when in practice GWWC's entire website is about global poverty and never mentions animal suffering once. Nor do I see how cause-agnosticism is squared with a tagline that exorts, "Join us in the fight against global poverty". I worry about Giving What We Can's mission being incrementally hollowed out to make it a more welcoming home to EAs who think global poverty is a distraction from a far more important cause; or it being half-heartedly maintained as an atavistic facade by a cognoscenti who take it as received wisdom that fighting global poverty isn't really the most important thing GWWC does anymore.

The other alternatives; of trying to construct a cause-neutral parallel to GWWC, preferably before the books launch, or a status quo devoid of a central hub new EAs can go to; are far from ideal either. Yet the former is not hopeless (c.f. Tom et al.'s work on a donation registry), and a community norm about giving could propagate without a centralized group (and perhaps we should fear a single EA group becoming too central to the movement). I'd prefer either to this.

Comment author: Toby_Ord 24 October 2014 10:41:16AM 8 points [-]

Thanks Gregory, that's a very helpful set of arguments.

Comment author: Dale 22 October 2014 11:54:45PM 5 points [-]

It's worth noting that GWWC already includes many MIRI donors. Certainly much of the CEA management thinks that Xrisk/animals/other speculative causes is more important than global poverty, and MIRI is even listed (in a very hard to see spot) on the GWWC front page (scroll down to the 'donated' pie chart, then hit the down arrow to scroll through the recipient charities). GWWC has accepted Xrisk charities as fulfilling the pledge at least since 2011, and perhaps longer.

However, I agree with you that this would damage GWWC's brand equity. It is already a hard enough sell at the moment. Once upon a time, GWWC was concerned about being publicly associated with 80k, let alone XRisk.

Comment author: Toby_Ord 23 October 2014 11:54:28AM 7 points [-]

I don't think it is accurate to say that it includes 'many' MIRI donors. At least not compared to its total of 644 members. Note that MIRI was listed as the 42nd out of 43 listed charities in order of how much members have donated to them, which seems about as marginal as it could be. In addition, the list of charities that our members have donated to is not supposed to be any kind of endorsement of them by Giving What We Can. We allow members to donate their pledged amounts anywhere so long as it is a sincere interpretation of the pledge.

View more: Next