Thank you for the response, and I'm glad that it's being improved, and that there seems to be a honest interest in doing better.
I feel "ensure others don't get the wrong idea about how seriously such estimates should be taken" is understating things- it should be reasonable for people to ascribe some non-zero level of meaning to issued estimates, and especially it should be that using them to compare between charities doesn't lead you massively astray. If it's "the wrong idea" to look at an estimate at all, because it isn't the true best reasoned expectation of results the evaluator has, I think the error was in the estimate rather than in expectation management, and find the deflection of responsibility here to the people who took ACE at all seriously concerning.
The solution here shouldn't be for people to trust things others say less in general.
Compare, say, GiveWell's analysis of LLINs (http://www.givewell.org/international/technical/programs/insecticide-treated-nets#HowcosteffectiveisLLINdistribution); it's very rough and the numbers shouldn't be assumed to be close to right (and responsibly, they describe all this), but their methodology makes them viable for comparison purposes.
Cost-effectiveness is important- it is the measure of where putting your money does the most good and how much good you can expect to do, and a fully inclusive of risks and data issues cost effectiveness estimate is basically what one is arriving at when one determines what is effective. Even if you use other selection strategies for top charities, incorrect cost effectiveness estimates are not good.
I agree: it is indeed reasonable for people to have read our estimates the way they did. But when I said that we don't want others to "get the wrong idea", I'm not claiming that the readers were at fault. I'm claiming that the ACE communications staff was at fault.
Internally, the ACE research team was fairly clear about what we thought about leafleting in 2014. But the communications staff (and, in particular, I) failed to adequately get across these concerns at the time.
Later, in 2015 and 2016, I feel that whenever an issue like leafleting came up publicly, ACE was good about clearly expressing our reservations. But we neglected to update the older 2014 page with the same kind of language that we now use when talking about these things. We are now doing what we can to remedy this, first by including a disclaimer at the top of the older leafleting pages, and second by planning a full update of the leafleting intervention page in the near future.
Per your concern about cost-effectiveness estimates, I do want to say that our research team will be making such calculations public on our Guesstimate page as time permits. But for the time being, we had to take down our internal impact calculator because the way that we used it internally did not match the ways others (like Slate Star Codex) were using it. We were trying to err on the side of openness by keeping it public for as long as we did, but in retrospect there just wasn't a good way for others to use the tool in the way we used it internally. Thankfully, the Guesstimate platform includes upper and lower bounds directly in the presented data, so we feel it will be much more appropriate for us to share with the public.
You said "I think the error was in the estimate rather than in expectation management" because you felt the estimate itself wasn't good; but I hope this makes it more clear that we feel that the way we were internally using upper and lower bounds was good; it's just that the way we were talking about these calculations was not.
Internally, when we look at and compare animal charities, we continue to use cost effectiveness estimates as detailed on our evaluation criteria page. We intend to publicly display these kinds of calculations on Guesstimate in the future.
As you've said, the lesson should not be for people to trust things others say less in general. I completely agree with this sentiment. Instead, when it comes to us, the lessons we're taking are: (1) communications staff needs to better explain our current stance on existing pages, (2) comm staff should better understand that readers may draw conclusions solely from older pages, without reading our more current thinking on more recently published pages, and (3) research staff should be more discriminating on what types of internal tools are appropriate for public use. There may also be further lessons that can be learned from this as ACE staff continues to discuss these issues internally. But, for now, this is what we're currently thinking.
Hi everyone! I’m here to formally respond to Sarah’s article, on behalf of ACE. It’s difficult to determine where the response should go, as it seems there are many discussions, and reposting appears to be discouraged. I’ve decided to post here on the EA forum (as it tends to be the central meeting place for EAs), and will try to direct people from other places to this longer response.
Firstly, I’d like to clarify why we have not inserted ourselves into the discussion happening in multiple Facebook groups and fora. We have recently implemented a formal social media policy which encourages ACE staff to respond to comments about our work with great consideration, and in a way that accurately reflects our views (as opposed to those of one staff member). We are aware that this might come across as “radio silence” or lack of concern for the criticism at hand—but that is not the case. Whenever there are legitimate critiques about our work, we take it very seriously. When there are accusations of intent to deceive, we do not take them lightly. The last thing we want to do is respond in haste only to realize that we had not given the criticism enough consideration. We also want to allow the community to discuss amongst themselves prior to posting a response. This is not only to encourage discussion amongst individual members of the community, but also so that we can prioritize responding to the concerns shared by the greatest number of community members.
It is clear to us now that we have failed to adequately communicate the uncertainty surrounding the outcomes of our leafleting intervention report. We absolutely disagree with claims of intentional deception and the characterization of our staff as acting in bad-faith—we have never tried to hide our uncertainty about the existing leafleting research report, and as others have pointed out, it is clearly stated throughout the site where leafleting is mentioned. However, our reasoning that these disclaimers would be obvious was based on the assumption that those interested in the report would read it in its entirety. After reading the responses to this article, it’s obvious that we have not made these disclaimers as apparent as they should be. We have added a longer disclaimer to the top of our leafleting report page, expressing our current thoughts and noting that we will update the report sometime in 2017.
In addition, we have decided to remove the impact calculator (a tool which included an ability to enter donations directed to leafleting and receive estimates of high and low bounds of animals spared) from our website entirely until we feel more confident that it is not misleading to those unfamiliar with cost effectiveness calculations and/or an understanding of how the low/best/high error bounds exemplify the uncertainty regarding those numbers. It is not typical for us to remove content from the site, but we intend to operate with abundant caution. This change seems to be the best option, given that people believe we are being intentionally deceptive in keeping them online.
Finally, leadership at ACE all agree it has been too long since we have updated our Mistakes page, so we have added new entries concerning issues we have reflected upon as an organization.
We also notice that there is concern among the community that our recommendations are suspect due to the weak evidence supporting our cost-effectiveness estimates of leafleting. The focus on leafleting for this criticism is confusing to us, as our cost-effectiveness estimates address many interventions, not only leafleting, and the evidence for leafleting is not much weaker than other evidence available about animal advocacy interventions. On top of that, cost-effectiveness estimates are only a factor in one of the seven criteria used in our evaluation process. In most cases, we don’t think that they have changed the outcome of our evaluation decisions. While we haven’t come up with a solution for clarifying this point, we always welcome and are appreciative of constructive feedback.
We are committed to honesty, and are disappointed that the content we've published on the website concerning leafleting has caused so much confusion as to lead anyone to believe we are intentionally deceiving our supporters for profit. On a personal note, I’m devastated to hear that our error in communication has led to the character assassination not only of ACE, but of the people who comprise the organization—some of the hardest working, well-intentioned people I’ve ever worked with.
Finally, I would like everyone to know that we sincerely appreciate the constructive feedback we receive from people within and beyond the EA movement.
*Edited to add links
Well said, Erika. I'm happy with most of these changes, though I'm sad that we have had to remove the impact calculator in order to ensure others don't get the wrong idea about how seriously such estimates should be taken. Thankfully, Allison plans on implementing a replacement for it at some point using the Guesstimate platform.
For those interested in seeing the exact changes ACE has made to the site, see the disclaimer at the top of the leafleting intervention page and the updates to our mistakes page.
Animal Charity Evaluators' post like this, for 2016, is here.
Cool. I'm in with $2k.
I'd like to contribute $1k. Would you like to coordinate together so we can meet the $5k threshold?
Edit: After further consideration, I decided to instead donate $500 to the donor lottery while increasing my direct donations elsewhere.
Me too! We're in the process of creating the survey now and will be distributing it in January. This is one thing we're going to address, and if you have suggestions about specific questions, we'd be interested in hearing them.
Please include a question about race. At the Effective Animal Advocacy Symposium this past weekend at Princeton, the 2015 EA Survey was specifically called out for neglecting to ask a question about the race of the respondents.
There're lots of great stuff you guys are doing, but I'd like to comment on one thing in particular: your t-shirts. They look awesome.
I know some EAs think they are low value, but, as an introvert, having a great EA t-shirt helps to initiate conversations with acquaintances when they ask about it. Plus, I imagine it would help build camaraderie between members of any local EA group.
Very cool. (c:
At the Effective Animal Advocacy Symposium, Garrett Broad pointed out in his talk that the 2015 Survey of Effective Altruists did not ask about race, which is worrying given how overwhelmingly white the movement is. To my knowledge this makes at least two public critiques of the movement on this specific topic.
He points out that the best way to deal with race issues is not to ignore the issue, but to bring it front and center. Could we please be sure to include a question about race on the 2016 version of this survey?
EDIT: Here's an image. I'll upload a video of his talk once ACE puts the videos of the conference online.
EDIT: The video is here. It's titled "Advocacy for Education" and Garret Broad's section of the talk begins at 33:20.
A few of us have experience working in politics and could conceivably accomplish some good by being an influencer in Trump's White House. Others of us have the ability to pitch Thiel on stuff. Since Thiel has sway in the Trump transition, this means we could conceivably get an EA or two into positions of influence in the Trump administration.
I'm not sure that it would be a good idea to actually do this, but I'm mentioning it because it doesn't seem outside the realm of possibility to actually do it, and it could plausibly be highly effective. Here are some of the questions we'd need to answer:
I don't know the answer to these questions. I don't know if this is even a workable idea. I certainly would hate to convince an EA to drop their current work for this if it doesn't turn out to be an influential position. But it seems possible that this could be a high-value opportunity, so I'm bringing it to everyone here.
For anyone working on pages for EA organizations, keep in mind that (1) you probably shouldn't be an employee of that organization and (2) considerable attention should be included in a criticism section. The ACE page was removed in part because the article was "too positive", and people like me were prohibited from adding substantive critical content to it because of my affiliation with the organization (per their conflict of interest policy).
I would not recommend relisting ACE in particular without a criticism section that cites criticism from several different sources. See the AfD page for ACE and compare to the AfD for 80k Hours for more details. (Note that the 80k Hours article survived deletion by being much less positive.)
In general, I like the idea of having something to wear that promotes discussion. It's especially useful for introverts like myself who have trouble bringing up EA in social contexts, but have no problem responding to questions about my shirt and using that as an introduction to EA.
The shirt given out at EA Global was good, as it just has the term "Effective Altruism" which tends to prompt questions. The shirt GiveWell sells is also excellent, as they are a very well named organization. But "Doing Good Effectively by Using Reason" seems clunky to me. I believe it is too long and it feels more pompous than something shorter like "GiveWell".
Contrary to what you've written, I actually think something like "Optimizing QALYs" might actually be good. It's short, easy to read, doesn't sound pompous, and will definitely prompt a question of what it means in a social situation. That is the kind of shirt that I'd actually wear and find useful.
Other shirts I'd find useful would be for various well-named organizations, like "Animal Charity Evaluators", "Giving What We Can", "Charity Science", etc. These don't even need slogans; they can use their name/logo alone, and I think I'd find the shirt useful.
© 2017 Effective Altruism Forum |
Powered by reddit