Comment author: Halstead 11 July 2018 01:42:28PM *  9 points [-]

Thanks for getting the conversation going on this topic, which hasn't received enough systematic attention by EAs. An excellent treatment of this issue is given by Paul Brest here - https://ssir.org/articles/entry/impact_investing . This suggests that the prospects of achieving market rate returns and having social impact are dim. One may be able to have counterfactual impact by accepting below market returns or at the extreme providing a grant to a company. (Open Phil has invested in Impossible Foods, presumably accepting below market returns).

One observation I have is that there is a big step between showing that impact investing might work in some conditions and actually finding good opportunities. It seems like identifying good opportunities would take serious up a lot of serious research time - of the same order we would expect to identify a recommended GW charity. A glance through some impact investing platforms suggests they offer quite shallow analysis of enterprises that look unlikely to be effective. So, I think we should acknowledge that this space is worth exploring but be very sceptical about any particular opportunity, whether that be Wave, World Tree or whatever

Comment author: kbog  (EA Profile) 11 July 2018 12:32:16PM *  5 points [-]

If capital markets are efficient and most people aren't impact investors, then there is no benefit to impact investing, as the coal company can get capital from someone else for the market rate as soon as you back out, and the solar company will lose most of its investors unless it offers a competitive rate of return. At the same time, there is no cost to impact investing.

In reality I think things are not always like this, but not only does inefficiency imply that impact investing has an impact, it also implies that you will get a lower financial return.

For most of us, our cause priorities are not directly addressed by publicly traded companies, so I think impact investing falls below the utility/returns frontier set by donations and investments. You can pick a combination of greedy investments and straight donations that is Pareto superior to an impact investment. If renewable energy for instance is one of your top cause priorities, then perhaps it is a different story.

Comment author: Halstead 11 July 2018 01:28:23PM *  2 points [-]

If the solar company is offering below market returns, then an impact investment is equivalent to a grant to that company. This opens up some space for impact investing to have counterfactual impact, provided the investment opportunity stands a decent chance of success.

Comment author: Ben_Todd 03 June 2018 10:24:44AM 2 points [-]

I'm interested in what norms we can use to better deal with the practical case.

e.g. Suppose:

1) GiveWell does research for a cost of $6 2) TLYCS does outreach using the research for a cost of $6 3) $10 is raised as a result.

Assume that if GiveWell didn't do the research, TLYCS wouldn't have raised the $10, and vice versa.

If you're a donor working out where to give, how should you approach the situation?

If you consider funding TLYCS with GiveWell held fixed, then you can spend $6 to raise $10, which is worth doing. But if you consider funding GiveWell+TLYCS together, then you can spend $12 to raise $10, which is not worth doing.

It seems like the solution is that the donor needs to think very carefully about which margin they're operating at. Here are a couple of options:

A) If GiveWell will definitely do the research whatever happens, then you ought to give. B) Maybe GiveWell won't do the research if they don't think anyone will promote it, so the two orgs are coupled, and that means you shouldn't fund either. (Funding TLYCS causes GiveWell to raise more, which is bad in this case) C) If you're a large donor who is able to cover both funding gaps, then you should consider the value of funding the sum, rather than each org individually.

It seems true that donors don't often consider situations like (B), which might be a mistake. Though sometimes they do - e.g. GiveWell considers the costs of malaria net distribution incurred by other actors.

Likewise, it seems like donors often don't consider situations like (C). e.g. If there are enough interactions, maybe the EA Funds should calculate the cost-effectiveness of a portfolio of EA orgs, rather than estimate the ratios for each individual org.

On the other hand, I don't think these cases where two orgs are both 100% necessary for 100% of the impact are actually that common. In practice, if GiveWell didn't exist, TLYCS would do something else with the $6, which would mean they raise somewhat less than $10; and vice versa. So, the two impacts are fairly unlikely to add up to much more than $12.

Comment author: Halstead 11 June 2018 07:33:36PM *  1 point [-]

In case B, it looks to me like the donor should give to TLYCS, in certain conditions, in others not.

(a) Suppose: Because you gave to TLYCS, GiveWell does the research at a cost of $6, fundraising from an otherwise ineffective donor, and getting $10 to GW charities. In this case, your $6 has raised $10 for effective charities minus the $6 from an otherwise ineffective donor (~0 value). So, I don't think causing GW to fundraise further would be bad in this case. Coordinating with GW to just get them to fundraise for donations to their effective charities is even better in this case, but donating to TLYCS is better than doing nothing.

(b) Suppose: Same as before, except GW fundraises from an effective donor who would otherwise have given the $6 to GW charities. In this case, giving to TLYCS is worse than doing nothing because you have spent $6 getting $10 to GW charities, minus what the effective donor would have given to had you not acted (-$6 to GW charities), so you've spent $6 getting $4 to effective charities. Doing nothing would be better, as then $6 goes to effective charities.

This shows that the counterfactual impact of funged/leveraged donations needs to be considered carefully. GiveWell is starting to do this - e.g. if govt money is leveraged or funged they try to estimate the cost-effectiveness of govt money. Outside that, this is probably something EA donors should take more account of.

Another case that should be considered is causing irrational prioritisation with a given amount of funds. Imagine case (a) above except that instead of fundraising, GiveWell moves money from another research project with a counterfactual value of $9 to GW charities because they have not considering these coordination effects (they reason that $10>$9). In that case, you're spending $6 to get $10 to GW charities minus the $9 that would have gone to GW charities.

Regarding C, this seems right. It would be a mistake for the EA funds to add up its impact as the sum of the impact of each of the individual grants it has made.

Comment author: Flodorner 11 June 2018 07:49:32AM *  0 points [-]

The claim does not seem to be exactly, that there is a 10% chance of an animal advocacy video affecting consumption decisions after 12 years for a given individual.

I'd interpret it as: there is a 5% chance of the mean duration of reduction, conditioned on the participant reporting to change their behaviour based on the video being higher than 12 years.

This could for example also be achieved by having a very long term impact on very few participants. This interpretation seems a lot more plausible, although i am not certain at all, wheter that claim correct. Long term follow up data would certainly be very helpful.

Comment author: Halstead 11 June 2018 08:01:08AM 1 point [-]

Yes I was speaking somewhat loosely. It is nevertheless in my view very implausible that the intervention would sustain its effect for that long - we're talking about the effect of one video here. Do you think the chance of fade-out within a year is less than 10%? What is your median estimate?

Comment author: Halstead 09 June 2018 12:26:05PM *  4 points [-]

The assumptions here about the persistence of the effect seem over-optimistic.

You measure the effect after one month and then assume that it will persist for 1 to 12 years (90% CI). So, you assign a less than 10% chance that the effect will fade out within a year. You made this decision "arbitrarily" on the basis of an ACE meta-analysis investigating how long people who say they don't eat meat have not eaten meat without interruption. The first to say is that this is testing a very different population and so is of questionable relevance to the Animal Equality intervention. In the ACE study, the sample is people who say they have made the commitment to be vegetarian. In yours, it is people who have been shown a video who say they haven't eaten pork a month on.

Given that we are working with fairly arbitrary intuitions here, I find it highly surprising that the 90% CI doesn't include fade out of the effect within a year. My median estimate is that the effect fades out within a year. I'd be curious to hear what other people think about this.

But you think there is around a 10% chance that the effect will fade out after 12 years. The claim is that there is a 10% chance that being shown an animal advocacy video on one day will have an effect on consumption decisions 12 years down the line. I would put the chance of this at ~0%.

If I am right and a more reasonable estimate of persistence seems to be closer to 6 months (I actually think I'm being conservative here - I'd guess closer to 2-3 months), this suggests you should revise your cost-effectiveness estimate down by an order of magnitude.

Comment author: Halstead 06 June 2018 01:26:25PM 0 points [-]

V interesting. I'd be curious to hear the arguments as to why we should persist with expensive RCTs if heterogeneity is as described.

Comment author: Ben_Todd 05 June 2018 06:03:55PM 1 point [-]

Makes sense. I don't think Joey would object if orgs were counting this though.

Comment author: Halstead 06 June 2018 10:54:00AM 0 points [-]

I don't agree. His logic entails that money/effort you leverage shouldn't be counted as your own counterfactual impact. If FHI convinces e.g. the UK government that biorisk is worth spending money on, then on Joey's approach, FHI would be wrong to count this additional money as it's own impact.

Comment author: Ben_Todd 05 June 2018 05:00:49AM -1 points [-]

I agree - I was talking a bit too loosely. When I said "assign credit of 30% of X" I meant "assign counterfactual impact of 30% of X". My point was just that even if you do add up all the counterfactual impacts (ignoring that this is a conceptual mistake like you point out), they rarely sum to more than 100%, so it's still not a big issue.

I'm not sure I follow the first paragraph about leveraging other groups.

Comment author: Halstead 05 June 2018 10:43:07AM 0 points [-]

You argued that counterfactual impact may be smaller than it appears. But it may also be larger than it first appears due to leveraging other orgs away from ineffective activities. e.g. an NGO successfully advocates for a policy change P1 - the benefits of P1 is their counterfactual impact. But as a result of the proven success of this type of project, 100 other NGOs start working on similar projects where before they worked on ineffective projects. This latter effect should also be counted as the first org's counterfactual impact. This could be understood as leveraging additional money into an effective space.

Comment author: Ben_Todd 03 June 2018 10:41:58AM 3 points [-]

On the practical point, one help is that I think cases like these are fairly uncommon:

The previous example used donations because it’s easy and clear cut to make the case that this is the wrong move without getting into more difficult issues, but it generalizes to talent as well. For example, recently, Fortify Health was founded. Clearly the founders deserve 100% impact- without them, the project certainly would not have happened. But wait a second: both of them think that without Charity Science’s support, the project would definitely not have happened. So, technically, Charity Science could also take 100% credit. (Since from our perspective, if we did not help Fortify Health it would not have happened, so it is a 100% counterfactually caused by Charity Science project). But wait a second, what about the donors who funded the project early on (because of Charity Science’s recommendation)? Surely they deserve some credit for impact as well! What about the fact that without the EA movement, it would have been much less likely for Charity Science and Fortify Health to connect? With multiple organizations and individuals, you can very easily attribute a lot more impact than actually happens.

In our impact evaluations, and in my experiences talking to others in the community, we would never give 100% of the impact to each group. For instance, if Charity Science didn't exist, the founders of Fortify might well have ended up doing a similar idea anyway - it's not as if Charity Science is the only group promoting evidence-based global health charities, and if Charity Science didn't exist, another group like them probably would have sprung up eventually. What's more, even if the founders didn't do Fortify, they would probably have done something else high-impact instead. So, the impact of Charity Science should probably be much less than 100% of Fortify. And the same is true for the other groups involved.

At 80,000 Hours, we rarely claim more than 30% of the impact of an event or plan change, and we most often model our impact as a speed-up (e.g. we assume the career changer would have eventually made the same shift, but we made it come 0.5-4 years earlier). We also sometimes factor in costs incurred by other groups. All this makes it hard for credit to add up to more than 100% in practice.

Comment author: Halstead 04 June 2018 02:06:04PM *  1 point [-]

good points. This can also go the other way though - an org could leverage money from otherwise very ineffective orgs. Especially with policy changes, it can sometimes be the case that a good org comes up with a campaign that steers the entire advocacy ecosystem to a more effective path. A good example of this is campaigns for ordinary air pollution regulations on coal plants, which were started in the 1990s by the Clean Air Task Force among others and now have hundreds of millions in funding from Bloomberg. If these campaigns weren't started, environmental NGOs in the US and Europe would plausibly be working on something much worse.

I don't think the notion of 'credit' is a useful one. At FP, when we were looking at orgs working on policy change, we initially asked them how much credit they should take for a particular policy change. They ended up saying things like "40%". I don't really understand what this means. It turned out to be best to ask them when the campaign and policy change would have happened had they not acted (obviously a very difficult question). It's best to couch things in terms of counterfactual impact throughout and not to convert into 'credit'.

Similarly with voting, if an election is decided by one vote and there are one million voters for the winning party, I think it is inevitably misleading to ask how much of the credit each voter should get. One naturally answers that they get one millionth of the credit, but this is wrong as a proposition about their counterfactual impact, which is what we really care about.

Indeed, focusing on credit can lead you to attribute impact in cases of redundant causation when an org actually has zero counterfactual impact. Imagine 100 orgs are working for a big policy change, and only 50 of them were necessary to the outcome (though this could be any combination of them and they were all equally important). In this case, funding one of the orgs had zero counterfactual impact because the change would have happened without them. But on the 'credit approach', you'd end up attributing one hundredth of the impact to each of the orgs

Comment author: rohinmshah  (EA Profile) 01 June 2018 09:32:13PM 1 point [-]

Forget about the organization's own counterfactual impact for a moment.

Do you agree that, from the world's perspective, it would be better in Joey's scenario if GWWC, Charity Science, and TLYCS were to all donate their money directly to AMF?

Comment author: Halstead 04 June 2018 01:54:28PM 1 point [-]

yes

View more: Prev | Next