6

rohinmshah comments on Thoughts on the "Meta Trap" - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (47)

You are viewing a single comment's thread. Show more comments above.

Comment author: Ben_Todd 21 December 2016 08:57:37PM 0 points [-]

Consider instead the case where a general member of a local group comes to a workshop and takes the GWWC pledge on the spot (which I think happens not infrequently?). The local group has done the job of finding the member and introducing her to EA, maybe raising the probability to 30%. 80K would count the full impact of that pledge, and the local group would probably also count a decent portion of that impact.

I can't speak for the other orgs, but 80k probably wouldn't count this as "full impact".

First, the person would have to say they made the pledge "due to 80k". Whereas if they were heavily influenced by the local group, they might say they would have taken it otherwise.

Second, as a first approximation, we use the same figure GWWC does for a value of a pledge in terms of donations. IIRC this already assumes only 30% is additional, once counterfactually adjusted. This % is based on their surveys of the pledgers. (Moreover, for the largest donors, who determine 75% of the donations, we ask them to make individual estimates too).

Taken together, 80k would attribute at most 30% of the value.

Third, you can still get the undercounting issue I mentioned. If someone later takes the pledge due to the local group, but was influenced by 80k, 80k probably wouldn't count it.

I don't know how 80k considers the impact of their career workshops, but I would bet money that they don't take into account the costs to the local group that hosts the workshop.

What would you estimate is the opportunity cost of student group organiser time per hour?

How would it compare to time spent by 80k staff?

Comment author: rohinmshah  (EA Profile) 21 December 2016 10:18:30PM 0 points [-]

First, the person would have to say they made the pledge "due to 80k".

Yes, I'm predicting that they would say that almost always (over 90% of the time).

this already assumes only 30% is additional, once counterfactually adjusted.

That does make quite a difference. It seems plausible then that impact is mostly undercounted rather than overcounted. This seems more like an artifact of a weird calculation (why use GWWC's counterfactual instead of having a separate one)? And you still have the issue that impact may be double counted, it's just that since you tend to undercount impact in the first place the effects seem to cancel out.

That's a little uncharitable of me, but the point I'm trying to make is that there is no correction for double-counting impact -- most of your counterarguments seem to be saying "we typically underestimate our impact so this doesn't end up being a problem". You aren't using the 30% counterfactual rate because you're worried about double counting impact with GWWC. (I'm correct about that, right? It would a really strange way to handle double counting of impact.)

Nitpick: This spreadsheet suggests 53%, and then adds some more impact based on changing where people donate (which could double count with GiveWell).

Third, you can still get the undercounting issue I mentioned. If someone later takes the pledge due to the local group, but was influenced by 80k, 80k probably wouldn't count it.

I agree that impact is often undercounted. I accept that impact is often undercounted, to such a degree that double counting would not get you over 100%. I still worry that people think "Their impact numbers are great and probably significant underestimates" without thinking about the issue of double counting, especially since most orgs make sure to mention how their impact estimates are likely underestimates.

Even if people just donated on the basis of "their impact numbers are great" without thinking about both undercounting and overcounting, I would worry that they are making the right decision for the wrong reasons. We should promote more rigorous thinking.

My perspective is something like "donors should know about these considerations", whereas you may be interpreting it as "people who work in meta don't know/care about these considerations". I would only endorse the latter in the one specific case of not valuing the time of other groups/people.

What would you estimate is the opportunity cost of student group organiser time per hour?

The number I use for myself is $20, mostly just made up so that I can use it in Fermi estimates.

How would it compare to time spent by 80k staff?

Unsure. Probably a little bit higher, but not much. Say $40?

(I have not thought much about the actual numbers. I do think that the ratio between the two should be relatively small.)

I also don't care too much that 80k doesn't include costs to student groups because those costs are relatively small compared to the costs to 80k (probably). This is why I haven't really looked into it. This is not the case with GWWC pledges or chapter seeding.

Comment author: Robert_Wiblin 23 December 2016 04:36:22AM 2 points [-]

Hey Rohin, without getting into the details, I'm pretty unsure whether correcting for impacts from multiple orgs makes 80,000 Hours look better or worse, so I'm not sure how we should act. We win out in some cases (we get bragging rights from someone who found out about EA from another source then changes their career) and lose in others (someone who finds out about GiveWell through 80k but doesn't then attribute their donations to us).

There's double counting yes, but the orgs are also legitimately complementary of one another - not sure if the double counting exceeds the real complementarity.

We could try to measure the benefit/cost of the movement as a whole - this gets rid of the attribution and complementarity problem, though loses the ability to tell what is best within the movement.

Comment author: rohinmshah  (EA Profile) 23 December 2016 07:00:09PM *  0 points [-]

I'm pretty unsure whether correcting for impacts from multiple orgs makes 80,000 Hours look better or worse

I'm a little unclear on what you mean here. I see three different factors:

  1. Various orgs are undercounting their impact because they don't count small changes that are part of a larger effort, even though in theory from a single player perspective, they should count the impact.

  2. In some cases, two (or more) organizations both reach out to an individual, but either one of the organizations would have been sufficient, so neither of them get any counterfactual impact (more generally, the sum of the individually recorded impacts is less than the impact of the system as a whole)

  3. Multiple orgs have claimed the same object-level impact (eg. an additional $100,000 to AMF from a GWWC pledge) because they were all counterfactually responsible for it (more generally, the sum of the individually recorded impacts is more than the impact of the system as a whole).

Let's suppose:

X is the impact of an org from a single player perspective

Y is the impact of an org taking a system-level view (so that the sum of Y values for all orgs is equal to the impact of the system as a whole)

Point 1 doesn't change X or Y, but it does change the estimate we make of X and Y, and tends to increase it.

Point 2 can only tend to make Y > X.

Point 3 can only tend to make Y < X.

Is your claim that the combination of points 1 and 2 may outweigh point 3, or just that point 2 may outweigh point 3? I can believe the former, but the latter seems unlikely -- it doesn't seem very common for many separate orgs to all be capable of making the same change, it seems more likely to me that in such cases all of the orgs are necessary which would be an instance of point 3.

We could try to measure the benefit/cost of the movement as a whole

Yeah, this is the best idea I've come up with so far, but I don't really like it much. (Do you include local groups? Do you include the time that EAs spend talking to their friends? If not, how do you determine how much of the impact to attribute to meta orgs vs. normal network effects?) It would be a good start though.

Another possibility is to cross-reference data between all meta orgs, and try to figure out whether for each person, the sum of the impacts recorded by all meta orgs is a reasonable number. Not sure how feasible this actually is (in particular, it's hard to know what a "reasonable number" would be, and coordinating among so many organizations seems quite hard).

Comment author: Ben_Todd 22 December 2016 09:14:54PM 1 point [-]

I agree the double-counting issue is pretty complex. (I think maybe the "fraction of value added" approach I mention in the value of coordination post is along the right lines)

I think the key point is that it seems unlikely that (given how orgs currently measure impact) they're claiming significantly more than 100% in aggregate. This is partly because there's already lots of adjustments that pick up some of this (e.g. asking people if they would have done X due to another org) and because there are various types of undercounting.

Given this, adding a further correction for double counting doesn't seem like a particularly big consideration - there are more pressing sources of uncertainty.

Comment author: rohinmshah  (EA Profile) 23 December 2016 07:02:00PM 0 points [-]

Yes, I agree with this. (See also my reply to Rob above.)