BenHoffman comments on Four quantitative models, aggregation, and final decision - Oxford Prioritisation Project - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (6)

You are viewing a single comment's thread.

Comment author: BenHoffman 21 May 2017 11:26:35PM *  2 points [-]

Regrettably, we were not able to choose shortlisted organisations as planned. My original intention was that we would choose organisations in a systematic, principled way, shortlisting those which had highest expected impact given our evidence by the time of the shortlist deadline. This proved too difficult, however, so we resorted to choosing the shortlist based on a mixture of our hunches about expected impact and the intellectual value of finding out more about an organisation and comparing it to the others.


Later, we realised that understanding the impact of the Good Food Institute was too difficult, so we replaced it with Animal Charity Evaluators on our shortlist. Animal Charity Evaluators finds advocates for highly effective opportunities to improve the lives of animals.

If quantitative models were used for these decisions I'd be interested in seeing them.

Comment author: kokotajlod 22 May 2017 04:40:54PM 5 points [-]

That second quote in particular seems to be a good example of what some might call measurability bias. Understandable, of course--it's hard to give out a prize on the basis of raw hunches--but nevertheless we should work towards finding ways to avoid it.

Kudos to OPP for being so transparent in their thought process though!

Comment author: ThomasSittler 23 May 2017 10:27:31AM 1 point [-]

Daniel - You are correct that our decision not to donate to GFI was an example of measurability bias. However, I would say the problematic decision was about the shortlist as a whole, rather than replacing GFI with ACE on the shortlist once it was chosen. When we abandoned GFI, we had already chosen the shortlist in part based on what it would be interesting to model. After gaining new information, i.e. discovering ACE could be more sensibly (and thus more interestingly) modelled, switching felt like the right call.

Our shortlisting decision was not as principled as we'd hoped, and I agree it is biased in various ways.

I think our ranking of the shortlisted organisations, while very uncertain, is probably less biased. Producing quantitative models of a biased shortlist of organisations still has some value, especially when they are in different cause areas.