The Important/Neglected/Tractable (INT) framework has become quite popular over the last two years. I believe it was first used by GiveWell for their cause and intervention investigations. Since then Giving What We Can and 80,000 Hours have adopted it with the addition of 'personal fit' as a further criterion, and it has been featured prominently in Will MacAskill's book Doing Good Better.
This qualitative framework is an alternative to straight cost-effectiveness calculations when they are impractical. Briefly the arguments in favour of scoring a cause or charity using a multiple-criteria framework are:
- It is more robust to a major error in one part of your research.
- It forces you to look at something from multiple different angles and seek support from 'many weak arguments' rather than just 'one strong argument'.
- In practice it leads to faster research progress than trying to come up with a persuasive cost-effectiveness estimate when raw information is scarce.
I favour continued use of this framework. But I have also noticed some traps it's easy to fall into when using it.
These criteria are 'heuristics' that are designed to point in the direction of something being cost-effective. Unfortunately, using the common definitions of these words they blur heavily into one another. For example:
- A cause can remain important because it's neglected.
- A cause can remain tractable because it's neglected.
- A cause can be tractable because it's important.
Sometimes I have seen projects evaluated using this framework dramatically overweight one observation, because it can fit into multiple criteria.
Naturally, a cause that's highly neglected will score well on neglectedness. It can then go on to score very well on importance because its neglectedness means the problem remains serious. Finally, it can also score well on tractability, because the most promising approaches are not yet being taken, because it's neglected.
Most of the case in favour of the cause then boils down to it being neglected, rather than just one third, as originally intended.
A way around this problem is to rate each criteria, while 'controlling for' the others.
For example, rather than ask 'how important is this cause?', instead ask 'how important would it be if nobody were doing anything about it?', or 'how important is this cause relative to others that are equally as neglected?'. Rather than ask 'how tractable is this cause on the margin?', instead ask 'how tractable would it be if nobody were working on it?', or 'how tractable is it relative to other causes that are equally neglected?'.
Those questions would be inappropriate if we weren't considering neglectedness separately, but fortunately we are.
If you do make this adjustment, make sure to do it consistently across all of the options you are comparing.
--
Another pitfall is to look at the wrong thing when evaluating 'Importance' (sometimes also called 'Scale'). Imagine you were looking into 'unconditional cash transfers' distributed by GiveDirectly. What should you look at to see if it's solving an 'Important' problem? Two natural options are:
- The total number of people living in extreme poverty who could benefit from cash transfers.
- The severity of the poverty of GiveDirectly's recipients.
Which should you use? It depends what you think you are funding.
When GiveDirectly was first developing 'unconditional cash transfers', carefully studying their impact, and popularising the approach, the most important measure was the first. If their cash transfers proved to be a huge success, then the number of people who could benefit from the innovation was the 1-2 billion people with extremely low incomes.
But today a lot of this research has been done, and many or most people in development are aware of the arguments for cash transfers. Scaling up GiveDirectly further is only going to have modest learning and demonstration effects.
For that reason, I now think the most suitable criteria for evaluating the importance of the problem GiveDirectly is solving, relative to others, is the second: how desperately poor are their recipients relative to other beneficiaries we could help?
To generalise:
- If someone is inventing something new, what you want to know is the size of the total problem that invention could contribute to solving.
- If someone is scaling up something that already exists, what you want to know is the severity of the problem is that it solves per recipient (or, alternatively per dollar that is spent on it).
If you are trying to compare inventing new things against scaling up things that already exists, be prepared to face serious challenges with any framework I know of.
I completely agree that neglectedness is often the route cause of the size and tractability, but that's the whole point. Neglectedness only matters inasmuch as it affects the other two criteria and is included on it's own mostly to aid analysis of tractability – it has no value on it's own. For instance, if a cause is low in importance/scale and tractability, it wouldn't matter what the neglectedness is at all. I think the neglectedness factor only comes into play if a problem is important and tractable (including on the margin), yet crowded. In that case, even though it would be a high priority for an organization to enter because the only two relevant factors (scale and tractability) are high, if it is a very popular cause, it may become less tractable/important in the near future with more entrants into the space.
I think that the type of importance analysis depends on whether the intervention is scalable or systemic, not whether or not it is new. In systemic action, you are tackling the entire issue, so you want to judge the importance of the issue's entirety. Scalable action works on a per unit basis, so the severity of the issue per person/animal/acre, etc is what matters, as you mentioned regarding GiveDirectly.
Lastly, I think that flow through effects should always be included among cause prioritization criteria, because (cheese warning) what matters at the end of the day, is how much the entire world benefits from an intervention, not just how much it benefits the targeted cause. For instance, research intended to progress one subject often ends up providing unforseen benefits in other areas.
An alternative option would be to scrap neglectedness as an independent criterion and consider it in importance and tractability.
I think we agree on the the systemic/scalable distinction - perhaps I just expressed myself poorly.