I'll regularly be cross-posting things from the Oxford Prioritisation Project blog to the EA forum. This serves partly as a commitment mechanism for us, whereby we would be embarrassed if we stopped posting. More importantly, we value the EA community's feedback on what we are doing. If you point out mistakes in our work or make well-considered suggestions, you'd stand a good chance of influencing our final decision.
Here's a recent post by Konstantin Sietzy, Jacob Lagerros, and myself.
Read and comment on the Google Document version of this post here.
By Tom Sittler, Konstantin Sietzy, Jacob Lagerros
Should the Oxford Prioritisation Project focus on donation opportunities that are ‘the right size’? Is it important to find a £10,000 funding gap for a specific purchase, (or by way of analogy, £10,000-shaped lego bricks)? This question can quickly become very confusing. Over a series of conversations, we have managed to move towards greater conceptual clarity around this issue, and we attempt to lay out our thinking here.
Our conclusion is that lego bricks are unlikely to be relevant to the Oxford Prioritisation Project, for two main reasons:
1) What appear to be genuine lego brick opportunities often turn out to only shift benefits forward in time, and these shifts in time are continuous rather than discrete.
2) In order to identify lego bricks, one needs very detailed information about organisations’ funding situations.
As we move along with the Oxford Prioritisation Project, we continue to encounter questions of a more strategic nature, intersecting our object-level investigation at various points. We sum this up as “meta-prioritisation”, or perhaps more simply, prioritisation strategy. Questions of meta-prioritisation may include considerations such as: how to resolve value disagreements resulting from pluralist moral attitudes among the team; deciding between building expertise in a focus area and keeping a bird’s eye view; or aiming at generating maximum direct impact vs. generating information useful to future donors.
We explore our first such question in this blog post: can, and should, the Oxford Prioritisation Project team target ‘lego bricks’ - giving opportunities where we could effect a step change rather than a mere continuous increase in the utility of the target organisation with our donation of £10,000?
There has been much discussion of this topic, owing to the fact that different team members had very different intuitions about it. Some thought there surely must be lego bricks, and the world would be a very strange place if there weren’t, while others have the exact opposite intuition.
We can contrast two approaches to finding donation opportunities. Under a ‘marginal’ approach, we would seek to find the organisation with the highest expected cost-effectiveness at the margin. Barring diminishing returns (unlikely at the scale of £10,000), we would behave in roughly the same way as if we were looking to donate £5,000 or £20,000. Under a ‘lego brick’ approach, instead, we would seek an opportunity tailored to the exact size of the donation. We would be looking for a highly specific £10,000 funding gap, like a hole in a lego brick tower that our donation could click into.
The lego brick approach has some intuitive pull. We want to solve problems with our donation, and it seems reasonable in general to choose a problem that’s the right size for one’s resources. We also have the intuition that donating >£100,000 and donating £10,000 should look very different and require the answer to different questions, meaning that the market for our funding opportunities is sufficiently different from than that explored by larger prioritization organizations such as GiveWell and the Open Philanthropy Project.
However, we believe this is to a large extent an illusion, and that searching for lego bricks would be a mistake for the Oxford Prioritisation Project. After much discussion, there remains some disagreement as to whether lego bricks are merely irrelevant for our project, because they are the wrong size, and are too hard to find (Konstantin, Jacob), or whether they do not meaningfully exist empirically (Tom).
This topic has been the source of much confusion among the three of us, and on the Oxford Prioritisation Project team. Many similar, but in fact subtly different ideas play into whether there are lego bricks, and it can be hard to get one’s concepts clear and avoid talking past each other. We hope this post can help us and others think more clearly in the future.
The lego brick approach assumes that at least some giving opportunities are noncontinuous. In other words, filling a funding gap with a £10,000 lego brick has not twice as much impact as giving this organisation £5,000, but perhaps five or ten times as much. Similarly, donating £20,000 has less than twice as much impact as £10,000. A lego brick therefore amounts to a step change in utility functions.
Figure 1: A normal step function
Looking at Figure 1, some moves along the horizontal axis (donations y) have no impact at all on the value u created by the organisation, while some much smaller moves create a step-increase in value.
The intuitive rationale for step-functions is that the purchases driving an organisation’s impact are discrete. The Against Malaria Foundation may only conduct an additional bednet distribution if it acquires an extra $3 million, and do nothing if it gets $2.99 million. An organisation may be unable to hire a new staff member if their funds fall just short of a threshold.
Does this mean it’s crucial to find such steps in the utility function and fill them? If the step-function view of the world is correct, then filling a correctly-sized gap matters vastly more than the effectiveness of the organisation one is choosing. Even if we found the most effective organisation in the world, but donated on a flat part of its step function, we would produce zero impact.
Keep in mind that ‘filling’ in our use means to ‘push the utility function over the edge’. It mustn't necessarily amount to filling the entire horizontal length of a step as indicated by the diagrams; rather, what matters is the final, value-generating step.
This whole post assumes that no-one else will fill the various funding gaps. If they were going to be filled anyway, our impact would come entirely from shifting someone else’s donation, regardless of whether or not there are lego bricks. We think that the various issues around donor coordination, are orthogonal to the issue we discuss here.
The Against Malaria Foundation can trace each bednet to a distance of 6 meters, because each one has a GPS tracker. Imagine a counterfactual world in which AMF had just distributed bed nets every year, without using trackers. Perhaps at some point they would have noticed that installing trackers on bed nets could increase their effectiveness, due to higher retention rates, etc. – but the one-off costs of developing a hardware and software solution for equipping nets with trackers would have been to high. So they keep donating bed nets every year, and while their overall positive impact continues to increase, an opportunity for a step change in utility is missed. (This is a fully hypothetical example, and we are sure it can be picked apart on its narrative details; what matters, though, is whether it illustrates the point).
A counter-argument is that, in this case, AMF should have just stopped spending money on bed nets the year they realised this opportunity, saved up money, and eventually restarted distribution with trackers installed, perhaps two years later. This would have ended up multiplying the value of each bed net considerably such that the positive impact of future net distributions would outweigh the disvalue of the two-year delay in net distribution.
In practice, organisations are unlikely to reduce their costs so drastically. Organisations generally have a high share of fixed costs in their total expenditure, which are difficult to substantially reduce temporarily. These fixed costs include office rental, staff salaries, etc.
We need a more complex model. Suppose a charity has yearly revenue R, and fixed costs F, such that its discretionary income is R-F=D.
We are interested in the size of D relative to potential lego bricks. For a lego brick opportunity of size L, a charity would need to delay its purchase by L/D years.
This conclusion significantly weakens lego brick arguments. For if we donated a proportion pL of L, we would reduce the delay by a proportion p. For instance, if an organisation only had D=£2,000/year, and could make a high-value purchase worth £10,000, it would take them 5 years to save the money if they received no donation, 4 years if they received a £2,000 donation, 2 years if they received a £6,000 donation, etc. In other words, a stepwise function u(y) turns into a smoother function, of ‘delay in utility’ as a function of y. (We call this the delay argument)
Figure 2: the delay argument.
Jacob pointed out that the above assumes that opportunities are not time-limited, so an organisation can always delay purchases. Hence we see an opportunity that is actually strongly time-limited, on the scale of months or a year, as a case of a genuine lego brick. (But we think these are very difficult to find for the Oxford Prioritisation Project. See below, ‘Time-dependent opportunities’).
However, all opportunities are time-limited to some extent. If L/D=5, it seems unreasonable to expect organisations will save for 5 years, given the risk that the opportunity may disappear in the meantime.
If D was very small, there would be many lego brick opportunities, even once we consider the delay argument.
The size of D is an empirical question. There is some disagreement between Tom and Konstantin about the size of D we should expect to encounter.
One argument for thinking D is unlikely to be small (say, smaller than £50,000) is that organisations want to hedge against the possibility of losing funding. Say an organisation had R=£1,000,000. They would be unlikely to set F=£990,000 because they would encounter major problems if they lost only one £10,000 donor. One rough guess is that we would expect D to be at least 10% of R.
However, Konstantin argued that F might be larger than we think, and thus D smaller. This is because organisations face a variety of ‘soft constraints’ (e.g. irrational inertia, misaligned incentives, ignorant donors).
For example, if organisations scale back some of their work in order to save, they may lose important funders or partners on the ground. To take an extreme example, F cannot be 0 for the Against Malaria foundation: if they simply stopped distributing bednets for multiple years they would lose the trust of their partners.
F might be higher than we think because of various principal-agent problems. Even charitable organisations are unlikely to find that their staff will happily accept salary freezes to sponsor a future beneficial but non-crucial utility increase. Scaling back some programmes may be perceived as having failed as an organisation or as a leader, so managers might be reluctant to do so to capture future benefits.
As savings build up, organisations might become tempted to use them to increase F, for example by hiring more staff, rather than to keep saving for a future purchase. Konstantin thinks this is especially true in cases where the purchase would be nonessential to the continued functioning of the organisations - such as in the GPS tracker example (AMF would still be distributing highly cost-effective bednets if they had no trackers). There may be internal disagreement about priorities and the necessity of investment D within the organisation or between the organisation and the donor community. Pausing discretionary spending in order to make a future purchase or hire is likely to be unpopular internally or externally . We would expect this inertia-effect to scale with L/D, i.e. as the opportunity becomes more costly relative to existing income and moves further into the future (see below). In this case, a lego-brick oriented donor might achieve significant impact if they act as an external shock. This belief is consistent with initial psychological research that suggests people follow mental accounting procedures to be more willing to spend unexpected windfalls than objectively identical amounts out of pre-planned budgets.
The extent to which these constraints exist, and whether they are a major source of lego bricks, remains an unresolved empirical disagreement between Konstantin and Tom.
Even if one were convinced that lego bricks did exist in a meaningful sense, finding lego bricks requires being knowledgeable in detail about an organization's’ funding situation, and the opportunities they are facing. But if we cannot know this, we are in effect facing many different step functions. Averaging out across them brings us back to a continuous function.
Figure 3: Under imperfect information, from the donor perspective step functions average out to being continuous
As described above, we see a strongly time-limited opportunity, on the scale of months or a year, as a case of a genuine lego brick. (“we could hire this amazing employee this month who will otherwise do a PhD and take up a teaching career”). However, given the time constraints of the Oxford Prioritisation Project, and the limitations of our networks and detailed knowledge of particular organisations, we think we’d be very unlikely to find a time-limited opportunity that would not otherwise have been filled, before the deadline for our decision.
If there were lego bricks to be found, what size would they be? A reasonable heuristic could be the cost to hire an employee for a year, let’s say £50,000. We struggle to think of examples of £10,000 lego bricks.
Furthermore, Konstantin gave another reason to expect lego bricks to be in the £100,000s of pounds rather than the low £10,000s. Soft constraints stemming from ‘irrational’ organisational behaviour are more likely to prove prohibitive where the amounts concerned are larger; both internal and external stakeholders are more likely to accept reducing discretionary spending for a cause they don’t consider necessary if this is a small amount relative to the organisation’s overall revenue than a large one. For example, AMF may plausibly convince team members and donors to save on stationery and distribute fewer bednets to the value of £9,000 for a year, but may not be able to argue the same with £500,000.
Overall, we agree that the Oxford Prioritisation Project should not attempt to find lego bricks.