4

Should the Oxford Prioritisation Project look for lego bricks?

I'll regularly be cross-posting things from the Oxford Prioritisation Project blog to the EA forum. This serves partly as a commitment mechanism for us, whereby we would be embarrassed if we stopped posting. More importantly, we value the EA community's feedback on what we are doing. If you point out mistakes in our work or make well-considered suggestions, you'd stand a good chance of influencing our final decision.

Here's a recent post by Konstantin Sietzy, Jacob Lagerros, and myself.

Read and comment on the Google Document version of this post here.

 

By Tom Sittler, Konstantin Sietzy, Jacob Lagerros

Summary:

Should the Oxford Prioritisation Project focus on donation opportunities that are ‘the right size’? Is it important to find a £10,000 funding gap for a specific purchase, (or by way of analogy, £10,000-shaped lego bricks)? This question can quickly become very confusing. Over a series of conversations, we have managed to move towards greater conceptual clarity around this issue, and we attempt to lay out our thinking here.

Our conclusion is that lego bricks are unlikely to be relevant to the Oxford Prioritisation Project, for two main reasons:

1) What appear to be genuine lego brick opportunities often turn out to only shift benefits forward in time, and these shifts in time are continuous rather than discrete.

2) In order to identify lego bricks, one needs very detailed information about organisations’ funding situations.

Another brick in the wall?

As we move along with the Oxford Prioritisation Project, we continue to encounter questions of a more strategic nature, intersecting our object-level investigation at various points. We sum this up as “meta-prioritisation”, or perhaps more simply, prioritisation strategy. Questions of meta-prioritisation may include considerations such as: how to resolve value disagreements resulting from pluralist moral attitudes among the team; deciding between building expertise in a focus area and keeping a bird’s eye view; or aiming at generating maximum direct impact vs. generating information useful to future donors.

We explore our first such question in this blog post: can, and should, the Oxford Prioritisation Project team target ‘lego bricks’ - giving opportunities where we could effect a step change rather than a mere continuous increase in the utility of the target organisation with our donation of £10,000?

There has been much discussion of this topic, owing to the fact that different team members had very different intuitions about it. Some thought there surely must be lego bricks, and the world would be a very strange place if there weren’t, while others have the exact opposite intuition.

What are lego bricks?

We can contrast two approaches to finding donation opportunities. Under a ‘marginal’ approach, we would seek to find the organisation with the highest expected cost-effectiveness at the margin. Barring diminishing returns (unlikely at the scale of £10,000), we would behave in roughly the same way as if we were looking to donate £5,000 or £20,000. Under a ‘lego brick’ approach, instead, we would seek an opportunity tailored to the exact size of the donation. We would be looking for a highly specific £10,000 funding gap, like a hole in a lego brick tower that our donation could click into.

The lego brick approach has some intuitive pull. We want to solve problems with our donation, and it seems reasonable in general to choose a problem that’s the right size for one’s resources. We also have the intuition that donating >£100,000 and donating £10,000 should look very different and require the answer to different questions, meaning that the market for our funding opportunities is sufficiently different from than that explored by larger prioritization organizations such as GiveWell and the Open Philanthropy Project.

However, we believe this is to a large extent an illusion, and that searching for lego bricks would be a mistake for the Oxford Prioritisation Project. After much discussion, there remains some disagreement as to whether lego bricks are merely irrelevant for our project, because they are the wrong size, and are too hard to find (Konstantin, Jacob), or whether they do not meaningfully exist empirically (Tom).

This topic has been the source of much confusion among the three of us, and on the Oxford Prioritisation Project team. Many similar, but in fact subtly different ideas play into whether there are lego bricks, and it can be hard to get one’s concepts clear and avoid talking past each other. We hope this post can help us and others think more clearly in the future.

The lego brick approach assumes that at least some giving opportunities are noncontinuous. In other words, filling a funding gap with a £10,000 lego brick has not twice as much impact as giving this organisation £5,000, but perhaps five or ten times as much. Similarly, donating £20,000 has less than twice as much impact as £10,000. A lego brick therefore amounts to a step change in utility functions.


Figure 1: A normal step function

Looking at Figure 1, some moves along the horizontal axis (donations y) have no impact at all on the value u created by the organisation, while some much smaller moves create a step-increase in value.

The intuitive rationale for step-functions is that the purchases driving an organisation’s impact are discrete. The Against Malaria Foundation may only conduct an additional bednet distribution if it acquires an extra $3 million, and do nothing if it gets $2.99 million. An organisation may be unable to hire a new staff member if their funds fall just short of a threshold.

Does this mean it’s crucial to find such steps in the utility function and fill them? If the step-function view of the world is correct, then filling a correctly-sized gap matters vastly more than the effectiveness of the organisation one is choosing. Even if we found the most effective organisation in the world, but donated on a flat part of its step function, we would produce zero impact.

Keep in mind that ‘filling’ in our use means to ‘push the utility function over the edge’. It mustn't necessarily amount to filling the entire horizontal length of a step as indicated by the diagrams; rather, what matters is the final, value-generating step.

This whole post assumes that no-one else will fill the various funding gaps. If they were going to be filled anyway, our impact would come entirely from shifting someone else’s donation, regardless of whether or not there are lego bricks. We think that the various issues around donor coordination, are orthogonal to the issue we discuss here.

Do lego bricks exist (even when organisations can save)?

The Against Malaria Foundation can trace each bednet to a distance of 6 meters, because each one has a GPS tracker. Imagine a counterfactual world in which AMF had just distributed bed nets every year, without using trackers. Perhaps at some point they would have noticed that installing trackers on bed nets could increase their effectiveness, due to higher retention rates, etc. – but the one-off costs of developing a hardware and software solution for equipping nets with trackers would have been to high. So they keep donating bed nets every year, and while their overall positive impact continues to increase, an opportunity for a step change in utility is missed. (This is a fully hypothetical example, and we are sure it can be picked apart on its narrative details; what matters, though, is whether it illustrates the point).

A counter-argument is that, in this case, AMF should have just stopped spending money on bed nets the year they realised this opportunity, saved up money, and eventually restarted distribution with trackers installed, perhaps two years later. This would have ended up multiplying the value of each bed net considerably such that the positive impact of future net distributions would outweigh the disvalue of the two-year delay in net distribution.

In practice, organisations are unlikely to reduce their costs so drastically. Organisations generally have a high share of fixed costs in their total expenditure, which are difficult to substantially reduce temporarily. These fixed costs include office rental, staff salaries, etc.

We need a more complex model. Suppose a charity has yearly revenue R, and fixed costs F, such that its discretionary income is R-F=D.

We are interested in the size of D relative to potential lego bricks. For a lego brick opportunity of size L, a charity would need to delay its purchase by L/D years.

This conclusion significantly weakens lego brick arguments. For if we donated a proportion  pL of L, we would reduce the delay by a proportion p. For instance, if an organisation only had D=£2,000/year, and could make a high-value purchase worth £10,000, it would take them 5 years to save the money if they received no donation, 4 years if they received a £2,000 donation, 2 years if they received a £6,000 donation, etc. In other words, a stepwise function u(y) turns into a smoother function, of ‘delay in utility’ as a function of y. (We call this the delay argument)


Figure 2: the delay argument.

Time-limited opportunities

Jacob pointed out that the above assumes that opportunities are not time-limited, so an organisation can always delay purchases. Hence we see an opportunity that is actually strongly time-limited, on the scale of months or a year, as a case of a genuine lego brick. (But we think these are very difficult to find for the Oxford Prioritisation Project. See below, ‘Time-dependent opportunities’).

However, all opportunities are time-limited to some extent. If L/D=5, it seems unreasonable to expect organisations will save for 5 years, given the risk that the opportunity may disappear in the meantime.

If D was very small, there would be many lego brick opportunities, even once we consider the delay argument.

The size of D is an empirical question. There is some disagreement between Tom and Konstantin about the size of D we should expect to encounter.

One argument for thinking D is unlikely to be small (say, smaller than £50,000) is that organisations want to hedge against the possibility of losing funding. Say an organisation had R=£1,000,000. They would be unlikely to set F=£990,000 because they would encounter major problems if they lost only one £10,000 donor. One rough guess is that we would expect D to be at least 10% of R.

However, Konstantin argued that F might be larger than we think, and thus D smaller. This is because organisations face a variety of ‘soft constraints’ (e.g. irrational inertia, misaligned incentives, ignorant donors).

For example, if organisations scale back some of their work in order to save, they may lose important funders or partners on the ground. To take an extreme example, F cannot be 0 for the Against Malaria foundation: if they simply stopped distributing bednets for multiple years they would lose the trust of their partners.

F might be higher than we think because of various principal-agent problems. Even charitable organisations are unlikely to find that their staff will happily accept salary freezes to sponsor a future beneficial but non-crucial utility increase. Scaling back some programmes may be perceived as having failed as an organisation or as a leader, so managers might be reluctant to do so to capture future benefits.

As savings build up, organisations might become tempted to use them to increase F, for example by hiring more staff, rather than to keep saving for a future purchase. Konstantin thinks this is especially true in cases where the purchase would be nonessential to the continued functioning of the organisations - such as in the GPS tracker example (AMF would still be distributing highly cost-effective bednets if they had no trackers). There may be internal disagreement about priorities and the necessity of investment D within the organisation or between the organisation and the donor community. Pausing discretionary spending in order to make a future purchase or hire is likely to be unpopular internally or externally . We would expect this inertia-effect to scale with L/D, i.e. as the opportunity becomes more costly relative to existing income and moves further into the future (see below). In this case, a lego-brick oriented donor might achieve significant impact if they act as an external shock. This belief is consistent with initial psychological research that suggests people follow mental accounting procedures to be more willing to spend unexpected windfalls than objectively identical amounts out of pre-planned budgets.

The extent to which these constraints exist, and whether they are a major source of lego bricks, remains an unresolved empirical disagreement between Konstantin and Tom.

Are lego bricks relevant for the Oxford Prioritisation Project?

Do we have a high enough chance of finding lego bricks?

Knowledge of funding situation

Even if one were convinced that lego bricks did exist in a meaningful sense, finding lego bricks requires being knowledgeable in detail about an organization's’ funding situation, and the opportunities they are facing. But if we cannot know this, we are in effect facing many different step functions. Averaging out across them brings us back to a continuous function.

Figure 3: Under imperfect information, from the donor perspective step functions average out to being continuous

Time-dependent opportunities

As described above, we see a strongly time-limited opportunity, on the scale of months or a year, as a case of a genuine lego brick. (“we could hire this amazing employee this month who will otherwise do a PhD and take up a teaching career”). However, given the time constraints of the Oxford Prioritisation Project, and the limitations of our networks and detailed knowledge of particular organisations, we think we’d be very unlikely to find a time-limited opportunity that would not otherwise have been filled, before the deadline for our decision.

£10,000-sized lego bricks?

If there were lego bricks to be found, what size would they be? A reasonable heuristic could be the cost to hire an employee for a year, let’s say £50,000. We struggle to think of examples of £10,000 lego bricks.

Furthermore, Konstantin gave another reason to expect lego bricks to be in the £100,000s of pounds rather than the low £10,000s. Soft constraints stemming from ‘irrational’ organisational behaviour are more likely to prove prohibitive where the amounts concerned are larger; both internal and external stakeholders are more likely to accept reducing discretionary spending for a cause they don’t consider necessary if this is a small amount relative to the organisation’s overall revenue than a large one. For example, AMF may plausibly convince team members and donors to save on stationery and distribute fewer bednets to the value of £9,000 for a year, but may not be able to argue the same with £500,000.

Conclusion

Overall, we agree that the Oxford Prioritisation Project should not attempt to find lego bricks.

Comments (13)

Comment author: Peter_Hurford  (EA Profile) 14 February 2017 12:01:20AM 4 points [-]

I think others have suggested this, but have you thought about putting your 10K GBP into a donor lottery or otherwise saving up to getting a larger donation size? I'd like to see research address that question (e.g., is a 100K donation >10x better than a 10K donation?).

Comment author: Benito 14 February 2017 03:24:59AM *  2 points [-]

I'll note that the team should still allocate its time to answering the object level question, so that if they win the lottery they know where they'll give the money.

Comment author: Peter_Hurford  (EA Profile) 14 February 2017 04:23:40AM 2 points [-]

I definitely agree. But it might be good to see research around whether a donor lottery is worthwhile and how to donate $100K.

Comment author: BenMillwood  (EA Profile) 25 February 2017 12:31:02PM *  0 points [-]

It's hard to recommend this over playing the lottery upfront, so that you actually know whether your research will be directly used or not.

If you think it's important that the research is done even if the money is not used, would you recommend just doing the research project even with merely a hypothetical £10k that isn't actually donated at the end?

Comment author: ThomasSittler 15 February 2017 09:52:32PM 1 point [-]

There are some pragmatic obstacles to that. For instance, team members have only signed up to work on the Oxford Prioritisation Project over a particular time period. If we ended up winning the donor lottery later on, we'd have to do lots of research then.

Comment author: Peter_Hurford  (EA Profile) 16 February 2017 04:40:45AM 0 points [-]

What's the time period?

One additional reason why I ask is that it seems like a lot of OxPri (do you have a preferred abbreviation? OPP is naughty and conflicts with OpenPhil) research to date has been of the form "field X looks interesting and may have some promising opportunities but it doesn't look like we could do anything for 10K GBP" which makes it feel like the donation size may be limiting.

Comment author: ThomasSittler 16 February 2017 12:30:57PM *  1 point [-]

Hey Peter. Thanks for your interest! We're using oxprio.

Quoting from our homepage: "The Project is funded with a grant from the Centre for Effective Altruism. It will begin in the 1st week of Hilary term 2017 (January 15) and will end at the deadline for reaching a decision, in the 2nd week of Trinity term 2017 (May 6). After we reach our decision, the £10,000 will be disbursed to our chosen recipient. The funding is not restricted to a pre-selected set of charities."

In my view, "doesn't look like we could do anything for £10,000" arguments are in part fallacious (http://effective-altruism.com/ea/17b/oxford_prioritisation_project_another_brick_in/) searches for lego bricks. In some cases we've encountered (like funding research into effective mental health interventions (https://oxpr.io/blog/2017/2/10/lovisa-tengberg-mental-health-an-update) donation size does appear limiting.

Comment author: casebash 14 February 2017 03:50:13AM 0 points [-]

That would defeat the purpose of the project. I think that the purpose is to spur research and the money is there for extra encouragement.

Comment author: Peter_Hurford  (EA Profile) 14 February 2017 04:23:02AM 3 points [-]

I don't think that's true for two reasons:

(1) A 10% chance of donating $100K should be roughly as motivating to a risk-neutral EA as a 100% chance of donating $10K (not taking into account arguments that the risk-neutral utility of money may be nonlinear).

(2) Research around whether to donate $100K or $10K (or how to donate $100K conditional on winning the lottery) would be useful.

Comment author: casebash 16 February 2017 11:24:07PM 1 point [-]

"A 10% chance of donating $100K should be roughly as motivating to a risk-neutral EA as a 100% chance of donating $10K (not taking into account arguments that the risk-neutral utility of money may be nonlinear)." - that's not how human psychology works.

Comment author: Peter_Hurford  (EA Profile) 16 February 2017 11:39:20PM 0 points [-]

How easy is it for an EA to overcome that?

Also, if there's a motivation - impact trade-off, how can we navigate that?

Comment author: BenMillwood  (EA Profile) 25 February 2017 12:23:32PM 1 point [-]

As a piece of presentational feedback, I found it a little frustrating to have a title like this one, and yet not have the term "lego brick" specifically and directly explained until something like the third paragraph :)

Comment author: BenMillwood  (EA Profile) 25 February 2017 12:29:30PM 0 points [-]

Another key reason against looking for "lego bricks" that I don't think you addressed is that marginal thinking is much more generalizable. You're publishing all your research work, and if I come along afterwards with £1k or £100k, a conclusion you made based on marginal thinking is much more likely to be useful to me than one tailored to your exact donation size.

My guess is that the value of your research in how it informs and influences others may even exceed the value of the £10k directly: if that's modestly likely to be true, it seems a strong recommendation to avoid "exact fit" opportunities.

I guess strictly speaking this kind of motivation falls out of scope for your project, which aims simply to find the best way to spend the £10k. But it's certainly a reason I'm glad you made this choice :)