5

Effective Altruism Prediction Registry

Predictions matter because they separate the signal from the noise. Anyone can make a vague proclamation that sometime in the future something good or bad will happen; it’s the people who put definitive dates and measurable claims on these statements that make it possible to evaluate whether they were actually right or wrong, and figure out what we should do next.

SuperForecasting makes a compelling case that tracking predictions is useful on both the individual and collective level. When we track a prediction we’re able to get feedback on whether we were right or wrong, and overtime we get better at making predictions. And it helps groups make better decisions by forcing disciplined thinking and avoiding unclear and confusing opinions and HiPPO.

 

Whenever you make a decision you’re making a prediction. These can be small predictions, like if I decide to leave at 7:30 I’ll make it to work on time, or big predictions, like if I attend a prestigious university I will be able to make a lot of money when I graduate. For Effective Altruists these decisions are often related to charitable giving, not just about what cause is the ‘best’ to give to, but also what is the marginal improvement you expect your donation to cause. I’m increasingly convinced that we need to do a better job as a community systematically tracking predictions around important topics.


If I give $3,000 to the Against Malaria Foundation, I’m predicting I will be able to save one life. Or, if I think that EAs should be making appeals based on equality or justice, instead of individual rights or outcomes, I’m predicting that this strategy, if implemented, will drive more donations than the other.


So we’re making predictions all the time, but the predictions sit in isolation from one another, in random posts in the various EA facebook groups and forums. And they lack the rigor of providing actual deadlines and numbers, for when/how we should assess the claims. This is a problem, because we’re missing out on the wealth of knowledge that would come from learning if/how these ideas actually worked.


I think this is low hanging fruit for us to improve EA. If we tracked predictions about the outcomes of campaigns, interventions, etc. we’d see a number of benefits.

  • Establish a strong track record of success for top performing charities

  • Provide guidance for decision makers. Predictions, from orgs and individuals, about EA campaigns, could help guide donations.

  • Elevate good forecasters, and boost effective ideas.


In particular EA orgs could make more explicit predictions about what the outcome of giving money would be. It was difficult to track down concrete predictions from notable EA Orgs for the Winter 2015 giving season, but two I found were from 80000 Hours, predicting 50 plan changes by 10/31/2016, and CFAR, with 1000 new alumni by 12/31/2016.

Some type of central repository of predictions being made about EA, where people can comment, provide their own predictions, and update their predictions as new evidence comes in. This could be as simple as a blog post or a more full fledged system/market. Prediction trackers have a long history, but with little successful adoption. I think that EA’s might be different because of the unique origins of the movement. Given the analytical culture of EAs, this could be uniquely well suited to our community.

I see three steps to doing this:

  • A simple forum where anyone can submit a declarative prediction about EA related events.

    • By December 31st, 2016 there will be 5,000 people signed up for the Giving What We Can pledge

  • Anyone can submit their predictions tied to this event

    • JohnSmith89 thinks there is a 20% probability this is true

  • After the deadline has past judges/people can vote on whether it actually happened, and you can see who successfully predicted it and with what accuracy.


With 75% confidence I’d say that by February 10th at least 15 people will have expressed interest in predictions about effective altruism.


Comments (23)

Comment author: tyleralterman 30 January 2016 05:23:20AM 4 points [-]

Very much support the thrust of this post. Oliver Habryka on the EA Outreach team is currently chatting with the Good Judgment Project team about implementing a prediction market in EA.

Comment author: Jess_Riedel 12 July 2016 11:19:46PM 0 points [-]

Update: the Good Judgment Project has just launched Good Judgement Open. https://www.gjopen.com/

Comment author: RyanCarey 29 January 2016 07:10:01AM *  4 points [-]

To get a good prediction market, we need more participation than the EA community would provide at its current size.

This is going to be a problem where the superset - creating a prediction market accessible by everyone - is easier to solve than the specific case - making a prediction market for the EA community.

Prediction Book is already taking on this role - a Prediction market by EAs - and would welcome help.

edit: fixed spelling

Comment author: Sebastian_Farquhar 02 February 2016 11:04:58PM 0 points [-]

Prediction markets benefit a lot from liquidity. Making it EA specific doesn't seem to gain all that much. But EAs should definitely practice forecasting formally and getting rewarded for reliable predictions.

Comment author: MichaelDickens  (EA Profile) 29 January 2016 04:11:07PM 0 points [-]

This comment has a lot of spelling mistakes and it's hard to understand. (I'm guessing you wrote it on mobile.) Can you go over it and remove the spelling mistakes?

Comment author: Peter_Hurford  (EA Profile) 29 January 2016 05:20:27PM 2 points [-]

My best translation...

To get a good prediction market, we need more participation than the EA community would provide at its current size.

This is going to be a problem where the superset - A prediction market accessible by every one - is easier to solve than the specific case - prediction market for the EA community.

Prediction <something> is already taking on this role - a prediction market by EAs - and would welcome help.

Comment author: MichaelDickens  (EA Profile) 29 January 2016 08:51:52PM 1 point [-]

I believe "Predictipn bool" is supposed to be "PredictionBook".

Comment author: RyanCarey 30 January 2016 02:00:42AM *  1 point [-]

Not sure why that happened. Fixed.

Comment author: Owen_Cotton-Barratt 30 January 2016 12:00:22PM 7 points [-]

It gave me a mental image of a drunk Ryan, slurring his words while making a coherent argument.

Comment author: atucker 31 January 2016 07:43:41PM 3 points [-]

Something that surprised me from the Superforecasting book is that just having a registry helps, even when those predictions aren't part of a prediction market.

Maybe a prediction market is overkill right now? I think that registering predictions could be valuable even without the critical mass necessary for the market to have much liquidity. It seems that the advantage of prediction markets is in incentivizing people to try to participate and do well, but if we're just trying to track predictions that EAs are already trying to make then that might be enough.

Also, one of FLI's cofounders (Anthony Aguirre) started a prediction registry: http://www.metaculus.com/ , http://futureoflife.org/2016/01/24/predicting-the-future-of-life/

Comment author: Tom_Ash  (EA Profile) 28 January 2016 10:56:10PM 2 points [-]

What tools for prediction markets are there besides http://predictionbook.com/ ? Any comments on what features they have or which are best for which purposes?

The only other one I know of is https://called.it/ which is mobile only (h/t John Maxwell).

People may be interested in https://www.facebook.com/groups/eapredictons/

Comment author: Richard_Batty 29 January 2016 08:22:59PM 2 points [-]

Augur (http://www.augur.net/) - a decentralised prediction market.

Comment author: ghabs 28 January 2016 11:22:21PM 0 points [-]

I'm not familiar with too many "personal" prediction sites, one's you can register your own predictions (outside of predictionbook).

Zocalo, (http://zocalo.sourceforge.net/) is a toolkit for building prediction markets, but isn't currently supported. https://www.cultivatelabs.com/ creates enterprise prediction markets. And Augur is a cryptocurrency based prediction market that is currently in Alpha, but you can spin up your own test nodes if you wanted to run a separate network..

I can't access the facebook group, is it public? would be interested to check it out!

Comment author: Tom_Ash  (EA Profile) 29 January 2016 01:43:28AM 0 points [-]

I can't access the facebook group, is it public?

You have to join to see posts.

Comment author: Tom_Ash  (EA Profile) 28 January 2016 10:34:48PM 1 point [-]

With 75% confidence I’d say that by February 10th at least 15 people will have expressed interest in predictions about effective altruism.

I hereby express interest. Others can do so in a comment under this!

Comment author: Linch 29 January 2016 03:30:22AM 1 point [-]
Comment author: DanielFilan  (EA Profile) 07 February 2016 09:55:17AM 0 points [-]

Am interested.

Comment author: Castand 07 February 2016 12:28:09AM 0 points [-]

And me

Comment author: jayd 07 February 2016 12:08:47AM 0 points [-]

Me also.

Comment author: LKor 04 February 2016 11:01:10PM 0 points [-]

Also interested, would prefer something not facebook-based. If something needed to be setup/maintained/whatnot, I'd be happy to help.

Comment author: Tom_Ash  (EA Profile) 01 February 2016 01:46:01AM 0 points [-]

What's the count up to now, counting all sources?

Comment author: ghabs 09 February 2016 03:32:49PM 0 points [-]

13 in total, from this post and an EA meetup - so 48 hours to find two more to be "well calibrated" on the prediction.

Comment author: Peter_Hurford  (EA Profile) 29 January 2016 05:18:56PM 0 points [-]

/interest expressed