Comment author: tylermjohn 20 April 2018 08:43:29PM 0 points [-]

thanks, gregory. it's valuable to have numbers on this but i have some concerns about this argument and the spirit in which it is made:

1) most arguments for x-risk reduction make the controversial assumption that the future is very positive in expectation. this argument makes the (to my mind even more) controversial assumption that an arbitrary life-year added to a presently-existing person is very positive, on average. while it might be that many relatively wealthy euro-american EAs have life-years that are very positive, on average, it's highly questionable whether the average human has life-years that are on average positive at all, let alone very positive.

2) many global catastrophic risks and extinction risks would affect not only humans but also many other sentient beings. insofar as these x-risks are risks of the extinction of not only humans but also nonhuman animals, to make a determination of the person-affecting value of deterring x-risks we must sum the value of preventing human death with the value of preventing nonhuman death. on the widely held assumption that farmed animals and wild animals have bad lives on average, and given the population of tens of billions of presently existing farmed animals and 10^13-10^22 presently existing wild animals, the value of the extinction of presently living nonhuman beings would likely swamp the (supposedly) negative value of the extinction of presently existing human beings. many of these animals would live a short period of time, sure, but their total life-years still vastly outnumber the remaining life-years of presently existing humans. moreover, most people who accept a largely person-affecting axiology also think that it is bad when we cause people with miserable lives to exist. so on most person-affecting axiologies, we would also need to sum the disvalue of the existence of future farmed and wild animals with the person-affecting value of human extinction. this may make the person-affecting value of preventing extinction extremely negative in expectation.

3) i'm concerned about this result being touted as a finding of a "highly effective" cause. $9,600/life-year is vanishingly small in comparison to many poverty interventions, let alone animal welfare interventions (where ACE estimates that this much money could save 100k+ animals from factory farming). why does $9,600/life-year suddenly make for a highly effective when we're talking about x-risk reduction, when it isn't highly effective when we're talking about other domains?

Comment author: Gregory_Lewis 20 April 2018 09:26:54PM *  2 points [-]

1) Happiness levels seem to trend strongly positive, given things like the world values survey (in the most recent wave - 2014, only Egypt had <50% of people reporting being either 'happy' or 'very happy', although in fairness there were a lot of poorer countries with missing data. The association between wealth and happiness is there, but pretty weak (e.g. Zimbabwe gets 80+%, Bulgaria 55%). Given this (and when you throw in implied preferences, commonsensical intuitions whereby we don't wonder about whether we should jump in the pond to save the child as we're genuinely uncertain it is good for them to extent their life), it seems the average human takes themselves to have a life worth living. (q.v.)

2) My understanding from essays by Shulman and Tomasik is that even intensive factory farming plausibly leads to a net reduction in animal populations, given a greater reduction in wild animals due to habitat reduction. So if human extinction leads to another ~100M years of wildlife, this looks pretty bad by asymmetric views.

Of course, these estimates are highly non-resilient even with respect to sign. Yet the objective of the essay wasn't to show the result was robust to all reasonable moral considerations, but that the value of x-risk reduction isn't wholly ablated on a popular view of population ethics - somewhat akin to how Givewell analysis on cash transfers don't try and factor in poor meat eater considerations.

3) I neither 'tout' - nor even state - this is a finding that 'xrisk reduction is highly effective for person-affecting views'. Indeed, I say the opposite:

Although it seems unlikely x-risk reduction is the best buy from the lights of the [ed: typo - as context suggests, meant 'person-affecting'] total view (we should be suspicious if it were), given $13000 per life year compares unfavourably to best global health interventions, it is still a good buy: it compares favourably to marginal cost effectiveness for rich country healthcare spending, for example.

Comment author: AGB 16 April 2018 01:58:51AM *  2 points [-]

I suspect that the motivation hacking you describe is significantly harder for researchers than for, say, operations, HR, software developers, etc. To take your language, I do not think that the cause area beliefs are generally 'prudentially useful' for these roles, whereas in research a large part of your job may on justifying, developing, and improving the accuracy of those exact beliefs.

Indeed, my gut says that most people who would be good fits for these many critical and under-staffed supporting roles don't need to have a particularly strong or well-reasoned opinion on which cause area is 'best' in order to do their job extremely well. At which point I expect factors like 'does the organisation need the particular skills I have', and even straightforward issues like geographical location, to dominate cause prioritisation.

I speculate that the only reason this fact hasn't permeated into these discussions is that many of the most active participants, including yourself and Denise, are in fact researchers or potential researchers and so naturally view the world through that lens.

Comment author: Gregory_Lewis 20 April 2018 12:17:35PM 0 points [-]

I'd hesitate to extrapolate my experience across to operational roles for the reasons you say. That said, my impression was operations folks place a similar emphasis on these things as I. Tanya Singh (one my colleagues) gave a talk on 'x risk/EA ops'. From the Q&A (with apologies to Roxanne and Tanya for my poor transcription):

One common retort we get about people who are interested in operations is maybe they don't need to be value-aligned. Surely we can just hire someone who has operations skills but doesn't also buy into the cause. How true do you think this claim is?

I am by no means an expert, but I have a very strong opinion. I think it is extremely important to be values aligned to the cause, because in my narrow slice of personal experience that has led to me being happy, being content, and that's made a big difference as to how I approach work. I'm not sure you can be a crucial piece of a big puzzle or a tightly knit group if you don't buy into the values that everyone is trying to push towards. So I think it's very very important.

Comment author: MichaelPlant 13 April 2018 10:53:55PM 1 point [-]

No, in my comments I note precisely the opposite

In which case I'm not understanding your model. The 'Cost per life year' box is $1bn/EV. How is that not a one off of $1bn? What have I missed?

If the cost is 1B total to reduce risk for the subsequent century

As noted above, if people only live 70 years, then on PAA there's no point wondering what happens after 70 years.

I guess I'd be surprised if pricing by TRIA gives a huge discount

yeah, I don't think people have looked at this enough to form views on the figure. McMahan does want to discount future wellbeing for people by some amount, but is reluctant to be pushed into giving a number. I'd guess it's something like 2% a year. The effect something like assuming a 2% pure time discount.

Comment author: Gregory_Lewis 14 April 2018 12:07:15AM 0 points [-]

The EV in question is the reduction in x-risk for a single year, not across the century. I'll change the wording to make this clearer.

Comment author: MichaelPlant 13 April 2018 09:17:56PM *  -1 points [-]

Thanks for doing this. I definitely worry about the cause-selection fallacy where we go "X is the top cause if you believe theory T; I don't believe T, therefore X can't be my top cause".

A couple of points.

As you've noted in the comments, you model this as $1bn total, rather than $1bn a year. Ignoring the fact that the person affecting advocate (PAA) only cares about present people (at time of initial decision to spend), if the cost-effectivnenes is even 10 lower then it probably no long counts as a good buy.

Other person affecting views consider people who will necessarily exist (however cashed out) rather than whether they happen to exist now (planting a bomb with a timer of 1000 years is still accrues person-affecting harm). In a 'extinction in 100 years' scenario, this view would still count the harm of everyone alive then who dies, although still discount the foregone benefit of people who 'could have been' subsequently in the moral calculus

This is true, although whatever money you put towards the extinction project is likely to change all the identities, thus necessary people are effectively the same as present people. Even telling people "hey, we're working on this X-risk project" is enough to change all future identities.

If you wanted to pump up the numbers, you could claim that advances in aging will mean present people will live a lot longer - 200 years rather than 70. This strikes me as reasonable, at least when presented as an alternative, more optimistic calculation.

You're implicitly using the life-comparative account of the badness of death - the badness of your death is equal to amount of happiness you would have had if you'd lived. On this view, it's much more valuable to save the lives of very young people, i.e. whenever they count as a person, say 6 months after conception, or something. However, most PAAs, as far I can tell, take the Time-Relative Interest Account (TRIA) of the badness of death, which holds it's better to save a 20-year old than a 2-year old because the the 2-year old doesn't yet have interests in continuing to live. On TRIA, abortion isn't a problem, whereas it's a big loss on the life-comparative (assuming the foetus is terminated after personhood). This interests stuff is usually cashed out, at least by Jeff McMahan, in terms of Parfitian ideas about personal identity (apologies to those who aren't familiar with this shorthand). On TRIA, the value of saving a life is the happiness it would have had times the psychological continuity with one's future self. Very young people, e.g. babies, have basically no psychological continuity so saving their lives isn't important. But people keep changing over time: 20-year old is quite psychological distinct from the 80-year old. On TRIA, we need to factor that in too. This fact seems to be overlooked in the literature, but on TRIA you apply a discount to the future based on this change in psychological continuity. To push the point, suppose we say that everyone's psychology totally changes over the course of 10 years. Then TRIA advocates won't care what happens in 10 years time. Hence PAAs who like TRIA, which, as I say, seems to be most of them, will discount the value of the future much more steeply than PAA who endores the life-comparative account. Upshot: if someone takes TRIA seriously - which no one should btw - and knows what it implies, you'll really struggle to convince them X-risk is important on your estimate.

Finally, anyone who endorses the procreative asymmetry - creating happy people is neutral, creating unhappy people is bad - will want to try to increase x-risk and blow up the world. Why? Well, the future can only be bad: the happy lives don't count as good, and the unhappy lives will count as bad. Halstead discusses this here, if I recall correctly. It's true, on the asymmetry, avoiding x-risk would be good regarding current people, but increasing x-risk will be good regarding future people, as it will stop their being any of them. And as X-risk (reduction) enthusiasts are keen to point out, there is potentially a lot of future still to come.

Comment author: Gregory_Lewis 13 April 2018 10:19:18PM 3 points [-]

As you've noted in the comments, you model this as $1bn total, rather than $1bn a year. Ignoring the fact that the person affecting advocate (PAA) only cares about present people (at time of initial decision to spend), if the cost-effectivneness is even 10 lower then it probably no long counts as a good buy.

No, in my comments I note precisely the opposite. The model assumes 1B per year. If the cost is 1B total to reduce risk for the subsequent century, the numbers get more optimistic (100x optimistic if you buy counterpart-y views, but still somewhat better if you discount the benefit in future years by how many from the initial cohort remain alive).

Further, the model is time-uniform, so it can collapse into a 'I can spend 1B in 2018 to reduce xrisk in this year by 1% from a 0.01% baseline, and the same number gets spit out. So if a PAA buys these numbers (as Alex says, I think my offers skew conservative to xrisk consensus if we take them as amortized across-century risk, they might be about right/'optimistic' if they are taken as an estimate for this year alone), this looks an approximately good buy.

Population ethics generally, and PA views within them, are far from my expertise. I guess I'd be surprised if pricing by TRIA gives a huge discount, as I take most people consider themselves pretty psychologically continuous from the ages of ~15 onwards. If this isn't true, or consensus view amongst PAAs is "TRIA, and we're mistaken to our degree of psychological continuity", then this plausibly shaves off an order of magnitude-ish and plonks it more in the 'probably not a good buy' category.

Comment author: Carl_Shulman 13 April 2018 06:43:25PM 6 points [-]

Other person affecting views consider people who will necessarily exist (however cashed out) rather than whether they happen to exist now (planting a bomb with a timer of 1000 years is still accrues person-affecting harm). In a 'extinction in 100 years' scenario, this view would still count the harm of everyone alive then who dies, although still discount the foregone benefit of people who 'could have been' subsequently in the moral calculus.

Butterfly effects change the identities of at least all yet-to-be conceived persons, so this would have to not be interested in particular people, but population sizes/counterparts.

Comment author: Gregory_Lewis 13 April 2018 09:31:26PM 1 point [-]

+1. Navigating this is easier said than done, and one might worry about some sort of temporal parochialism being self-defeating (persons at t1-tn are all better off if they cooperate across cohort with future-regarding efforts instead of all concerning themselves with those who are morally salient at their corresponding t).

My impression is those with person-affecting sympathies prefer trying to meet these challenges rather than accept the moral character of destructive acts change with a (long enough) delay, or trying to reconcile this with the commonsensical moral importance of more normal future-regarding acts (e.g. climate change, town planning, etc.)

Comment author: Alex_Barry 13 April 2018 11:31:18AM *  1 point [-]

Edit: My comment is wrong - i had misread the price as £1 billion as a one-off, but it is £1 billion per year

I'm not quite able to follow what role annualising the risk plays in your model, since as far as I can tell you seem to calculate your final cost effectiveness in terms purely of the risk reduction in 1 year. This seems like it should undercount the impact 100-fold.

e.g. if I skip annualising entirely, and just work in century blocks I get:

  • still 247 Billion Life years at stake
  • 1% chance of x-risk, reduced to 0.99% by £1 billion project X.
  • This expected £ per year of life at 10^9/0.01%*247*10^9 = ~40, which is about 1/100 of your answer.

I might well have misunderstood some important part of your model, or be making some probability-related mistake.

Comment author: Gregory_Lewis 13 April 2018 11:58:51AM 2 points [-]

The mistake might be on my part, but I think where this may be going wrong is I assume the cost needs to be repeated each year (i.e. you spent 1B to reduce risk by 1% in 2018, then have to spend another 1B to reduce risk by 1% in 2019). So if you assume a single 1B pulse reduces x risk across the century by 1%, then you do get 100 fold better results.

I mainly chose the device of some costly 'project X' as it is hard to get a handle on (e.g.) whether 10^-10 reduction in xrisk/$ is a plausible figure or not. Given this, I might see if I can tweak the wording to make it clearer - or at least make any mistake I am making easier to diagnose.


The person-affecting value of existential risk reduction

Introduction The standard motivation for the far future cause area in general, and existential risk reduction in particular, is to point to the vast future that is possible providing we do not go extinct (see Astronomical Waste ). One crucial assumption made is a 'total' or 'no-difference' view of population... Read More
Comment author: Gregory_Lewis 12 April 2018 07:19:30AM 7 points [-]


FWIW I am one of the people doing something similar to what you advocate: I work in biorisk for comparative advantage reasons, although I think AI risk is a bigger deal.

That said, this sort of trading might be easier within broad cause areas than between them. My impression is received wisdom among the far future EAs is that both AI and bio are both 'big deals': AI might be (even more) important, yet bio (even more) neglected. For this reason even though I suspect most (myself included) would recommend a 'pluripotent far future EA' to look into AI first, it wouldn't take much to tilt the scales the other way (e.g. disposition, comparative advantage, and other things you cite). It also means individuals may not suffer a motivation hit if they are merely doing a very good thing rather than the very best thing by their lights. I think a similar thing applies to means that further a particular cause (whether to strike out on ones own versus looking for a role in an existing group, operations versus research, etc.)

When the issue is between cause areas, one needs to grapple with decisive considerations open chasms which are hard to cross with talent arbitrage. In the far future case, the usual story around astronomical waste etc. implies (pace Tomasik) that work on the far future is hugely more valuable than work in another cause area like animal welfare. Thus even if one is comparatively advantaged in animal welfare, one may still think their marginal effect is much greater in the far future cause area.

As you say, this could still be fertile ground for moral trade, and I also worry about more cynical reasons that explain this hasn't happened (cf. fairly limited donation trading so far). Nonetheless, I'd like to offer a few less cynical reasons that draw the balance of my credence.

As you say, although Allison and Bettina should think, "This is great, by doing this I get to have a better version of me do work on the cause I think is most important!" They might mutually recognise their cognitive foibles will mean they will struggle with their commitment to a cause they both consider objectively less important, and this term might outweigh their comparative advantage.

It also may be the case that developing considerable sympathy to a cause area may not be enough. Both intra- and outside EA, I generally salute well-intentioned efforts to make the world better: I wish folks working on animal welfare, global poverty, or (developed world) public health every success. Yet when I was doing the latter, despite finding it intrinsically valuable, I struggled considerably with motivation. I imagine the same would apply if I traded places with an 'animal-EA' for comparative advantage reasons.

It would been (prudentially) better if I could 'hack' my beliefs to find this work more intrinsically valuable. Yet people are (rightly) chary to try and hack prudentially useful beliefs (cf. Pascal's wager, where Pascal anticipated the 'I can't just change my belief in God' point, and recommended atheists go to church and other things which would encourage religious faith to take root), given it may have spillover into other domains where they take epistemic accuracy is very important. If cause area decisions mostly rely on these (which I hope they do), there may not be much opportunity to hack away this motivational bracken to provide fertile ground for moral trade. 'Attitude hacking' (e.g. I really like research, but I'd be better at ops, so I try to make myself more motivated by operations work) lacks this downside, and so looks much more promising.

Further, a better ex ante strategy across the EA community might be not to settle for moral trade, but instead discuss the merits of the different cause areas. Both Allison and Bettina take the balance of reason on their side, and so might hope either a) they get their counterpart to join them, or b) they realise they are mistaken and so migrate to something more important. Perhaps this implies an idealistic view of how likely people are to change their minds about these matters. Yet the track record of quite a lot people changing their minds about what cause areas are the most important (I am one example) gives some cause for hope.

Comment author: Dunja 01 March 2018 10:31:27AM 2 points [-]

Sure :) I saw that one on their website as well. But a few papers over the course of 2-3 years isn't very representative for an effective research group, is it? If you look at groups by scholars who do get (way smaller) grants in the field of AI, their output is way more effective. But even if we don't count publications, but speak in terms of effectiveness of a few publications, I am not seeing anything. If you are, maybe you can explain it to me?

Comment author: Gregory_Lewis 04 March 2018 12:20:40PM 0 points [-]

I regret I don't have much insight to offer on the general point. When I was looking into the bibliometrics myself, very broad comparison to (e.g.) Norwegian computer scientists gave figures like '~0.5 to 1 paper per person year', which MIRI's track record seemed about on par if we look at peer reviewed technical work. I wouldn't be surprised to find better performing research groups (in terms of papers/highly cited papers), but slightly moreso if these groups were doing AI safety work.

Comment author: Dunja 01 March 2018 09:17:16AM *  2 points [-]

Oh but you are confusing conference presentations with conference publications. Check the links you've just sent me: they discuss the latter, nor the former. You cannot cite conference presentation (or that's not what's usually understood under "citations", and definitely not in the links from your post), but only a publication. Conference publications in the field of AI are usually indeed peer-reviewed and yes, indeed, they are often even more relevant than journal publications, at least if published in prestigious conference proceedings (as I stated above).

Now, on MIRI's publication page there are no conference publications in 2017, and for 2016 there are mainly technical reports, which is fine, but should again not be confused with regular (conference) publications, at least according to the information provided by the publisher. Note that this doesn't mean technical reports are of no value! To the contrary. I am just making an overall analysis of the state of the art of MIRI's publications, and trying to figure out what they've published, and then how this compares with a publication record of similarly sized research groups in a similar domain. If I am wrong in any of these points, I'll be happy to revise my opinion!

Comment author: Gregory_Lewis 01 March 2018 10:18:30AM *  1 point [-]

This paper was in 2016, and is included in the proceedings of the UAI conference that year. Does this not count?

View more: Next