Comment author: JoeW 16 April 2017 10:33:11PM *  0 points [-]

Thanks for the reply. You're right that we can't be sure that conscious beings will do good things, but we don't have that assurance for any outcome we might push for.

If AIs are conscious, then a multipolar future filled with vast numbers of unaligned AIs could very plausibly be a wonderful future, brimming with utility. This isn't overwhelmingly obvious, but it's a real possibility. By contrast, if AIs aren't conscious then this scenario would represent a dead future. So distinguishing the two seems quite vital to understanding whether a multipolar outcome is bad or good.

You point out that even compared to the optimistic scenario I describe above, a correctly-aligned singleton could do better, by ensuring the very best future possible. True, but if a singleton isn't inevitable, trying to create one will itself pose a serious existential risk, as would any similar world takeover attempt. And even if the attempt is successful, we all agree that creating an aligned singleton is a very difficult task. Most singleton outcomes result in a universe almost entirely full of dead matter, produced by the singleton AI optimising for something irrelevant; even if it's conscious itself, resources that could have been put towards creating utility are almost all wasted as paperclips or whatever.

So it seems to me that, unless you're quite certain we're headed for a singleton future, the question of whether AIs will be conscious or not has a pretty huge impact on what path we should try to take.

Comment author: RobBensinger 17 April 2017 03:26:30AM *  1 point [-]

You're right that we can't be sure that conscious beings will do good things, but we don't have that assurance for any outcome we might push for.

One way to think about the goal is that we want to "zero in" on valuable futures: it's unclear what exactly a good future looks like, and we can't get an "assurance," but for example a massive Manhattan Project to develop whole brain emulation is a not-implausible path to zeroing in, assuming WBE isn't too difficult to achieve on the relevant timescale and assuming you can avoid accelerating difficult-to-align AI too much in the process. It's a potentially promising option for zeroing in because emulated humans could be leveraged to do a lot of cognitive work in a compressed period of time to sort out key questions in moral psychology+philosophy, neuroscience, computer science, etc. that we need to answer in order to get a better picture of good outcomes.

This is also true for a Manhattan Project to develop a powerful search algorithm that generates smart creative policies to satisfy our values, while excluding hazardous parts of the search space -- this is the AI route.

Trying to ensure that AI is conscious, without also solving WBE or alignment or global coordination or something of that kind in the process, doesn't have this "zeroing in" property. It's more of a gamble that hopefully good-ish futures have a high enough base rate even when we don't put a lot of work into steering in a specific direction, that maybe arbitrary conscious systems would make good things happen. But building a future of conscious AI systems opens up a lot of ways for suffering to end up proliferating in the universe, just as it opens up a lot of ways for happiness to end up proliferating in the universe. Just as it isn't obvious that e.g. rabbits experience more joy than suffering in the natural world, it isn't obvious that conscious AI systems in a multipolar outcome would experience more joy than suffering. (Or otherwise experience good-on-net lives.)

True, but if a singleton isn't inevitable, trying to create one will itself pose a serious existential risk, as would any similar world takeover attempt.

I think if an AI singleton isn't infeasible or prohibitively difficult to achieve, then it's likely to happen eventually regardless of what we'd ideally prefer to have happen, absent some intervention to prevent it. Either it's not achievable, or something needs to occur to prevent anyone in the world from reaching that point. If you're worried about singletons, I don't think pursuing multipolar outcomes and/or conscious-AI outcomes should be a priority for you, because I don't think either of those paths concentrates very much probability mass (if any) into scenarios where singletons start off feasible but something blocks them from occurring.

Multipolar scenarios are likelier to occur in scenarios where singletons simply aren't feasible, as a background fact about the universe; but conditional on singletons being feasible, I'm skeptical that achieving a multipolar AI outcome would do much (if anything) to prevent a singleton from occurring afterward, and I think it would make alignment much more difficult.

Alignment and WBE look like difficult tasks, but they have the "zeroing in" property, and we don't know exactly how difficult they are. Alignment in particular could turn out to be much harder than it looks or much easier, because there's so little understanding of what specifically is required. (Investigating WBE has less value-of-information because we already have some decent WBE roadmaps.)

Comment author: JoeW 14 April 2017 10:41:09PM *  1 point [-]

I'm still not sure how the consciousness issue can just be ignored. Yes, given the assumption that AIs will be mindless machines with no moral value, obviously we need to build them to serve humans. But if AIs will be conscious creatures with moral value like us, then...? In this case finding the right thing to do seems like a much harder problem, as it would be far from clear that a future in which machine intelligences gradually replace human intelligences represents a nightmare scenario, or even an existential risk at all. It's especially frustrating to see AI-risk folks treat this question as an irrelevance, since it seems to have such enormous implications on how important AI alignment actually is.

(Note that I'm not invoking 'ghost in the machine', I am making the very reasonable guess that our consciousness is a physical process that occurs in our brains, that it's there for the same reason other features of our minds are there - because it's adaptive - and that similar functionality might very plausibly be useful for an AI too.)

Comment author: RobBensinger 15 April 2017 01:45:36AM *  6 points [-]

Consciousness is something people at MIRI have thought about quite a bit, and we don't think people should ignore the issue. But it's a conceptually separate issue from intelligence, and it's important to be clear about that.

One reason to prioritize the AI alignment issue over consciousness research is that a sufficiently general solution to AI alignment would hopefully also resolve the consciousness problem: we care about the suffering, happiness, etc. of minds in general, so if we successfully build a system that shares and promotes our values, that system would hopefully also look out for the welfare of conscious machines, if any exist. That includes looking out for its own welfare, if it's conscious.

In contrast, a solution to the consciousness problem doesn't get us a solution to the alignment problem, because there's no assurance that conscious beings will do things that are good (including good for the welfare of conscious machines).

Some consciousness-related issues are subsumed in alignment, though. E.g., a plausible desideratum for early general AI systems is "limit its ability to model minds" ('behaviorist' AI). And we do want to drive down the probability that the system is conscious if we can find a way to do that.

Comment author: RyanCarey 04 April 2017 11:45:44PM 0 points [-]

Agree that that's the most common operationalization of a GCR. It's a bit inelegant for GCR not to include all x-risks though, especially given that it is used interchangeably within EA.

It would odd if the onset of a permanently miserable dictatorship didn't count as a global catastrophe because no lives were lost.

Comment author: RobBensinger 05 April 2017 05:53:14PM *  0 points [-]

Could you or Will provide an example of a source that explicitly uses "GCR" and "xrisk" in such a way that there are non-GCR xrisks? You say this is the most common operationalization, but I'm only finding examples that treat xrisk as a subset of GCR, as the Bostrom quote above does.

Comment author: William_MacAskill 30 March 2017 11:34:06PM *  9 points [-]

Agree that GCRs are a within-our-lifetime problem. But in my view mitigating GCRs is unlikely to be the optimal donation target if you are only considering the impact on beings alive today. Do you know of any sources that make the opposite case?

And it's framed as long-run future because we think that there are potentially lots of things that could have a huge positive on the value of the long-run future which aren't GCRs - like humanity having the right values, for example.

Comment author: RobBensinger 31 March 2017 08:43:25PM *  2 points [-]

And it's framed as long-run future because we think that there are potentially lots of things that could have a huge positive on the value of the long-run future which aren't GCRs - like humanity having the right values, for example.

I don't have much to add to what Rob W and Carl said, but I'll note that Bostrom defined "existential risk" like this back in 2008:

A subset of global catastrophic risks is existential risks. An existential risk is one that threatens to cause the extinction of Earth-originating intelligent life or to reduce its quality of life (compared to what would otherwise have been possible) permanently and drastically.

Presumably we should replace "intelligent" here with "sentient" or similar. The reason I'm quoting this is that on the above definition, it sounds like any potential future event or process that would cost us a large portion of the future's value counts as an xrisk (and therefore as a GCR). 'Humanity's moral progress stagnates or we otherwise end up with the wrong values' sounds like a global catastrophic risk to me, on that definition. (From a perspective that does care about long-term issues, at least.)

I'll note that I think there's at least some disagreement at FHI / Open Phil / etc. about how best to define terms like "GCR", and I don't know if there's currently a consensus or what that consensus is. Also worth noting that the "risk" part is more clearly relevant than the "global catastrophe" part -- malaria and factory farming are arguably global catastrophes in Bostrom's sense, but they aren't "risks" in the relevant sense, because they're already occurring.

Comment author: RobBensinger 30 March 2017 09:47:22PM *  12 points [-]

To some, it’s just obvious that future lives have value and the highest priority is fighting existential threats to humanity (‘X-risks’).

I realize this is just an example, but I want to mention as a side-note that I find it weird what a common framing this is. AFAIK almost everyone working on existential risk think it's a serious concern in our lifetimes, not specifically a "far future" issue or one that turns on whether it's good to create new people.

As an example of what I have in mind, I don't understand why the GCR-focused EA Fund is framed as a "long-term future" fund (unless I'm misunderstanding the kinds of GCR interventions it's planning to focus on), or why philosophical stances like the person-affecting view and presentism are foregrounded. The natural things I'd expect to be foregrounded are factual questions about the probability and magnitude over the coming decades of the specific GCRs EAs are most worried about.

Comment author: RobBensinger 17 March 2017 10:43:57PM *  3 points [-]

I think wild animal suffering isn't a long-term issue except in scenarios where we go extinct for non-AGI-related reasons. The three likeliest scenarios are:

  1. Humans leverage AGI-related technologies in a way that promotes human welfare as well as (non-human) animal welfare.

  2. Humans leverage AGI-related technologies in a way that promotes human welfare and is effectively indifferent to animal welfare.

  3. Humans accidentally use AGI-related technologies in a way that is indifferent to human and animal welfare.

In all three scenarios, the decision-makers are likely to have "ambitious" goals that favor seizing more and more resources. In scenario 2, efficient resource use almost certainly implies that biological human bodies and brains get switched out for computing hardware running humans, and that wild animals are replaced with more computing hardware, energy/cooling infrastructure, etc. Even if biological humans who need food stick around for some reason, it's unlikely that the optimal way to efficiently grow food in the long run will be "grow entire animals, wasting lots of energy on processes that don't directly increase the quantity or quality of the food transmitted to humans".

In scenario 1, wild animals might be euthanized, or uploaded to a substrate where they can live whatever number of high-quality lives seems best. This is by far the best scenario, especially for people who think (actual or potential) non-human animals might have at least some experiences that are of positive value, or at least some positive preferences that are worth fulfilling. I would consider this extremely likely if non-human animals are moral patients at all, though scenario 1 is also strongly preferable if we're uncertain about this question and want to hedge our bets.

Scenario 3 has the same impact on wild animals as scenario 1, and for analogous reasons: resource limitations make it costly to keep wild animals around. 3 is much worse than 1 because human welfare matters so much; even if the average present-day human life turned out to be net-negative, this would be a contingent fact that could be addressed by improving global welfare.

I consider scenario 2 much less likely than scenarios 1 and 3; my point in highlighting it is to note that scenario 2 is similarly good for the purpose of preventing wild animal suffering. I also consider scenario 2 vastly more likely than "sadistic" scenarios where some agent is exerting deliberate effort to produce more suffering in the world, for non-instrumental reasons.

Comment author: ThomasSittler 11 March 2017 10:40:23AM 4 points [-]

I think I've only ever seen cause-neutrality used to mean cause-impartiality.

Comment author: RobBensinger 15 March 2017 01:18:31AM 2 points [-]

The discussion of CFAR's pivot to focusing on existential risk seemed to use "cause-neutral" to mean something like "cause-general".

Confusingly, the way "cause-neutral" was used there directly contradicts its use here: there, it meant avoiding cause-impartially favoring a specific cause based on its apparent expected value, in favor of a cause-partial commitment to pet causes like rationality and EA capacity-building. (Admittedly, at the organizational level it often makes sense to codify some "pet causes" even if in principle the individuals in that organization are trying to maximize global welfare impartially.)

Comment author: RobBensinger 25 February 2017 11:15:11PM 4 points [-]

One of the spokes of the Leverhulme Centre for the Future of Intelligence is at Imperial College London, headed by Murray Shanahan.

Comment author: RobBensinger 27 February 2017 08:02:25PM 3 points [-]
Comment author: Raemon 25 February 2017 10:37:20PM 1 point [-]

Thanks, fixed.

Actually, is anyone other than DeepMind in London? (the section I brought this up was on volunteering, which I assume is less relevant for DeepMind than FHI)

Comment author: RobBensinger 25 February 2017 11:15:11PM 4 points [-]

One of the spokes of the Leverhulme Centre for the Future of Intelligence is at Imperial College London, headed by Murray Shanahan.

In response to comment by kbog  (EA Profile) on Why I left EA
Comment author: klloyd 20 February 2017 10:47:17PM 0 points [-]

You're probably correct, reading up I realise I didn't understand it as well as I think I did, but I still have a few questions. If one is a particularist and anti-realist how do those judgements have any force that can possibly be called moral? As for moral uncertainty, I meant that if one ascribes some non-zero probability to there being genuine moral demands on one, it would seem one still has reason to follow them. If you're right then nothing you do matters so you've lost nothing. If you're wrong you have done something good. So, it would seem moral uncertainty gives one reasons to act in a certain way, because some probability of doing good has some motivating power even if not as much as certainly doing good. I think I was mixed up about non-cognitivism, but some people seem to be called non-cognitivists and realists? For example David Hume, who I've heard called a non-cognitivist and a consequentialist, and Simon Blackburn who is called a quasi-realist despite being a non-cognitivist. Are either of these people properly called realists?

In response to comment by klloyd on Why I left EA
Comment author: RobBensinger 20 February 2017 10:58:07PM *  2 points [-]

I think the intuition that moral judgments need to have "force" or make "demands" is a bit of a muddle, and some good readings for getting un-muddled here are:

  1. Peter Hurford's "A Meta-Ethics FAQ"
  2. Eliezer Yudkowsky's Mere Goodness
  3. Philippa Foot's "Morality as a System of Hypothetical Imperatives"
  4. Peter Singer's "The Triviality of the Debate Over 'Is-Ought' and the Definition of 'Moral'"

Kyle might have some better suggestions for readings here.

View more: Next