Hide table of contents

I believe the far future is a very important consideration in doing the most good, but I don’t focus on reducing extinction risks like those from unfriendly artificial intelligence. This post introduces and outlines some of the key considerations that went into that decision, and leaves discussion of the best answers to those considerations for future work.

The different types of x-risk

In “Astronomical Waste”, written by Nick Bostrom in 2003, an existential risk (commonly referred to as a x-risk) is described as “one where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.” In effective altruism and rationalist circles, we have most commonly addressed x-risks of the former type, those that could “annihilate Earth-originating intelligent life.” Let’s call those population risks.


The other risks could “permanently and drastically curtail [Earth-originating intelligent life’s] potential.” Let’s assume our “potential” is to create a very large and very good civilization, with “very large” referencing the number of sentient beings and “very good” being up to the reader. (I’ll assume that you believe there are at least some scenarios with a very large civilization that wouldn’t be very good, such as those filled with more suffering than happiness.)


I think humanity could fail in either or both of these respects. In this post, I’ll focus on the risk of creating a very large but not very good civilization, but I think the other types are worth exploring some other time. We’ll use the term quality risk to mean a risk of having a very large civilization that’s not very good.


Note that considering quality risks explicitly isn’t a new idea at all. Nick Beckstead wrote about it in August 2014, referring to previous mentions of it in 2009, 2013, and another in 2014. Beckstead explicitly states, “Astronomical waste may involve changes in quality of life, rather than size of population.”

“I think we should focus our efforts almost entirely on improving the expected value of the far future. Should I work on quality risks or extinction risks?”

I think more people in the effective altruism and rationalist communities should be asking themselves this question. I think people often make the jump from “the far future is very important” to “I should work on the most important extinction risks” too quickly.

“But what if I’m not exclusively focused on the far future?”

There are reasonable justifications for not focusing entirely on the far future, such as concerns about the tractability of making a substantive difference, and wanting to give limited weight to linear arguments like that most commonly used to justify focusing on the far future. I personally give significant weight to both far future and near-term outcomes. But for this post, I’ll focus exclusively on far future impact.

“But why can’t I just reduce both sorts of x-risk?”

Some activities might reduce both quality and extinction risks. For example, getting more people involved in EA might increase the number of people working in each area. Also, research that increases the likelihood of friendly AI might not only reduce the risk of extinction, but also might affect the relative likelihoods of different non-extinction AI scenarios, some of which might be better than others. I think this is super interesting, but for this post, I’ll only consider the tradeoff itself.

Considerations

To better understand whether to focus on quality risks or extinction risks, I think there are a number of informative questions we can ask. Unfortunately, the questions that seem most useful if we had answers also seem quite intractable and little research has been done on them.


Because quality risk is such a diverse category, I’ll use widening our moral circles as an example of how we could increase the expected value of the far future given we continue to exist, mainly because it’s what I see as most promising and have considered most fully. By this, I mean, “Increasing the extent to which we accommodate the interests of all sentient individuals, rather than just those who are most similar to us or have the most power in society.” It seems that narrow moral circles and a lack of concern for all sentient beings could lead civilization to be much worse, given it continues to exist, posing a substantial quality risk.


Here’s a list of some particularly interesting considerations, in my opinion. Explanations are included for considerations with less obvious relevance, and I think there’s a good chance I’m leaving out a few important ones or not breaking them down optimally (e.g. “tractability” could be segmented into several different important considerations).


  • Quality risk tractability: How tractable is social change, such as widening moral circles? e.g. Does activism better society, or is it really just things like globalization and economic optimization that make moral change occur?


  • Extinction risk tractability: How tractable is extinction risk? e.g. Can determined individuals and small groups affect how humanity utilizes its technology, or will powerful and too-hard-to-affect governments determine our fate?


  • Scale: Are dystopian futures sufficiently likely that reducing quality risks has more potential impact, since these risks would constitute the difference between a far future much worse than nonexistence and a far future much better than nonexistence?


  • Neglectedness: How many resources are currently being spent on each sort of risk? Is one more neglected, either in terms of attention from the effective altruism community or in terms of society as a whole?

    (We could expect diminishing returns from investment in a specific cause area as more and more people take the “low hanging fruit.” Increasing returns might also apply in some situations.)


  • Inevitability of moral progress: Is the “moral fate” of the universe inevitable, given we continue to exist? Are we inexorably doomed because of the intrinsic evils of markets and evolution (i.e. systems that optimize for outcomes like survival and potentially lead to morally bad outcomes like mass suffering)? Or are we destined for utopia because we’ll implement a powerful AI that creates the best possible universe?

    (If moral fate is inevitable, then it’s not much use trying to change it.)


  • Moral value of the far future: In expectation, is humanity’s future existence a good thing? Will we create a utopian shockwave across the universe, or create cruel simulations of large numbers of sentient beings for our amusement or experimentation? [Edit: Apologize for the formatting issue.] (If it’s a neutral or bad outcome, then working on extinction risk seems less promising and maybe even harmful.)


  • Political power in the far future: Will all sentient beings of the future, if they exist, be politically powerful in the sense humans in the developed world (arguably) are today? Or will many of them be like the animals suffering in animal agriculture?

    (If they are powerful, then it seems widening our moral circles isn’t as important, but if the powerful beings of the future need to account for the interests of powerless individuals, then having wide moral circles seems very important.)


  • Quality of life getting ‘stuck’: If we continue to exist, will our quality of life get ‘stuck’ at some point, such as if an authoritarian government comes into power or we start sending probes out into space who are unable to communicate with others?

    (Quality improvements seem more urgent if it’s more likely to get stuck.)

Future research and discussion

Further analyses of these considerations and introduction of new ones could be quite valuable [Edit: As noted in the comments, some analyses do exist for some of these questions. I could have been clearer about that.]. My best guesses on these questions — which are largely just intuitions at this point — lead me to favor working on widening our moral circles instead of reducing extinction risk, but I could change my mind on that and I hope others would also be willing to do so. It’s also worth considering the prioritization of finding better answers to these questions, even if they seem mostly intractable.

I worry that many of us who focus on long-term impact haven’t given much thought to these considerations and mostly just went with the norms of our social circles. Upon considering them now, I think it’s tempting to just settle them in the direction that favors our current activities, and I hope people try to avoid doing so.


Comments34
Sorted by Click to highlight new comments since: Today at 11:15 PM

This is a critically important and neglected topic, and I'm glad you wrote about it. I've written about this distinction before but I think you did a much better job of explaining why it matters.

Here are some more writings on the subject, along with a summary of my favorite points from each article:

Michael Bitton: Why I Don't Prioritize GCRs

  • GCR prevention only matters if they will happen soon enough
  • If one GCR happens first, the others don't matter, but we don't know which will come first
  • Efforts to direct humanity have a poor track record

Brian Tomasik: Values Spreading Is Often More Important than Extinction Risk

  • Most people are incentivized to prevent extinction but not many people care about my/our values
  • (Mathematical argument that I can't really simplify but is worth reading in full)

Paul Christiano: Against Moral Advocacy

  • If we try to change values now, they will tend to drift
  • I don't want to lock in my current values because they could be wrong
  • Values tend to be similar, so it is possible to pursue competing objectives with only modest losses in efficiency

Paul Christiano: Why Might the Future Be Good?

  • The far future might be good either because (1) rational self-interested agents will make gains from trade or (2) future altruists will share my values and have power
  • Whether we expect (1) or (2) changes what we should do now
  • Natural selection might make people more selfish, but everyone is incentivized to survive no matter their values so selfish people won't have an adaptive advantage
  • People who care more about the (far) future will have more influence on it, so natural selection favors them

I'd double-upvote this if I could. Providing (high-quality) summaries along with links is a great pro-social norm!

A couple of remarks:

GCR prevention only matters if they will happen soon enough

The very same, from a future perspective, applies to values-spreading.

Most people are incentivized to prevent extinction but not many people care about my/our values

This is a suspiciously antisocial approach that only works if you share Brian's view that not only are their no moral truths for future people to (inevitably) discover, but nonetheless it is very important to promote one's current point of view on moral questions over whatever moral views are taken in the future.

The very same, from a future perspective, applies to values-spreading.

Why do you think that? There are different values we can change that seem somewhat independent.

This is a suspiciously antisocial approach

That seems mean and unfair. Having different values than the average person doesn't make you antisocial or suspicious; it just makes you different. In fact, I'd say most EAs have different values than average :)

The very same, from a future perspective, applies to values-spreading.

Why do you think that? There are different values we can change that seem somewhat independent.

If you spread some value and then extinction eventuates, then your work does not matter in the long run. So this doesn't separate the two courses of action, on the long-run view.

That seems mean and unfair. Having different values than the average person doesn't make you antisocial or suspicious; it just makes you different. In fact, I'd say most EAs have different values than average :)

That's not how it's antisocial. It's antisocial in the literal sense that it is antagonistic to social practises. Basically it's more than believing in uncommon values, it's acting out in a way that violates what we recognise to at least be valid heuristics like not engaging in zero sum competition, and especially not moralising efforts to do so. If EAs and humanity can't cooperate while disagreeing, it's bad news. Calling it mean and unfair is a premature judgement.

If you mean antisocial in the literal sense, you could and should probably have clarified that originally.

If you mean it in the usual sense, where it's approximately synonymous with labelling Brian as 'offensive' or 'uncaring', then the charge of 'mean and unfair' seems reasonable.

Either way you shouldn't be surprised that someone would interpret in the usual sense and consider it unfair.

That's not really an accurate representation, I'm trying to say that it's anti-cooperative, which it mostly is, moreso than offensive or uncaring.

This certainly gets quite a bit of attention in internal conversations at the Future of Humanity Institute. Bostrom discussed it when first(?) writing about existential risk in 2001, under the name shrieks. Note I wouldn't recommend reading that paper except for historical interest -- his more modern exposition in Existential Risk Prevention as Global Priority is cleaner and excellent. I think your quality risk coincides with Bostrom's notion of flawed realisation, although you might also mean to include subsequent ruination. Could you clarify?

Anyhow I'll give my view briefly:

  • Much of the focus on risk from AI is about flawed realisations (from locking in the wrong values) more than never getting big.
  • Aside from concrete upcoming cases to lock-in values, it's unclear whether we can affect the long-term trajectory. However we might be able to, so this gives only a modest reason to discount working to mitigate the risks of flawed realisations.
  • There are lots of plausible ways to indirectly help reduce future risk (both extinction risk and other kinds), by putting us in a better position to face future challenges. The further off challenges are, the more this looks like the right strategy. For extinction risks, some of them are close enough that the best portfolio looks like it includes quite a bit of directly addressing the risks. For risks of flawed realisation apart from AI, my guess is that the portfolio should be skewed heavily towards this capacity-building.
  • Many of the things we think of to do to improve long-term capacity to deal with challenges look less neglected right now than the direct risks. But not all (e.g. I think nurturing the growth of a thoughtful EA movement may be helpful here), and we should definitely be open to finding good opportunities in this space.
  • I would like to see more work investigating the questions in this area.

When I get multiple downvotes I like to use this to learn not to do things which people find unhelpful. Often I can go back and re-read my comment and work out what people didn't like. Here I'm not so sure -- something about tone? The fact that I gave my own framing of the issue more than building on the framing in the OP? Mixing two unrelated points (history of discussion and my views now) in one comment?

I'd appreciate pointers from anyone who downvoted, or who didn't but felt a temptation to. I don't want to discuss whether my post 'deserved' downvotes, I just want to understand what about it would drive them.

The downvoting throughout this thread looks funny. Absent comments, I'd view it as a weak signal.

"Quality risk" is meant to include both of those ideas, just any situation where we get "very large" (~"technologically mature") but not "very good."

Michael Dickens wrote about quality risks vs existential risks here and here.

Thanks for noting. I should have included links to those and other existing materials in the post. Was just trying to go quickly and show an independent perspective.

I talk about values spreading rather than quality risks, but they're similar since the most commonly discussed way to mitigate quality risks is via values spreading.

(I would actually be interested in a discussion of what we could do to mitigate quality risks other than values spreading.)

In the first post, there is a link to a study. The estimate for all humanity dying in war within one century is 4% but the chance of them perishing in a nuclear war is 1%. This implies a 3% chance of extinction in a non-nuclear war. Am I reading this right? How would that make sense?

You are reading it right, and you're right to be suspicious of these numbers (although it doesn't seem totally impossible to me, if the wars involved some other extremely deadly weapons).

I think what's going on is that the numbers are just survey responses from a group of people (admittedly experts). In such circumstances people tend to give numbers which represent their impressions on issues, without necessarily checking for consistency / reflective equilibrium. I'm particularly suspicious of the fact that the ratio between estimated chances of fatalities for the two kinds of conflict doesn't scale with the number of fatalities.

Ok. This is not too related to this thread, but I wonder how big a risk a US-Russia (or equivalently-sized) nuclear war is in the next century. This article suggests between a 1% per year and 1% per decade risk, based on Martin Hellman's work. Hellman prefers 1% per year. But a risk that high seems hard to square with how folks are acting now-you would think that if the risk was that high, the wealthiest and smartest folks in the world would avoid living in cities that are vulnerable to being blown up. I don't think they are avoiding cities. So I am inclined to go with 1% per decade. This is still very uncertain.

2/9-Looks like the experts have spoken and so I raise that to 2% per decade. But it could still be 1% for a "total" war that destroys major cities.

2/13-Here is an even better report that gives about the same odds, but a 6.8% chance of a nuclear conflict anywhere in the world killing more people than in WWII. The highest odds seem to be from countries like Israel, Iran, North Korea, India, and Pakistan, as opposed to the great powers.

Thanks a lot for this article! I just wanted to link to Lukas Gloor's new paper on Fail-Safe AI, which discusses the reduction of "quality future-risks" in the context of AI safety. It turns out that there might be interventions that are less directed at achieving a perfect outcome, but instead try to avoid the worst outcomes. And those interventions might be more tractable (because they don't aim at such a tiny spot in value-space) and more neglected than other work on the control problem. https://foundational-research.org/wp-content/uploads/2016/08/Suffering-focused-AI-safety.pdf

Do you see any specific examples where reducing other types of existential risks could increase quality risks?

Moving towards political singleton, and increasing surveillance technology, both look like they should help to reduce risks of human extinction. But they may well increase the risk of locking in a value system which is suboptimal, whereas a more varied society could do better in expectation (particularly after different parts of the society trade with each other).

If you expect the far future to be net negative in expectation, then reducing existential risk necessarily increases quality risk. In this essay I list some reasons why the far future might be net negative:

  • We sustain or worsen wild animal suffering on earth.
  • We colonize other planets and fill them with wild animals whose lives are not worth living.
  • We create lots of computer simulations of extremely unhappy beings.
  • We create an AI with evil values that creates lots of suffering on purpose. (But this seems highly unlikely.)

In the essay I discuss how likely I think these scenarios are.

In your essay you place a lot of weight on other people's opinions. I wonder, if for some reason you decided to disregard everyone else's opinion, do you know if you would reach a different conclusion?

My probabilities would be somewhat different, yes. I originally wrote "I’d give about a 60% probability that the far future is net positive, and I’m about 70% confident that the expected value of the far future is net positive." If I didn't care about other people's opinions, I'd probably revise this to something like 50%/60%.

It seems to me that the most plausible future scenario is we continue doing what we've been doing, the dominant effect of which is that we sustain wild animal populations that are probably net negative. I've heard people give arguments for why we shouldn't expect this, but I'm generally wary of arguments that say "the world will look like this 1000 years from now, even though it has never looked like this before and hardly anybody expects this to happen," which is the type of argument used to justify that wild animal suffering won't be a problem in the far future.

I believe most people are overconfident in their predictions about what the far future will look like (an, in particular, on how much the far future will be dominated by wild animal suffering and/or suffering simulations). But the fact that pretty much everyone I've talked to expects the far future to be net positive does push me in that direction, especially people like Carl Shulman and Brian Tomasik* who seem to think exceptionally clearly and level-headedly.

*This isn't exactly what Brian believes; see here.

Okay. Do you see any proxies (besides other people's views) that, if they changed in our lifetime, might shift your estimates one way or the other?

Off the top of my head:

  • We develop strong AI.
  • There are strong signals that we would/wouldn't be able to encode good values in an AI.
  • Powerful people's values shift more toward/away from caring about non-human animals (including wild animals) or sentient simulations of non-human minds.
  • I hear a good argument that I hadn't already heard or thought of. (I consider this pretty likely, given how little total thought has gone into these questions.)

I have created a roadmap of x-risks prevention, and I think that it is complete and logically ordered. I will make longer post about it if i will be able to get enough carma. )) The pdf is here: http://immortality-roadmap.com/globriskeng.pdf

Thanks, this made me think more deeply about quality risks - appreciate it!

My tendency is to think that a good path forward may be to focus on goal factoring activities, namely ones that hit many types of risks at once - x-risks and quality risks. I appreciate what Owen brought up here about building up the EA movement as one way to do so. I think spreading rationality is another way to do so.

Both trying to change attitudes to animals, and reducing extreme poverty, have been advocated on the basis that it could create trajectory changes that improves the quality of the future.

Christiano has some thoughts here:

http://rationalaltruist.com/2013/06/13/against-moral-advocacy/

Moral value of the far future Political power in the far future

These two are discussed in Superintelligence.

Also see these sections of the original x-risk paper:

"Crunches – The potential of humankind to develop into posthumanity[7] is permanently thwarted although human life continues in some form.

Shrieks – Some form of posthumanity is attained but it is an extremely narrow band of what is possible and desirable.

Whimpers – A posthuman civilization arises but evolves in a direction that leads gradually but irrevocably to either the complete disappearance of the things we value or to a state where those things are realized to only a minuscule degree of what could have been achieved."

http://www.nickbostrom.com/existential/risks.html

"Quality of life getting ‘stuck’"

See Bostrom's comments on a negative 'singleton'.

A fantastically interesting article. I wish I'd seen it earlier -- about the time this was published (last February) I was completing an article on "agential risks" that ended up in the Journal of Evolution and Technology. In it, I distinguish between "existential risks" and "stagnation risks," each of which corresponds to one of the disjuncts in Bostrom's original definition. Since these have different implications -- I argue -- for understanding different kinds of agential risks, I think it would be good to standardize the nomenclature. Perhaps "population risks" and "quality risks" are preferable (although I'm not sure "quality risks" and "stagnations risks" have exactly the same extension). Thoughts?

(Btw, the JET article is here: http://jetpress.org/v26.2/torres.pdf.)

Thanks for investing your thoughts in this area.

This has been a prominent part of existential risk reduction discussion since at least 2003 (edit 2013) when Nick Beckstead wrote his article about "Trajectory Changes", which are a slightly cleaner version of your " quality risks". (1) Trajectory changes are events whose impact persists in the long term, though not by preventing extinction.

This article was given to Nick Bostrom at the time, who replied to it at the time, which gives you a ready made reply to your article from the leader and originator of the existential risk idea:


One can arrive at a more probably correct principle by weakening, eventually arriving at something like 'do what is best' or 'maximize expected good'. There the well-trained analytic philosopher could rest, having achieved perfect sterility. Of course, to get something fruitful, one has to look at the world not just at our concepts.

Many trajectory changes are already encompassed within the notion of an existential catastrophe. Becoming permanently locked into some radically suboptimal state is an xrisk. The notion is more useful to the extent that likely scenarios fall relatively sharply into two distinct categories---very good ones and very bad ones. To the extent that there is a wide range of scenarios that are roughly equally plausible and that vary continuously in the degree to which the trajectory is good, the existential risk concept will be a less useful tool for thinking about our choices. One would then have to resort to a more complicated calculation. However, extinction is quite dichotomous, and there is also a thought that many sufficiently good future civilizations would over time asymptote to the optimal track.

In a more extended and careful analysis there are good reasons to consider second-order effects that are not captured by the simple concept of existential risk. Reducing the probability of negative-value outcomes is obviously important, and some parameters such as global values and coordination may admit of more-or-less continuous variation in a certain class of scenarios and might affect the value of the long-term outcome in correspondingly continuous ways. (The degree to which these complications loom large also depends on some unsettled issues in axiology; so in an all-things-considered assessment, the proper handling of normative uncertainty becomes important. In fact, creating a future civilization that can be entrusted to resolve normative uncertainty well wherever an epistemic resolution is possible, and to find widely acceptable and mutually beneficial compromises to the extent such resolution is not possible---this seems to me like a promising convergence point for action.)

It is not part of the xrisk concept or the maxipok principle that we ought to adopt some maximally direct and concrete method of reducing existential risk (such as asteroid defense): whether one best reduces xrisk through direct or indirect means is an altogether separate question.


The reason people don't usually think about trajectory changes (and quality risks) is not that they've just overlooked that possibility. It's that absent some device for fixing them in society, the (expected) impact of most societal changes decays over time. Changing a political system or introducing and spreading new political and moral ideologies is one of the main kinds of trajectory changes proposed. However, it is not straightforward to argue that such an ideology would be expected to thrive for millenea when almost all other poliyical and ethical ideologies have not. In contrast, a whole-Earth extinction event could easily end life in our universe for eternity.

So trajectory changes (or quality risks) are important in theory, to be sure. The challenge that the existential risk community has not yet successfully achieved, is to think of ones that are probable and worth moving altruistic resources towards, that could as easily be used to reduce extinction risk.

1.http://lesswrong.com/lw/hjb/a_proposed_adjustment_to_the_astronomical_waste/

Thanks for sharing. I think my post covers some different ground (e.g. the specific considerations) than that discussion, and it's valuable to share an independent perspective.

I do agree it touches on many of the same points.

I might not agree with your claim that it's been a "prominent" part of discussion. I rarely see it brought up. I also might not agree that "Trajectory Changes" are a slightly cleaner version of "quality risks," but those points probably aren't very important.

As to your own comments at the end:

The reason people don't usually think about trajectory changes (and quality risks) is not that they've just overlooked that possibility.

Maybe. Most of the people I've spoken with did just overlook (i.e. didn't give more than an hour or two of thought - probably not more than 5 minutes) the possibility, but your experience may be different.

It's that absent some device for fixing them in society, the (expected) impact of most societal changes decays over time.

I'm not sure I agree, although this claim is a bit vague. If society's value (say, moral circles) is rated on a scale of 1 to 100 at every point in time and is currently at, say, 20, then even if there's noise that moves it up and down, a shift of 1 will increase the expected value at every future time period.

You might mean something different.

However, it is not straightforward to argue that such an ideology would be expected to thrive for millenea when almost all other poliyical and ethical ideologies have not.

I don't think it's about having the entire "ideology" survive, just about having it affect future ideologies. If you widen moral circles now, then the next ideology that comes along might have slightly wider circles than it would otherwise.

The challenge that the existential risk community has not yet successfully achieved, is to think of ones that are probable and worth moving altruistic resources towards, tgat couod as easily be used to reduce exyinction risk.

As a community, I agree. And I'm saying that might be because we haven't put enough effort into considering them. Although personally, I see at least one of those (widening moral circles) as more promising than any of the extinction risks currently on our radar. But I'm always open to arguments against that.

(On a lighter note) On re-reading Nick Beckstead's post, I spent a while thinking "dear lord, he was an impossibly careful-thinking/well-informed teenager"*. Then I realised you'd meant 2013, not 2003 ;)

(*This is not to say he wasn't a very smart, well-informed teenager, of course. And with the EA community in particular, this would not be so unlikely - the remarkable quality and depth of analysis in posts being published on the EA forum by people in their late teens and early twenties is one of the things that makes me most excited about the future!)

It's that absent some device for fixing them in society, the (expected) impact of most societal changes decays over time.

AGI is plausibly such a device. MIRI and Bostrom seem to place reasonable probability on a goal-preserving superintelligence (since goal preservation is a basic AI drive). AGI could preserve values more thoroughly than any human institutions possibly can, since worker robots can be programmed not to have goal drift, and in a singleton scenario without competition, evolutionary pressures won't select for new values.

So it seems the values that people have in the next centuries could matter a lot for the quality of the future from now until the stars die out, at least in scenarios where human values are loaded to a nontrivial degree into the dominant AGI(s).

The effect of most disasters decays over time, but this does not mean that a disaster so big it ends humanity is not possible. So I don't see why that most societal changes decay over time bears on whether large trajectory changes could happen. Maybe someday, there will be a uniquely huge change.

Also, I don't understand why Bostrom mentions a "thought" that all sufficiently good civilizations will converge toward an optimal track. This seems like speculation.

Here is a concern I have. It may be that reducing many types of existential risk, like of nuclear war, could lower economic growth.

http://www.nytimes.com/2014/06/14/upshot/the-lack-of-major-wars-may-be-hurting-economic-growth.html?hpw&rref=

How do we know that by avoiding war we are not increasing another sort of existential risk, or the risk of permanent economic stagnation? Depending on how much we want to risk a total nuclear war, human development on Earth might have many permanent equilibria.