Comment author: Lee_Sharkey 04 October 2017 12:16:04AM 0 points [-]

AI accidents brings to my mind trying to prevent robots crashing into things. 90% of robotics work could be classed as AI accident prevention because they are always crashing into things.

It is not just funding confusion that might be a problem. If I'm reading a journal on AI safety or taking a class on AI safety what should I expect? Robot mishaps or the alignment problem? How will we make sure the next generation of people can find the worthwhile papers/courses?

I take the point. This is a potential outcome, and I see the apprehension, but I think it's a probably a low risk that users will grow to mistake robotics and hardware accidents for AI accidents (and work that mitigates each) - sufficiently low that I'd argue expected value favours the accident frame. Of course, I recognize that I'm probably invested in that direction.

Perhaps we should take a hard left and say that we are looking at studying Artificial Intelligence Motivation? People know that an incorrectly motivated person is bad and that figuring out how to motivate AIs might be important. It covers the alignment problem and the control problem.

Most AI doesn't look like it has any form of motivation and is harder to rebrand as such, so it is easier to steer funding to the right people and tell people what research to read.

I think this steers close to an older debate on AI “safety” vs “control” vs “alignment”. I wasn't a member of that discussion so am hesitant to reenact concluded debates (I've found it difficult to find resources on that topic other than what I've linked - I'd be grateful to be directed to more). I personally disfavour 'motivation' on grounds of risk of anthropomorphism.

Comment author: WillPearson 04 October 2017 07:13:34PM 0 points [-]

I take the point. This is a potential outcome, and I see the apprehension, but I think it's a probably a low risk that users will grow to mistake robotics and hardware accidents for AI accidents (and work that mitigates each) - sufficiently low that I'd argue expected value favours the accident frame. Of course, I recognize that I'm probably invested in that direction.

I would do some research onto how well sciences that have suffered brand dilution do.

As far as I understand it Research institutions have high incentives to

  1. Find funding
  2. Pump out tractible digestible papers

See this kind of article for other worries about this kind of thing.

You have to frame things with that in mind, give incentives so that people do the hard stuff and can be recognized for doing the hard stuff.

Nanotech is a classic case of a diluted research path, if you have contacts maybe try and talk to Erik Drexler, he is interested in AI safety so might be interested in how the AI Safety research is framed.

I think this steers close to an older debate on AI “safety” vs “control” vs “alignment”. I wasn't a member of that discussion so am hesitant to reenact concluded debates (I've found it difficult to find resources on that topic other than what I've linked - I'd be grateful to be directed to more). I personally disfavour 'motivation' on grounds of risk of anthropomorphism.

Fair enough I'm not wedded to motivation (I see animals having motivation as well, so not strictly human). It doesn't seem to cover Phototaxis which seems like the simplest thing we want to worry about. So that is an argument against motivation. I'm worded out at the moment. I'll see if my brain thinks of anything better in a bit.

Comment author: Lee_Sharkey 03 October 2017 01:54:41PM 1 point [-]

I think this proposition could do with some refinement. AI safety should be a superset of both AGI safety and narrow-AI safety. Then we don't run into problematic sentences like "AI safety may not help much with AGI Safety", which contradicts how we currently use 'AI safety'.

To address the point on these terms, then:

I don't think AI safety runs the risk of being so attractive that misallocation becomes a big problem. Even if we consider risk of funding misallocation as significant, 'AI risk' seems like a worse term for permitting conflation of work areas.

Yes, it's of course useful to have two different concepts for these two types of work, but this conceptual distinction doesn't go away with a shift toward 'AI accidents' as the subject of these two fields. I don't think a move toward 'AI accidents' awkwardly merges all AI safety work.

But if it did: The outcome we want to avoid is AGI safety getting too little funding. This outcome seems more likely in a world that makes two fields of N-AI safety and AGI safety, given the common dispreference for work on AGI safety. Overflow seems more likely in the N-AI Safety -> AGI Safety direction when they are treated as the same category than when they are treated as different. It doesn't seem beneficial for AGI safety to market the two as separate types of work.

Ultimately, though, I place more weight on the other reasons why I think it's worth reconsidering the terms.

Comment author: WillPearson 03 October 2017 06:26:27PM *  0 points [-]

I agree it is worth reconsidering the terms!

The agi/narrow ai distinction is beside the point a bit, I'm happy to drop it. I also have an AI/IA bugbear so I'm used to not liking how things are talked about.

Part of the trouble is we have lost the marketing war before it even began, every vaguely advanced technology we have currently is marketing itself as AI, that leaves no space for anything else.

AI accidents brings to my mind trying to prevent robots crashing into things. 90% of robotics work could be classed as AI accident prevention because they are always crashing into things.

It is not just funding confusion that might be a problem. If I'm reading a journal on AI safety or taking a class on AI safety what should I expect? Robot mishaps or the alignment problem? How will we make sure the next generation of people can find the worthwhile papers/courses?

AI risks is not perfect, but is not at least it is not that.

Perhaps we should take a hard left and say that we are looking at studying Artificial Intelligence Motivation? People know that an incorrectly motivated person is bad and that figuring out how to motivate AIs might be important. It covers the alignment problem and the control problem.

Most AI doesn't look like it has any form of motivation and is harder to rebrand as such, so it is easier to steer funding to the right people and tell people what research to read.

It doesn't cover my IA gripe, which briefly is: AI makes people think of separate entities with their own goals/moral worth. I think we want to avoid that as much of possible. General Intelligence augmentation requires its own motivation work, but one so that the motivation of the human is inherited by the computer that human is augmenting. I think that my best hope is that AGI work might move in that direction.

Comment author: WillPearson 02 October 2017 09:30:42PM 0 points [-]

So what are the risks of this verbal change?

Potentially money gets mis-allocated: Just like all chemistry got rebranded nanotech during that phase in the 2000, if there is money in AI safety, computer departments will rebrand research as AI safety to prevent AI accidents. This might be a problem when governments start to try and fund AI Safety.

I personally want to be able to differentiate different types of work, between AI Safety and AGI Safety. Both are valuable, we are going to living in a world of AI for a while and it may cause catastrophic problems (including problems that distract us from AGI safety) and learning to mitigating them might help us with AGI Safety. I want us to be able continue to look at both as potentially separate things, because AI Safety may not help much with AGI Safety.

Comment author: WillPearson 29 September 2017 11:27:02AM *  2 points [-]

I think an important thing for Ai strategy is to figure out ishow to fund empirical studies into questions that impinge on crucial considerations.

For example funding studies into the nature of IQ. I'll post an article on that later but wanted to flag it here as well.

Comment author: John_Maxwell_IV 28 September 2017 10:04:03AM *  17 points [-]

In Tetlock's book Superforecasting, he distinguishes between two skills related to forecasting: generating questions, and answering them. This "disentanglement research" business sounds more like the first sort of work. Unfortunately, Tetlock's book focuses on the second skill, but I do believe he talks some about the first skill (e.g. giving examples of people who are good at it).

I would imagine that for generating questions, curiosity and creativity are useful. Unfortunately, the Effective Altruism movement seems to be bad at creativity.

John Cleese gave this great talk about creativity in which he distinguishes between two mental modes, "open mode" and "closed mode". Open mode is good for generating ideas, whereas closed mode is good for accomplishing well-defined tasks. It seems to me that for a lot of different reasons, the topic of AI strategy might put a person in closed mode:

  • Ethical obligation - Effective altruism is often framed as an ethical obligation. If I recall correctly, surveys indicate that around half of the EA community sees EA as more of an obligation than an opportunity. Obligations don't typically create a feeling of playfulness.

  • Size of the problem - Paul Graham writes: "Big problems are terrifying. There's an almost physical pain in facing them." AI safety strategy is almost the biggest problem imaginable.

  • Big names - People like Nick Bostrom, Eliezer Yudkowsky, and Eric Drexler have a very high level of prestige within the EA community. (The status difference between them and your average EA is greater than what I've observed between the students & the professor in any college class I remember taking.) Eliezer in particular can get very grumpy with you if you disagree with him. I've noticed that I'm much more apt to generate ideas if I see myself as being at the top of the status hierarchy, and if there is no penalty for coming up with a "bad" idea (even a bad idea can be a good starting point). One idea for solving the EA community's creativity problem is to encourage more EAs to develop Richard Feynman-level indifference to our local status norms.

  • Urgency - As you state in this post, every second counts! Unfortunately urgency typically has the effect of triggering closed mode.

  • Difficulty - As you state in this post, many brilliant people have tried & failed. For some people, this fact is likely to create a sense of intimidation which precludes creativity.

For curiosity, one useful exercise I've found is Anna Salamon's practice of setting a 7-minute timer and trying to think of as many questions as possible within that period. The common pattern here seems to be "quantity over quality". If you're in a mental state where you feel a small amount of reinforcement for a bad idea, and a large amount of reinforcement for a good idea, don't be surprised if a torrent of ideas follows (some of which are good).

Another practice I've found useful is keeping a notebook. Harnessing "ambient thought" and recording ideas as they come to me, in the appropriate notebook page, seems to be much more efficient on a per-minute basis than dedicated brainstorming.

If I was attacking this problem, my overall strategic approach would differ a little from what you are describing here.

I would place less emphasis on intellectual centralization and more emphasis on encouraging people to develop idiosyncratic perspectives/form their own ontologies. Rationale: if many separately developed idiosyncratic perspectives all predict that a particular action X is desirable, that is good evidence that we should do X. There's an analogy to stock trading here. (Relatedly, the finance/venture capital industry might be the segment of society that has the most domain expertise related to predicting the future, modulo principle-agent problems that come with investing other peoples' money. Please let me know if you can think of other candidates... perhaps the intelligence community?)

Discipline could be useful for reading books & passing classes which expand one's library of concepts, but once you get to the original reasoning part, discipline gets less useful. Centralization could be useful for making sure that the space of ideas relevant to AI strategy gets thoroughly covered through our collective study, and for helping people find intellectual collaborators. But I would go for beers, whiteboards, and wikis with long lists of crowdsourced pros and cons, structured to maximize the probability that usefully related ideas will at one point or another be co-located in someone's working memory, before any kind of standard curriculum. I suspect it's better to see AI strategy as a fundamentally interdisciplinary endeavor. (It might be useful to look at successful interdisciplinary research groups such as the Santa Fe Institute for ideas.) And forget all that astronomical waste nonsense for a moment. We are in a simulation. We score 1 point if we get a positive singularity, 0 points otherwise. Where is the loophole in the game's rules that the designers didn't plan for?

[Disclaimer: I haven't made a serious effort to survey the literature or systematically understand the recommendations of experts on either creativity or curiosity, and everything in this comment is just made up of bits and pieces I picked up here and there. If you agree with my hunch that creativity/curiosity are a core part of the problem, it might be worth doing a serious lit review/systematically reading authors who write about this stuff such as Thomas Kuhn, plus reading innovators in various fields who have written about their creative process.]

Comment author: WillPearson 28 September 2017 11:03:12AM 1 point [-]

I agree that creativity is key.

I'd would point out that you may need discipline to do experiments based upon your creative thoughts (if the information you need is not available). If you can't check your original reasoning against the world, you are adrift in a sea of possibilities.

Comment author: klevanoff  (EA Profile) 27 September 2017 11:08:06PM *  7 points [-]

Carrick, this is an excellent post. I agree with most of the points that you make. I would, however, like to call attention to the wide consensus that exists in relation to acting prematurely.

As you observe, there are often path dependencies at play in AI strategy. Ill-conceived early actions can amplify the difficulty of taking corrective action at a later date. Under ideal circumstances, we would act under as close to certainty as possible. Achieving this ideal, however, is impractical for several interrelated reasons:

  1. AI strategy is replete with wicked problems. The confidence that we can have in many (most?) of our policy recommendations must necessarily be relatively low. If the marginal costs of further research are high, then undertaking that research may not be worthwhile.

  2. Delaying policy recommendations can sometimes be as harmful as or more harmful than making sub-par policy recommendations. There are several reasons for this. First, there are direct costs (e.g., lives lost prior to implementing sanitary standards). Second, delays allow other actors--most of whom are less concerned with rigor and welfare--to make relative gains in implementing their favored policies. If outcomes are path dependent, then inaction from AI strategists can lead to worse effects than missteps. Third, other actors are likely to gain influence if AI strategists delay. Opaque incentive structures and informal networks litter the path from ideation to policymaking. Even if there are not path dependencies baked into the policies themselves, there are sociopolitical path dependencies in the policymaking process. Gaining clout at an early stage tends to increase later influence. If AI strategists are unwilling to recommend policies, others will do so and reap the reputational gains entailed. Inversely, increased visibility may confer legitimacy to AI strategy as a discipline.

  3. Policy communities in multiple countries are becoming more aware of AI, and policymaking activity is poised to increase. China's national AI strategy, released several months ago, is a long-range plan, the implementation of which is being carried out by top officials. For the CCP, AI is not a marginal issue. Westerners will look to Chinese policies to inform their own decisions. In Washington, think tanks are increasingly recognizing the importance of AI. The Center for a New American Security, for example, now has a dedicated AI program (https://www.cnas.org/research/technology-and-national-security/artificial-intelligence) and is actively hiring. Other influential organizations are following suit. While DC policymakers paid little attention to AlphaGo, they definitely noticed Putin's comments on AI's strategic importance earlier this month. As someone with an inside vantage point, I can say with a high degree of confidence that AI will not remain neglected for long. Inaction on the part of AI strategists will not mean an absence of policy; it will mean the implementation of less considered policy.

As policy discussions in relation to AI become more commonplace and more ideologically motivated, EAs will likely have less ability to influence outcomes, ceteris paribus (hence Carrick's call for individuals to build career capital). Even if we are uncertain about specific recommendations--uncertainty that may be intractable--we will need to claim a seat at the table or risk being sidelined.

There are also many advantages to starting early. To offer a few:

  1. If AI strategists are early movers, they can wield disproportionate influence in framing the discourse. Since anchoring effects can be large, introducing policymakers to AI through the lens of safety rather than, say, national military advantage is probably quite positive in expectation.

  2. Making policy recommendations can be useful in outsourcing cognitive labor. Once an idea becomes public, others can begin working on it. Research rarely becomes policy overnight. In the interim period, proponents and critics alike can refine thinking and increase the analytical power brought to bear on a topic. This enables greater scrutiny for longer-range thought that has no realistic path to near-term implementation, and may result in fewer unidentified considerations.

  3. Taking reversible harmful actions at an early stage allows us to learn from our mistakes. If these mistakes are difficult to avoid ex ante, and we wait until later to make them, the consequences are likely to be more severe. Of course, we may not know which actions are reversible. This indicates to me that researching path dependence in policymaking would be valuable.

This is not a call for immediate action, and it is not to suggest that we should be irresponsible in making recommendations. I do, however, think that we should increasingly question the consensus around inaction and begin to consider more seriously how much uncertainty we are willing to accept, as well as when and how to take a more proactive approach to implementation.

Comment author: WillPearson 28 September 2017 09:09:54AM 0 points [-]

I think it is important to note that in the political world there is the vision of two phases of AI development, narrow AI and general AI.

Narrow AI is happening now. The 30+% job loss predictions in the next 20 years, all narrow AI. This is what people in the political sphere are preparing for, from my exposure to it.

General AI is conveniently predicted more that 20 years away, so people aren't thinking about it because they don't know what it will look like and they have problems today to deal with.

Getting this policy response right to narrow AI does have a large impact. Large scale unemployment could destabilize countries, causing economic woes and potentially war.

So perhaps people interested in general AI policy should get involved with narrow AI policy, but make it clear that this is the first battle in a war, not the whole thing. This would place them well and they could build up reputations etc. They could be be in contact with the disentanglers so that when the general AI picture is clearer, they can make policy recommendations.

I'd love it if the narrow-general AI split was reflected in all types of AI work.

In response to S-risk FAQ
Comment author: aspencer 26 September 2017 03:00:33PM 1 point [-]

This sentence in your post caught my attention: " Even if the fraction of suffering decreases, it's not clear whether the absolute amount will be higher or lower."

To me, it seems like suffering should be measured by suffering / population, rather than by the total amount of suffering. The total amount of suffering will grow naturally with the population, and suffering / population seems to give a better indication of the severity of the suffering (a small group suffering a large amount is weighted higher than a large group suffering a small amount, as I intuitively think is correct).

My primarily concern with this (simplistic) method of measuring the severity of suffering is that it ignores the distribution of suffering within a population (i.e there could be a sub population with a large amount of suffering). However, I don't think that's a compelling enough reason to discount working to minimize the fraction of suffering rather than absolute suffering.

Are there compelling arguments for why we should seek to minimize total suffering?

In response to comment by aspencer on S-risk FAQ
Comment author: WillPearson 27 September 2017 08:44:26PM 0 points [-]

How do you feel about the mere addition paradox? These questions are not simple.

Comment author: WillPearson 27 September 2017 08:01:33PM *  4 points [-]

I would broadly agree. I think this is an important post and I agree with most of the ways to prepare. I think we are not there yet for large scale AI policy/strategy.

There are few things that I would highlight as additions. 1) We need to cultivate the skills of disentanglement. Different people might be differently suited, but like all skills it is one that works better with practice and people to practice with. Lesswrong is trying to place itself as that kind of place. It is having a little resurgence with the new website www.lesserwrong.com. For example there has been lots of interesting discussion on the problems of Goodheart's law, which will be necessary to at least somewhat solve if we are to get AISafety groups that actually do AISafety research and don't just optimise some research output metric to get funding.

I am not sure if lesswrong is the correct place, but we do need places for disentanglers to grow.

2) I would also like to highlight the fact that we don't understand intelligence and that there have been lots of people studying it for a long time (psychologists etc) that I don't think we do enough to bring into discussing artificial versions of the thing they have studied. Lots of work on policy side of AI safety models it as utility maximimising agent in the economic style. I am pretty skeptical that is a good model of humans or of the AIs we will create. Figuring out what better models might be, is on the top of my personal priority list.

Edited to add 3) It seems like a sensible policy is to fund a competition in the style of at super forecasting aimed at AI and related technologies. This should give you some idea of the accuracy of peoples view on technology development/forecasting.

I would caution that we are also in the space of wicked problems so it may be there is never a complete certainty of the way we should move.

Comment author: Tuukka_Sarvi 21 September 2017 05:20:51PM *  0 points [-]

I think I have now a better understanding of what you meant.

I think there are at least three optimization problems here: 1) what to produce? (investment decision) 2) how to produce? (organization of operations) and 3) how to use the returns , for EA-minded, how to donate?

Company traditionally optimizes 2) and 1) in a more restricted manner (within their field of business or local opportunities)

I think there might be some problems with a hypothetical "benevolent" company that also commits to donate all the profits to an charity or portfolio of charities. Firstly, it would decrease the possible investor base because only strictly altruistic investors would be interested and thus it would not likely able to raise as much funding as a "non-benevolent" company (altruistic investors are also interested in "non-benevolent" companies because they can freely donate any profits they make). Secondly, there is disagreement among altruists of how to best donate. Thus, if profits are given to investors, each altruist can choose personally how to donate. So even altruistic investors might be hesitant to invest in a "benevolent" company I outlined here.

"So I was trying to break down the concept of ownership some more and arguing that in a benevolent world private ownership might only mean keeping control over operations."

There is still disagreement about how to best donate (to do most good) among individuals which gives support to the argument that profits should be paid out even among altruistic investor base

Comment author: WillPearson 26 September 2017 01:11:21PM *  0 points [-]

There is still disagreement about how to best donate (to do most good) among individuals which gives support to the argument that profits should be paid out even among altruistic investor base

True, but to if I put myself in the perfect altruist company owner shoes I would really want to delegate the allocation of the my charitable giving, because I am too busy running my company to have much good information about who to donate to.

If I come happen to come in to some information about what good charitable giving is, I should be able to take the information to whoever I have delegated it too and they should incorporate it (being altruists wanting to do the most good as well).

It seems only when you distrust other agents, either morally, or their ability to update on information should you allocate it yourself.

Does that explain my intuitions?

Comment author: Michelle_Hutchinson 12 September 2017 02:44:28PM *  2 points [-]

Will, you might be interested in these conversation notes between GiveWell and the Tax Justice Network: http://files.givewell.org/files/conversations/Alex_Cobham_07-14-17_(public).pdf (you have to c&p the link)

Comment author: WillPearson 22 September 2017 08:06:50PM 0 points [-]

Michelle, thanks. Yes very interesting!

View more: Next