Comment author: ThomasSittler 13 April 2018 07:02:58AM *  3 points [-]

I think the marginal effects are crucial. If we think of the community as needing one ops person and one research person, the marginal value in each area drops to zero once that role is filled.

Take a simple model where everyone follows their absolute advantage (taking into account marginal considerations and what everyone else is doing). You still get problems when Alice and Bob need to choose simultaneously. Alice does not know what Bob will do, so she does not know what all the marginal considerations are. Alice and Bob should coordinate on each following their comparative advantage.

Owen Cotton-Barratt has a nice explanation of comparative advantage applied to EA.

Comparative advantage is a tricky one to wrap one's head around intuitively! (I study economics and I've often made mistakes about it, in public).

Comment author: jsteinhardt 17 April 2018 12:48:32AM 1 point [-]

If we think of the community as needing one ops person and one research person, the marginal value in each area drops to zero once that role is filled.

Yes, but these effects only show up when the number of jobs is small. In particular: If there are already 99 ops people and we are looking at having 99 vs. 100 ops people, the marginal value isn't going to drop to zero. Going from 99 to 100 ops people means that mission-critical ops tasks will be done slightly better, and that some non-critical tasks will get done that wouldn't have otherwise. Going from 100 to 101 will have a similar effect.

In contrast, in the traditional comparative advantage setting, there remain gains-from-coordination/gains-from-trade even when the total pool of jobs/goods is quite large.

The fact that gains-from-coordination only show up in the small-N regime here, whereas they show up even in the large-N regime traditionally, seems like a crucial difference that makes it inappropriate to apply standard intuition about comparative advantage in the present setting.

If we want to analyze this more from first principles, we could pick one of the standard justifications for considering comparative advantage and I could try to show why it breaks down here. The one I'm most familiar with is the one by David Ricardo (https://en.wikipedia.org/wiki/Comparative_advantage#Ricardo's_example).

Comment author: jsteinhardt 13 April 2018 02:34:44AM 4 points [-]

I'm worried that you're mis-applying the concept of comparative advantage here. In particular, if agents A and B both have the same values and are pursuing altruistic ends, comparative advantage should not play a role---both agents should just do whatever they have an absolute advantage at (taking into account marginal effects, but in a large population this should often not matter).

For example: suppose that EA has a "shortage of operations people" but person A determines that they would have higher impact doing direct research rather than doing ops. Then in fact the best thing is for person A to work on direct research, even if there are already many other people doing research and few people doing ops. (Of course, person A could be mistaken about which choice has higher impact, but that is different from the trade considerations that comparative advantage is based on.)

I agree with the heuristic "if a type of work seems to have few people working on it, all else equal you should update towards that work being more neglected and hence higher impact" but the justification for that again doesn't require any considerations of trading with other people . In general, if A and B can trade in a mutually beneficial way, then either A and B have different values or one of them was making a mistake.

Comment author: Joey 05 November 2017 01:13:32PM 3 points [-]

The use of the term talent constrained vs talent limited was not intentional.

Overall I think salary is not a large factor in our talent concerns. We have experimented with different levels of salaries between 10k and 50k USD and have not found increasing the salary increases the talent pool in the traits we would like to see more of. It could be that 50k is still too low or that we are not marketing our jobs in communities that are very income sensitive. I would guess that normally we are looking for pretty hardcore/dedicated EAs and that tends to correlate very strongly with people who take low salaries.

Comment author: jsteinhardt 06 November 2017 09:23:04PM *  5 points [-]

FWIW, 50k seems really low to me (but I live in the U.S. in a major city, so maybe it's different elsewhere?). Specifically, I would be hesitant to take a job at that salary, if for no other reason than I thought that the organization was either dramatically undervaluing my skills, or so cash-constrained that I would be pretty unsure if they would exist in a couple years.

A rough comparison: if I were doing a commissioned project for a non-profit that I felt was well-run and value-aligned, my rate would be in the vicinity of $50USD/hour. I'd currently be willing to go down to $25USD/hour for a project that is something I basically would have done anyways. Once I get my PhD I think my going rates would be higher, and for a senior-level position I would probably expect more than either of these numbers, unless it was a small start-up-y organization that I felt was one of the most promising organizations in existence.

EDIT: So that people don't have to convert to per-year salaries in their heads, the above numbers if annualized would be $100k USD/year and $50k USD/year.

Comment author: Wei_Dai 13 July 2017 11:37:14AM 5 points [-]

That actually didn't cross my mind before, so thanks for pointing it out. After reading your comment, I decided to look into Open Phil's recent grants to MIRI and OpenAI, and noticed that of the 4 technical advisors Open Phil used for the MIRI grant investigation (Paul Christiano, Jacob Steinhardt, Christopher Olah, and Dario Amodei), all either have a ML background or currently advocate a ML-based approach to AI alignment. For the OpenAI grant however, Open Phil didn't seem to have similarly engaged technical advisors who might be predisposed to be critical of the potential grantee (e.g., HRAD researchers), and in fact two of the Open Phil technical advisors are also employees of OpenAI (Paul Christiano and Dario Amodei). I have to say this doesn't look very good for Open Phil in terms of making an effort to avoid potential blind spots and bias.

Comment author: jsteinhardt 13 July 2017 03:16:20PM 4 points [-]

(Speaking for myself, not OpenPhil, who I wouldn't be able to speak for anyways.)

For what it's worth, I'm pretty critical of deep learning, which is the approach OpenAI wants to take, and still think the grant to OpenAI was a pretty good idea; and I can't really think of anyone more familiar with MIRI's work than Paul who isn't already at MIRI (note that Paul started out pursuing MIRI's approach and shifted in an ML direction over time).

That being said, I agree that the public write-up on the OpenAI grant doesn't reflect that well on OpenPhil, and it seems correct for people like you to demand better moving forward (although I'm not sure that adding HRAD researchers as TAs is the solution; also note that OPP does consult regularly with MIRI staff, though I don't know if they did for the OpenAI grant).

Comment author: jsteinhardt 11 July 2017 03:55:45PM *  5 points [-]

This doesn't match my experience of why I find Paul's justifications easier to understand. In particular, I've been following MIRI since 2011, and my experience has been that I didn't find MIRI's arguments (about specific research directions) convincing in 2011*, and since then have had a lot of people try to convince me from a lot of different angles. I think pretty much all of the objections I have are ones I generated myself, or would have generated myself. Although, the one major objection I didn't generate myself is the one that I feel most applies to Paul's agenda.

( * There was a brief period shortly after reading the sequences that I found them extremely convincing, but I think I was much more credulous then than I am now. )

Comment author: jsteinhardt 11 July 2017 03:59:21PM 5 points [-]

I think the argument along these lines that I'm most sympathetic to is that Paul's agenda fits more into the paradigm of typical ML research, and so is more likely to fail for reasons that are in many people's collective blind spot (because we're all blinded by the same paradigm).

Comment author: Wei_Dai 11 July 2017 08:42:52AM *  1 point [-]

the gap seems much smaller to me when it comes to the justification for thinking HRAD is promising vs justification for Paul's approach being promising

This seems wrong to me. For example, in the "learning to reason from human" approaches, the goal isn't just to learn to reason from humans, but to do it in a way that maintains competitiveness with unaligned AIs. Suppose a human overseer disapproves of their AI using some set of potentially dangerous techniques, how can we then ensure that the resulting AI is still competitive? Once someone points this out, proponents of the approach, to continue thinking their approach is promising, would need to give some details about how they intend to solve this problem. Subsequently, justification for thinking the approach is promising is more subtle and harder to understand. I think conversations like this have occurred for MIRI's approach far more than Paul's, which may be a large part of why you find Paul's justifications easier to understand.

Comment author: jsteinhardt 11 July 2017 03:55:45PM *  5 points [-]

This doesn't match my experience of why I find Paul's justifications easier to understand. In particular, I've been following MIRI since 2011, and my experience has been that I didn't find MIRI's arguments (about specific research directions) convincing in 2011*, and since then have had a lot of people try to convince me from a lot of different angles. I think pretty much all of the objections I have are ones I generated myself, or would have generated myself. Although, the one major objection I didn't generate myself is the one that I feel most applies to Paul's agenda.

( * There was a brief period shortly after reading the sequences that I found them extremely convincing, but I think I was much more credulous then than I am now. )

Comment author: Wei_Dai 09 July 2017 08:53:55AM 18 points [-]

3c. Other research, especially "learning to reason from humans," looks more promising than HRAD (75%?)

From the perspective of an observer who can only judge from what's published online, I'm worried that Paul's approach only looks more promising than MIRI's because it's less "mature", having received less scrutiny and criticism from others. I'm not sure what's happening internally in various research groups, but the amount of online discussion about Paul's approach has to be at least an order of magnitude less than what MIRI's approach has received.

(Looking at the thread cited by Rob Bensinger, various people including MIRI people have apparently looked into Paul's approach but have not written down their criticisms. I've been trying to better understand Paul's ideas myself and point out some difficulties that others may have overlooked, but this is hampered by the fact that Paul seems to be the only person who is working on the approach and can participate on the other side of the discussion.)

I think Paul's approach is certainly one of the most promising approaches we currently have, and I wish people paid more attention to it (and/or wrote down their thoughts about it more), but it seems much too early to cite it as an example of an approach that is more promising than HRAD and therefore makes MIRI's work less valuable.

Comment author: jsteinhardt 10 July 2017 03:44:18AM 9 points [-]

Shouldn't this cut both ways? Paul has also spent far fewer words justifying his approach to others, compared to MIRI.

Personally, I feel like I understand Paul's approach better than I understand MIRI's approach, despite having spent more time on the latter. I actually do have some objections to it, but I feel it is likely to be significantly useful even if (as I, obviously, expect) my objections end up having teeth.

Comment author: JoshuaFox 27 February 2017 11:06:56AM *  2 points [-]

Outreach can be valuable, although it is rare to have high-value opportunities. If you can publish, lecture or talk 1-on-1 with highly relevant audiences, then you may sway the Zeitgeist a little and so contribute towards getting donors or researchers on board.

Relevant audiences include:

  • tech moguls and other potential big donors; people who may have the potential to become or at influence those moguls.

  • researchers in relevant areas such as game theory; smart people in elite educational tracks who may have the potential to become or influence such researchers.

Comment author: jsteinhardt 28 February 2017 06:48:15AM 11 points [-]

I already mention this in my response to kbog above, but I think EAs should approach this cautiously; AI safety is already an area with a lot of noise, with a reputation for being dominated by outsiders who don't understand much about AI. I think outreach by non-experts could end up being net-negative.

Comment author: kbog  (EA Profile) 27 February 2017 06:52:45AM *  2 points [-]

What about online activism? There are lots of debates in various corners of the Internet over AI which often involve people in various areas of academia and tech. It seems like it could be valuable and feasible for people who are sufficiently educated on the basic issues of AI alignment to correct misconceptions and spread good ideas.

As another idea, there are certain kinds of information which would be worth collecting: surveys of relevant experts, taxonomies of research ideas and developments in the field, information about the political and economic sides of AI research. I suppose this could fall into gruntwork for safety orgs, but they don't comprehensively ask for every piece of information and work which could be useful.

Also - this might sound strange, but if someone wants to contribute then it's their choice: students and professionals might be more productive if they had remote personal assistants to handle various tasks which are peripheral to one's primary tasks and responsibilities, and if someone is known to be an EA, value aligned on cause priorities, and moderately familiar with the technical work, then having someone do this seems very feasible.

Comment author: jsteinhardt 28 February 2017 06:46:01AM 12 points [-]

In general I think this sort of activism has a high potential for being net negative --- AI safety already has a reputation as something mainly being pushed by outsiders who don't understand much about AI. Since I assume this advice is targeted at the "average EA" (who presumably doesn't know much about AI), this would only exacerbate the issue.

Comment author: Kerry_Vaughan 29 January 2017 08:12:35PM 7 points [-]

I don't agree with thejadedone's conclusions or think his post is particularly well-thought-out, but I don't think raising the bar on criticism like this is very productive if you care about getting good criticism. (If you think thejadedone's criticism is bad criticism, then I think it makes sense to just argue for that rather than saying that they should have made it privately.)

I agree with this and wasn't trying to say something to the contrary. What I was trying to do is note that the post makes a relatively minor issue into an expose on EA and on 80K. I think this is unnecessary and unwarranted by the issue. What is was trying to do is note one way of handling the issue if your goal is merely to gain more information or see that a problem gets fixed.

I think public criticism is fine. I think a good, but not required, practice is to show the criticism to the organization ahead of publishing it so that they can correct factual inaccuracies. I think that would have improved the criticism substantially in this case.

Comment author: jsteinhardt 30 January 2017 06:08:29AM 1 point [-]

Thanks for clarifying; your position seems reasonable to me.

View more: Next