Comment author: Wei_Dai 13 July 2017 11:37:14AM 5 points [-]

That actually didn't cross my mind before, so thanks for pointing it out. After reading your comment, I decided to look into Open Phil's recent grants to MIRI and OpenAI, and noticed that of the 4 technical advisors Open Phil used for the MIRI grant investigation (Paul Christiano, Jacob Steinhardt, Christopher Olah, and Dario Amodei), all either have a ML background or currently advocate a ML-based approach to AI alignment. For the OpenAI grant however, Open Phil didn't seem to have similarly engaged technical advisors who might be predisposed to be critical of the potential grantee (e.g., HRAD researchers), and in fact two of the Open Phil technical advisors are also employees of OpenAI (Paul Christiano and Dario Amodei). I have to say this doesn't look very good for Open Phil in terms of making an effort to avoid potential blind spots and bias.

Comment author: jsteinhardt 13 July 2017 03:16:20PM 4 points [-]

(Speaking for myself, not OpenPhil, who I wouldn't be able to speak for anyways.)

For what it's worth, I'm pretty critical of deep learning, which is the approach OpenAI wants to take, and still think the grant to OpenAI was a pretty good idea; and I can't really think of anyone more familiar with MIRI's work than Paul who isn't already at MIRI (note that Paul started out pursuing MIRI's approach and shifted in an ML direction over time).

That being said, I agree that the public write-up on the OpenAI grant doesn't reflect that well on OpenPhil, and it seems correct for people like you to demand better moving forward (although I'm not sure that adding HRAD researchers as TAs is the solution; also note that OPP does consult regularly with MIRI staff, though I don't know if they did for the OpenAI grant).

Comment author: jsteinhardt 11 July 2017 03:55:45PM *  5 points [-]

This doesn't match my experience of why I find Paul's justifications easier to understand. In particular, I've been following MIRI since 2011, and my experience has been that I didn't find MIRI's arguments (about specific research directions) convincing in 2011*, and since then have had a lot of people try to convince me from a lot of different angles. I think pretty much all of the objections I have are ones I generated myself, or would have generated myself. Although, the one major objection I didn't generate myself is the one that I feel most applies to Paul's agenda.

( * There was a brief period shortly after reading the sequences that I found them extremely convincing, but I think I was much more credulous then than I am now. )

Comment author: jsteinhardt 11 July 2017 03:59:21PM 4 points [-]

I think the argument along these lines that I'm most sympathetic to is that Paul's agenda fits more into the paradigm of typical ML research, and so is more likely to fail for reasons that are in many people's collective blind spot (because we're all blinded by the same paradigm).

Comment author: Wei_Dai 11 July 2017 08:42:52AM *  1 point [-]

the gap seems much smaller to me when it comes to the justification for thinking HRAD is promising vs justification for Paul's approach being promising

This seems wrong to me. For example, in the "learning to reason from human" approaches, the goal isn't just to learn to reason from humans, but to do it in a way that maintains competitiveness with unaligned AIs. Suppose a human overseer disapproves of their AI using some set of potentially dangerous techniques, how can we then ensure that the resulting AI is still competitive? Once someone points this out, proponents of the approach, to continue thinking their approach is promising, would need to give some details about how they intend to solve this problem. Subsequently, justification for thinking the approach is promising is more subtle and harder to understand. I think conversations like this have occurred for MIRI's approach far more than Paul's, which may be a large part of why you find Paul's justifications easier to understand.

Comment author: jsteinhardt 11 July 2017 03:55:45PM *  5 points [-]

This doesn't match my experience of why I find Paul's justifications easier to understand. In particular, I've been following MIRI since 2011, and my experience has been that I didn't find MIRI's arguments (about specific research directions) convincing in 2011*, and since then have had a lot of people try to convince me from a lot of different angles. I think pretty much all of the objections I have are ones I generated myself, or would have generated myself. Although, the one major objection I didn't generate myself is the one that I feel most applies to Paul's agenda.

( * There was a brief period shortly after reading the sequences that I found them extremely convincing, but I think I was much more credulous then than I am now. )

Comment author: Wei_Dai 09 July 2017 08:53:55AM 18 points [-]

3c. Other research, especially "learning to reason from humans," looks more promising than HRAD (75%?)

From the perspective of an observer who can only judge from what's published online, I'm worried that Paul's approach only looks more promising than MIRI's because it's less "mature", having received less scrutiny and criticism from others. I'm not sure what's happening internally in various research groups, but the amount of online discussion about Paul's approach has to be at least an order of magnitude less than what MIRI's approach has received.

(Looking at the thread cited by Rob Bensinger, various people including MIRI people have apparently looked into Paul's approach but have not written down their criticisms. I've been trying to better understand Paul's ideas myself and point out some difficulties that others may have overlooked, but this is hampered by the fact that Paul seems to be the only person who is working on the approach and can participate on the other side of the discussion.)

I think Paul's approach is certainly one of the most promising approaches we currently have, and I wish people paid more attention to it (and/or wrote down their thoughts about it more), but it seems much too early to cite it as an example of an approach that is more promising than HRAD and therefore makes MIRI's work less valuable.

Comment author: jsteinhardt 10 July 2017 03:44:18AM 8 points [-]

Shouldn't this cut both ways? Paul has also spent far fewer words justifying his approach to others, compared to MIRI.

Personally, I feel like I understand Paul's approach better than I understand MIRI's approach, despite having spent more time on the latter. I actually do have some objections to it, but I feel it is likely to be significantly useful even if (as I, obviously, expect) my objections end up having teeth.

Comment author: JoshuaFox 27 February 2017 11:06:56AM *  2 points [-]

Outreach can be valuable, although it is rare to have high-value opportunities. If you can publish, lecture or talk 1-on-1 with highly relevant audiences, then you may sway the Zeitgeist a little and so contribute towards getting donors or researchers on board.

Relevant audiences include:

  • tech moguls and other potential big donors; people who may have the potential to become or at influence those moguls.

  • researchers in relevant areas such as game theory; smart people in elite educational tracks who may have the potential to become or influence such researchers.

Comment author: jsteinhardt 28 February 2017 06:48:15AM 10 points [-]

I already mention this in my response to kbog above, but I think EAs should approach this cautiously; AI safety is already an area with a lot of noise, with a reputation for being dominated by outsiders who don't understand much about AI. I think outreach by non-experts could end up being net-negative.

Comment author: kbog  (EA Profile) 27 February 2017 06:52:45AM *  2 points [-]

What about online activism? There are lots of debates in various corners of the Internet over AI which often involve people in various areas of academia and tech. It seems like it could be valuable and feasible for people who are sufficiently educated on the basic issues of AI alignment to correct misconceptions and spread good ideas.

As another idea, there are certain kinds of information which would be worth collecting: surveys of relevant experts, taxonomies of research ideas and developments in the field, information about the political and economic sides of AI research. I suppose this could fall into gruntwork for safety orgs, but they don't comprehensively ask for every piece of information and work which could be useful.

Also - this might sound strange, but if someone wants to contribute then it's their choice: students and professionals might be more productive if they had remote personal assistants to handle various tasks which are peripheral to one's primary tasks and responsibilities, and if someone is known to be an EA, value aligned on cause priorities, and moderately familiar with the technical work, then having someone do this seems very feasible.

Comment author: jsteinhardt 28 February 2017 06:46:01AM 11 points [-]

In general I think this sort of activism has a high potential for being net negative --- AI safety already has a reputation as something mainly being pushed by outsiders who don't understand much about AI. Since I assume this advice is targeted at the "average EA" (who presumably doesn't know much about AI), this would only exacerbate the issue.

Comment author: Kerry_Vaughan 29 January 2017 08:12:35PM 7 points [-]

I don't agree with thejadedone's conclusions or think his post is particularly well-thought-out, but I don't think raising the bar on criticism like this is very productive if you care about getting good criticism. (If you think thejadedone's criticism is bad criticism, then I think it makes sense to just argue for that rather than saying that they should have made it privately.)

I agree with this and wasn't trying to say something to the contrary. What I was trying to do is note that the post makes a relatively minor issue into an expose on EA and on 80K. I think this is unnecessary and unwarranted by the issue. What is was trying to do is note one way of handling the issue if your goal is merely to gain more information or see that a problem gets fixed.

I think public criticism is fine. I think a good, but not required, practice is to show the criticism to the organization ahead of publishing it so that they can correct factual inaccuracies. I think that would have improved the criticism substantially in this case.

Comment author: jsteinhardt 30 January 2017 06:08:29AM 1 point [-]

Thanks for clarifying; your position seems reasonable to me.

Comment author: the_jaded_one 29 January 2017 01:36:47PM *  3 points [-]

One way to resolve our initial skepticism would be to have a trusted expert in the field

And in what field is Chloe Cockburn a "trusted expert"?

If we go by her twitter, we might say something like "she is an expert left-wing, highly political, anti-trump, pro-immigration activist"

Does that seem like a reasonable characterization of Chloe Cockburn's expertise to you?

Characterizing her as "Trusted" seems pretty dishonest in this context. Imagine someone who has problems with EA and Cosecha, for example because they were worried about political bias in EA. Now imagine that they got to know her opinions and leanings, e.g. on her twitter. They wouldn't "trust" her to make accurate calls about the effectiveness of donations to a left-wing, anti-Trump activist cause, because she is a left-wing anti-Trump activist. She is almost as biased as it is possible to be on this issue, the exact opposite of the kind of person whose opinion you should trust. Of course, she may have good arguments since those can come from biased people, but she is being touted as a "trusted expert", not a biased expert with strong arguments, so that is what I am responding to.

Comment author: jsteinhardt 29 January 2017 07:18:44PM 6 points [-]

OpenPhil made an extensive write-up on their decision to hire Chloe here: http://blog.givewell.org/2015/09/03/the-process-of-hiring-our-first-cause-specific-program-officer/. Presumably after reading that you have enough information to decide whether to trust her recommendations (taking into account also whatever degree of trust you have in OpenPhil). If you decide you don't trust it then that's fine, but I don't think that can function as an argument that the recommendation shouldn't have been made in the first place (many people such as myself do trust it and got substantial value out of the recommendation and of reading what Chloe has to say in general).

I feel your overall engagement here hasn't been very productive. You're mostly repeating the same point, and to the extent you make other points it feels like you're reaching for whatever counterarguments you can think of, without considering whether someone who disagreed with you would have an immediate response. The fact that you and Larks are responsible for 20 of the 32 comments on the thread is a further negative sign to me (you could probably condense the same or more information into fewer better-thought-out comments than you are currently making).

Comment author: Kerry_Vaughan 29 January 2017 12:50:09AM *  10 points [-]

It seems to me that your basic argument is something like this

1) Working on highly political topics isn't very effective 2) The charities recommended by 80K are highly political 3) Therefore, the charities recommended by 80K aren't very effective.

Maybe I've missed it, but the only support I see for 1) is this allusion to Hanson:

There are standard arguments, for example this by Robin Hanson from 10 years ago about why it is not smart or "effective" to get into these political tugs-of-war if one wants to make a genuine difference in the world.

But, the evidence from that quote and from Hanson's post, doesn't provide enough support for the premise. I don't think Hanson is saying that all cases of highly political topics are ineffective, and if he were, it seems clear that the general heuristic is nowhere near strong enough to support such a conclusion.

Instead, he's saying that we ought to be skeptical of highly political causes. Fair enough. One way to resolve our initial skepticism would be to have a trusted expert in the field recommend a highly political intervention. This is exactly what we have here.

It could be that 80K shouldn't recommend highly political charities even if they're effective. If so, that seems like a PR/communication problem which could be fixed by distancing themselves from the recommendations. They seem to have already done this, but could do it further by making it as clear as possible that they've outsourced this recommendation to OpenPhil.

The fact that they let Cosecha (and to a lesser extent The Alliance for Safety and Justice) through reduces my confidence in 80,000 hours and the EA movement as a whole. Who thought it would be a good idea to get EA into the culture war with these causes, and also thought that they were plausibly among the most effective things you can do with money? Are they taking effectiveness seriously? What does the political diversity of meetings at 80,000 hours look like? Were there no conservative altruists present in discussions surrounding The Alliance for Safety and Justice and Cosecha, and the promotion of them as "beneficial for everyone" and "effective"?

This is needlessly hyperbolic. Criminal Justice Reform is one among many causes 80K mentions as options in their research. They outsourced this recommendation to an expert in the field. Maybe they did a poor job of outsourcing this, but inferring some widespread problems seems entirely unjustified.

Instead of writing this like some kind of expose, it seems you could get the same results by emailing the 80K team, noting the political sensitivity of the topic, and suggesting that they provide some additional disclaimers about the nature of the recommendation.

So basically, Chloe likes organizing, and she likes Carlos.

I would expect significantly more detailed analysis. Why does Chloe like organizing? What type of organizing does she like? What evidence is there it work? What has Cosecha done in the past? How much money did they spend? How strong is the evidence of policy impact? How strong is the evidence for the desirability of the policies? What are the negative effects? What are the relevant counterfactuals?

@Larks: The recommendation is not intended to be a full-fledged write-up of the organization's effectiveness. It's a quick note of support from an expert in the field. We could debate whether 80K should trust this kind of quick recommendation, but asking that Chloe explores the issue in significantly more details seems unfair given the context.

So basically, Chloe likes organizing, and she likes Carlos.

This is pretty unfair. She provides quite a few lines of evidence in favor of this particular organizer. Liking him is not the root cause of recommending him here.

Disclosure: I work for CEA which houses 80K. I also know nearly everyone on the 80K team.

Comment author: jsteinhardt 29 January 2017 03:35:26AM *  10 points [-]

Instead of writing this like some kind of expose, it seems you could get the same results by emailing the 80K team, noting the political sensitivity of the topic, and suggesting that they provide some additional disclaimers about the nature of the recommendation.

I don't agree with the_jaded_one's conclusions or think his post is particularly well-thought-out, but I don't think raising the bar on criticism like this is very productive if you care about getting good criticism. (If you think the_jaded_one's criticism is bad criticism, then I think it makes sense to just argue for that rather than saying that they should have made it privately.)

My reasons are very similar to Benjamin Hoffman's reasons here.

Comment author: Brian_Tomasik 14 January 2017 05:45:08PM 5 points [-]

I'm surprised to hear that people see criticizing EA as incurring social costs. My impression was that many past criticisms of EA have been met with significant praise (e.g., Ben Kuhn's). One approach for dealing with this could be to provide a forum for anonymous posts + comments.

Comment author: jsteinhardt 14 January 2017 07:24:21PM 4 points [-]

In my post, I said

anything I write that wouldn't incur unacceptably high social costs would have to be a highly watered-down version of the original point, and/or involve so much of my time to write carefully that it wouldn't be worthwhile.

I would expect that conditioned on spending a large amount of time to write the criticism carefully, it would be met with significant praise. (This is backed up at least in upvotes by past examples of my own writing, e.g. Another Critique of Effective Altruism, The Power of Noise, and A Fervent Defense of Frequentist Statistics.)

View more: Next