Comment author: Fluttershy 08 March 2017 01:44:16PM *  2 points [-]

I'd like to respond to your description of what some people's worries about your previous proposal were, and highlight how some of those worries could be addressed, hopefully without reducing how helpfully ambitious your initial proposal was. Here goes:

the risk of losing flexibility by enforcing what is an “EA view” or not

It seems to me like the primary goal of the panel in the original proposal was to address instances of people lowering the standard of trustworthiness within EA and imposing unreasonable costs (including unreasonable time costs) on individual EAs. I suspect that enumerating what sorts of things "count" as EA endeavors isn't a strictly necessary prerequisite for forming such a panel.

I can see why some people held this concern, partly because "defining what does and doesn't count as an EA endeavor" clusters in thing-space with "keeping an eye out for people acting in untrustworthy and non-cooperative ways towards EAs", but these two things don't have to go hand in hand.

the risk of consolidating too much influence over EA in any one organisation or panel

Fair enough. As with the last point, the panel would likely consolidate less unwanted influence over EA if it focused solely on calling out sufficiently dishonestly harmful behavior by anyone who self-identified as an EA, and made no claims as to whether any individuals or organizations "counted" as EAs.

the risk of it being impossible to get agreement, leading to an increase in politicisation and squabbling

This seems like a concern that's good, in that a bit harder for me to address satisfactorily. Hopefully, though, there would some clear-cut cases the panel could choose to consider, too; the case of Intentional Insights' poor behavior was eventually quite clear, for one. I would guess that the less clear cases would tend to be the ones where a clear resolution would be less impactful.

In response, we toned back the ambitions of the proposed ideas.

I'd have likely done the same. But that's the wrong thing to do.

In this case, the counterfactual to having some sort of panel to call out behavior which causes unreasonable amounts of harm to EAs is relying on the initiative of individuals to call out such behavior. This is not a sustainable solution. Your summary of your previous post puts it well:

There’s very little to deal with people representing EA in ways that seem to be harmful; this means that the only response is community action, which is slow, unpleasant for all involved, and risks unfairness through lack of good process.

Community action is all that we had before the Intentional Insights fiasco, and community action is all that we're back to having now.

I didn't get to watch the formation of the panel you discuss, but it seems like a nontrivial amount of momentum, which was riled up by the harm Intentional Insights caused EA, went into its creation. To the extent that that momentum is no longer available because some of it was channeled into the creation of this panel, we've lost a chance at building a tool to protect ourselves against agents and organizations who would impose costs on, and harm EAs and EA overall. Pending further developments, I have lowered my opinion of everyone directly involved accordingly.

Comment author: Gregory_Lewis 09 March 2017 01:36:54PM 5 points [-]

FWIW, as someone who contributed to the InIn document, I approve of (and recommended during discussion) the less ambitious project this represents.

Comment author: Gregory_Lewis 29 January 2017 08:12:34PM *  8 points [-]

I don't see the merit of upbraiding 80k for aggregation various sources of 'EA philanthropic advice' because one element of this relies on political views one may disagree with. Not including Cockburn's recommendations whilst including all the other OpenPhil staffers also implies political views others would find disagreeable. It's also fairly clear from the introduction the post (at least for non-animal charities) was canvassing all relevant recommendations rather than editorializing.

That said, it is perhaps unwise to translate 'advice from OpenPhil staffers' into 'EA recommendations'. OpenPhil is clear about how it licenses itself to try and 'pick hits' which may involve presuming or taking a bet on a particular hot button political topic (i.e. Immigration, criminal justice, abortion), being willing to take a particular empirical bet in the face of divided expertise, and so forth. For these reasons OpenPhil are not a 'Givewell for everything else', and their staffer's recommendations, although valuable for them to share and 80k to publicise, should carry the health warning that they are often conditional on quite large and non-resilient conjunctions of complicated convictions - which may not represent 'expert consensus' on these issues.

Comment author: Gregory_Lewis 14 January 2017 04:18:08PM *  12 points [-]

I note Constantin's post, first, was extraordinary uncharitable and inflammatory (e.g. the title for the section discussing Wiblin's remark "Keeping promises as a symptom of Autism", among many others); second, these errors were part of a deliberate strategy to 'inflame people against EA'; third, this strategy is hypocritical given the authors (professed) objections to any hint of 'exploitative communication'. Any of these in isolation is regrettable. In concert they are contemptible.

{ETA: Although in a followup post Constantin states her previous comments which were suggestive of bad faith were an "emotional outburst", it did not reflect her actual intentions either at the time of writing or subsequently.}

My view is that, akin to Hostadter's law, virtues of integrity are undervalued even when people try to account for undervaluing them: for this reason I advocate all-but lexical priority to candour, integrity, etc. over immediate benefits. The degree of priority these things should be accorded seems a topic on which reasonable people can disagree: I recommend Elmore's remarks as a persuasive defence of according these virtues a lower weight.

'Lower', however, still means 'quite a lot': if I read Elmore correctly, her view is not one can sacrifice scrupulously honest communication for any non-trivial benefit, but that these norms should on occasion be relaxed if necessary to realise substantial gains. The great majority of EAs seem to view these things as extremely important, and the direction of travel appears to me that 'more respected' EAs tend to accord these even greater importance (see MacAskill; c.f. Tsipursky).

My impression is that EAs - both individually and corporately - do fairly well in practice as well as principle. As Naik notes, many orgs engage in acts of honesty and accountability supererogatory to secure funding. When they do err, they tend to be robustly challenged (often by other EAs), publicly admit their mistake, and change practice (all of the examples Constantin cites were challenged at the time, I also think of Harris's concerns with promotion of EA global, GPP's mistaken use of a Stern report statistic, and now ACE). Similar sentiments apply to an individual level: my 'anecdata' is almost the opposite of Fluttershy's extremely bad experience: I sincerely believe (and even more sincerely hope) that mine is closer to the norm than theirs.

In absolute terms, I don't think EA in toto has a 'lying problem' (or a 'being misleading', 'not being scrupulously honest' problem). It seems to do quite well at this, and the rate and severity of the mistakes I see don't cause great alarm (although it can and should do better). Although relative terms are less relevant, I think it does better than virtually any other group I can think of.

I offer some further remarks on issues raised by some of the examples given which do not fit into the 'lying problem' theme:

1) Ironic, perhaps, that the best evidence for Todd's remark on the 'costs of criticism' arise from the aftermath of a post which (in part) unjustly excoriates him for that particular remark. My impression is that bad criticism is on average much more costly than bad praise, and some asymmetry in how these are treated seem reasonable.

I do not know whether journalistic 'best practice' around 'right of reply' extends to providing the criticism in full to its subject - regardless, it seems good practice to adopt for the reasons Todd explains. I have done this with my co-contributors re. Intentional Insights, and I have run a (yet to be published) piece about MIRI by MIRI as it had some critical elements to it. Naturally, if a critic does not do this for whatever reason, it does not mean their criticism should be ignored (I have yet to see a case of criticism 'shunned' for these reasons) but I think this is a norm worth encouraging.

2) Nonetheless, it may not have been advisable for the head of one 'part' of CEA to bring this up in context of criticism addressed to another part of CEA. Issues around appropriate disclosure have been mentioned before. In addition, remarks by 'EA public figures' may be taken as indicative of the view of their organisations or EA in toto even if explicitly disclaimed as 'personal opinion only'. A regrettable corollary (as Gordon-Brown notes) is a chilling effect on 'EA public figures' refraining from making unguarded remarks publicly. The costs of not doing so may be worse: if EA grows further, we may collectively regret 'providing more ammunition' to external critics to misuse.

3) Given the social costs towards an individual critic, there may be benefit (by organisations or, better, an independent 'Grand Inquisition' collaboration) to canvass these anonymously. The commonest shared could then be explored further: this would be valuable whether they point to a common misconception or a common fault. In the meanwhile, anyone is welcome to disclose criticisms or concerns to me in confidence.

4) Certain practices could be more widely adopted by EA orgs - beyond recording predictions, a prominent 'mistakes' page (per Givewell) would be desirable, likewise scrupulous declaration of relevant conflicts of interests.

5) (I owe this to Carl Shulman). Donors could also pitch in by carefully evaluating empirical or normative claims made by particular EA organisations: Plant, Dickens, and Hoffman would all be laudable examples, and I hope both to contribute some of my own work to this genre and to encourage others to do likewise.

Comment author: Gregory_Lewis 09 January 2017 04:14:21PM 10 points [-]

Likewise, note that the organization that is mobilizing this project, Intentional Insights, has drawn some criticism in the EA movement and has publicly distanced itself from the EA movement, while still continuing to operate informed by EA concerns.

Well done for linking to the post, but the EA movement has publicly distanced itself from Intentional Insights too:

Will MacAskill (on behalf of CEA)

As documented in the Open Letter, Intentional Insights have been systematically misleading in their public communications on many occasions, have astroturfed, and have engaged in morally dubious hiring practices. But what’s been most remarkable about this affair is how little Gleb has been willing to change his actions in light of this documentation. If I had been in his position, I’d have radically revised my activities, or quit my position long ago. Making mistakes is something we all do. But ploughing ahead with your plans despite extensive, deep and well-substantiated criticism of them by many thoughtful members of the EA community — who are telling you not just that your plans are misguided but that they are actively harmful — is not ok. It’s the opposite of what effective altruism stands for.

Because of this, we want to have no association with Intentional Insights. We do not consider them a representative of EA, we do not want to have any of CEA’s images or logos (including Giving What We Can) used in any of Intentional Insights’ promotional materials; we will not give them a platform at EAG or EAGx events; and we will encourage local group leaders not to have them speak.

Ben Henry (on behalf of the EA FB group moderators):

You may have read this post by Jeff Kaufman et al on a number of concerns with Intentional Insights' operations. In light of the activities it describes and the community's feelings about InIn's behavior, the moderators would like to announce that we are planning to not accept post submissions of InIn content to this group from now on. People working with/at InIn will remain able to comment, and we will consider non-InIn posts by InIn staff on a case-by-case basis.

My view on Intentional Insights remains unchanged from my comment on the open letter. This doesn't look like a good project, but this is in any case overdetermined by the impression InIn is a very bad organisation. I strongly recommend EAs not to get involved.

Comment author: Gregory_Lewis 02 December 2016 11:29:18PM 3 points [-]

Good write-up.

I find the very long list of badly-designed studies you note in the introduction a cause for consternation, and I'm glad this has been done much better.

However, I couldn't see a power calculation in the study, nor in the pre-registration, so I worry the planned recruitment of 3000 was either plucked from the air or decided on due to budget constraint. Yet performing this calculation given an effect size you'd be interested in is generally preferable to spending money on an underpowered study (which I'm pretty sure this is).

Given the large temporal fluctuations (e.g. the large reduction in control group), the pretty modest effects, I remain sceptical - leave alone the obvious biases like social desirability etc. Another reanalysis which might reassure would be monte carlo permutation of food groups: if very few random groups show reduction in consumption to a similar magnitude as meat, great (and, of course, vice versa).

Comment author: kbog  (EA Profile) 02 December 2016 04:11:11PM -1 points [-]

Yes, I see that's what he's trying to hint at, but there's zero indication that donations have any of the same effects on professionalism that close interpersonal relationships have. The problem is not "I don't understand your argument", it's "you're alleging something out of the blue with no support." It should be clear that dating a coworker and donating to the organization are completely different issues in many relevant respects. I can easily draw up examples of behavior which don't reduce professionalism and are actually more comparable - voluntary overtime, for instance, which companies don't forbid, or people covering expenses for the organization, which is common in nonprofits and some government agencies.

Comment author: Gregory_Lewis 02 December 2016 10:55:39PM 1 point [-]

I actually regret this paragraph for the opposite reason: the risk it came across as a veiled side-swipe, or (even worse) someone might take me to be offering gratuitous commentary on their private life. (I have disclaimed accordingly).

Pablo has interpreted me correctly. I agree donations etc. are different from the others, but I aver they are similar in that they undermine 'corporate professional' type norms that larger EA groups are well-advised to adopt, if not to the same degree (FWIW, I think voluntary overtime is at least slightly worrisome for similar reasons).

Comment author: Gregory_Lewis 01 December 2016 05:32:03PM *  9 points [-]

[ETA: I somewhat regret eliding the issue with donations to other issues re. interpersonal relationships. I take the opportunity to stress it wasn't written with a particular individual/s 'in mind'.]

I broadly agree with Owen. Beyond the considerations already mentioned, I prefer increasing professionalisation of those working in (at least the larger) EA orgs, and norms against salary sacrifice etc. I think would help this desirable direction of travel.

In most corporations (and most charities), including very high performing ones (e.g. Google, Deepmind, GiveDirectly, etc.) situations where employees are sacrificing considerable fractions of their salary, living in long term shared housing with their colleagues, or can name multiple fellow employees among their past or current sexual partners are rare. These arrangements are less surprising for smaller groups (e.g. the model of the start-up in the garage), but much more so when you have formal structures, a board, and turnover in the 6-7 figures.

I don't claim expertise here, but I'd suspect there's a reason for the ubiquity of 'corporate' structures, oft-maligned as they are, and I'd venture it has something to do with ensuring clarity and accuracy of decision making. It is better that, insofar as possible, corporate decisions are not interleaved with pecuniary, interpersonal, or bedroom issues. Similarly, norms that retard development of conflicts of interest like these are preferable to relying on staff to navigate them appropriately.

The failure modes are manifold. The hypothetical challenges around firing someone who is, in addition to one's subordinate, a housemate, ex, and current partner of another staff member need not be explicated. Although I do not want to regurgitate the unhappy episode around Intentional Insights, there were records of virtual assistants offering to 'donate' money to Intentional Insights, the net result of which was to reduce what InIn owed them and had been late paying. I doubt any EA org has or will do anything similar, but it illustrates the risks of a parallel financial transaction alongside salary which could be used to bypass proper process (some possibilities: maybe I'm worried you might fire me for poor performance, so I increase my salary sacrifice by way of cutting my price; I fail to negotiate a raise I think I deserve from you, so I simply reduce my salary sacrifice/donation by the requisite amount (which annoys you); I don't really take your standards re. dress, punctuality etc. seriously as although I am in principle getting paid £X, de facto you're paying me barely minimum wage etc. etc.)

I'd generally prefer EA staff get paid close to market, and there's a norm discouraging donations to ones employer.

Comment author: Kathy 25 October 2016 01:50:47PM *  3 points [-]

I suspect the reason InIn's quality is low is because, given their reputation disadvantage, they cannot attract and motivate the best writers and volunteers. I strongly relate to your concerns about the damage that could be done if InIn does not improve. I have severely limited my own involvement with InIn because of the same things you describe. My largest time contribution by far has been in giving InIn feedback about reputation problems and general quality. A while back, I felt demoralized with the problems, myself, and decided to focus more on other things instead. That Gleb is getting so much attention for these problems right now has potential to be constructive.

Gleb can't improve InIn until he really understands the problem that's going on. I think this is why Intentional Insights has been resistant to change. I hope I provided enough insight in my comment about social status instincts for it to be possible for us all to overcome the inferential distance.

I'm glad to see that so many people have come together to give Gleb feedback on this. It's not just me trying to get through to him by myself anymore. I think it's possible for InIn to improve up to standards with enough feedback and a lot of work on Gleb's part. I mean, that is a lot of work for Gleb, but given what I've seen of his interest in self-improvement and his level of dedication to InIn, I believe Gleb is willing to go through all of that and do whatever it takes.

Really understanding what has gone wrong with Intentional Insights is hard, and it will probably take him months. After he understands the problems better, he will need a new plan for the organization. All of that is a lot of work. It will take a lot of time.

I think Gleb is probably willing to do it. This is a man who has a tattoo of Intentional Insights on his forearm. Because I believe Gleb would probably do just about anything to make it work, I would like to suggest an intervention.

In other words, perhaps we should ask him to take a break from promoting Intentional Insights for a while in order to do a bunch of self-improvement, make his major updates and plan out a major version upgrade for Intentional Insights.

Perhaps I didn't get the memo, but I don't think we've tried organizing in order to demand specific constructive actions first before talking about shutting down Intentional Insights and/or driving Gleb out of the EA movement.

The world does need an org that promotes rationality to a broader audience... and rationalists aren't exactly known for having super people skills... Since Gleb is so dedicated and is willing to work really hard, and since we've all finally organized in public to do something about this, maybe we aught to try using this new source of leverage to heave him onto the right track.

Comment author: Gregory_Lewis 30 October 2016 08:05:08AM 7 points [-]

Hello Kathy,

I have read your replies on various comment threads on this post. If you'll forgive the summary, your view is that Tsipursky's behaviour may arise from some non-malicious shortcomings he has, and that, with some help, these can be mitigated, thus leading InIn to behave better and do more good. In medicalese, I'm uncertain of the diagnosis, strongly doubt the efficacy of the proposed management plan, and I anticipate a bleak prognosis. As I recommend generally, I think your time and laudable energy is better spent elsewhere.

A lot of the subsequent discussion has looked at whether Tsipursky's behaviour is malicious or not. I'd guess in large part it is not: deep incompetence combined with being self-serving and biased towards ones org to succeed probably explain most of it - regrettably, Tsipursky's response to this post (e.g. trumped-up accusations against Jeff and Michelle, pre-emptive threats if his replies are downvoted, veiled hints at 'wouldn't it be bad if someone in my position started railing against EA', etc.) seem to fit well with malice.

Yet this is fairly irrelevant. Tsipursky is multiply incompetant: at creating good content, at generating interest in his org (i.e. almost all of its social media reach is ilusory), at understanding the appropriate ambit for promotional efforts, at not making misleading statements, and at changing bad behaviour. I am confident that any EA I know in a similar position would not have performed as badly. I highly doubt this can all be traced back to a single easy-to-fix flaw. Furthermore, I understand multiple people approached Tsipursky multiple times about these issues; the post documents problems occurring over a number of months. The outside view is not favourable to yet further efforts.

In any case, InIn's trajectory in the EA community is probably fairly set at this point. As I write this, InIn is banned from the FB group, CEA has officially disavowed it, InIn seems to have lost donors and prospective donations from EAs, and my barometer of 'EA public opinion' is that almost all EAs who know of InIn and Tsipursky have very adverse attitudes towards both. Given the understandable reticience of EAs towards corporate action like this, one can anticipate these decisions have considerable inertia. A nigh-Damascene conversion of Tsipursky and InIn would be required for these things to begin to move favourably to InIn again.

In light of all this, attempting to 'reform InIn' now seems almost as ill-starred as trying to reform a mismanaged version of homeopaths without borders: such a transformation is required to be surely worth starting afresh. The opportunity cost is also substantial as there are other better performing EA outreach orgs (i.e. all of them), which promise far greater returns on the margin for basically any return one migh be interested in. Please help them out instead.

Comment author: Gregory_Lewis 22 October 2016 06:58:07PM *  4 points [-]

Many thanks for the reply, Rob, and apologies for missing the AMA - although this discussion may work better in this thread anyway.

Respectfully, my reading of the Open Phil report suggests it is more broadly adverse than you suggest: in broad strokes, the worries are 1) That the research MIRI is undertaking probably isn't that helpful at improving AI risk; and 2) The research output MIRI has made along these lines is in any case unimpressive. I am sympathetic to both lines of criticism, but I am more worried by the latter than the former: AI risk is famously recondrite, thus diversity of approaches seems desirable.

Some elements of Open Phil's remarks on the latter concern seem harsh to me - in particular the remark that the suite of papers presented would be equivalent to 1-3 year's work from an unsupervised grad student is inapposite given selection, and especially given the heartening progress of papers being presented at UAI (although one of these is by Armstrong, who I gather is principally supported by FHI).

Yet others are frankly concerning. It is worrying that many of the papers produced by MIRI were considered unimpressive. It is even more worrying that despite the considerable efforts Open Phil made to review MIRI's efficacy - comissioning academics to review, having someone spend a hundred hours looking at them, etc. - they remain unconvinced of the quality of your work. That they emphasize fairly research-independent considerations in offering a limited grant (e.g. involvement in review process, germinating SPARC, hedging against uncertainty of approaches) is hardly a ringing endorsement; that they expressly benchmark MIRI's research quality as less than a higher end academic grantee likewise; comparison to other grants Open Phil have made in the AI space (e.g. 1.1M to FLI, 5.5M for a new center at UC Berkeley) even more so.

It has been remarked on this forum before MIRI is a challenging organisation to evaluate as the output (technical research in computer science) is opaque to most without a particular quantitative background. MIRI's predictions and responses to Open Phil implies a more extreme position: even domain experts are unlikely to appreciate the value of MIRI's work without a considerable back-and-forth with MIRI itself. I confess scepticism at this degree of inferential distance, particularly given the Open Phil staff involved in this report involved several people who previously worked with MIRI.

I accept MIRI may not be targetting conventional metrics of research success (e.g. academic publications). Yet across most proxy indicators (e.g. industry involvement, academic endorsement, collaboration) for MIRI 'doing good research', the evidence remains pretty thin on the ground - and, as covered above, direct assessment of research quality by domain experts is mixed at best. I look forward to the balance of evidence shifting favourably: the new conference papers are promising, ditto the buzz around logical induction (although I note the blogging is by people already in MIRI's sphere of influence/former staff, and MIRI's previous 'blockbuster result' in decision theory has thus far underwhelmed). Yet this hope, alongside the earnest assurances of MIRI that - if only experts gave them the time - they would be persuaded of their value, is not a promissory note that easily justifies an organisation with a turnover of $2M/year, nor fundraising for over a million dollars more.

Comment author: Gregory_Lewis 27 October 2016 07:22:42PM 5 points [-]

I take this opportunity to note I have made an even-odds bet with Carl Shulman for $1000, donated to the charity of the winner's choice over whether Open Phil's next review of MIRI has a more favourable evaluation of their research.

Comment author: So8res 27 October 2016 04:26:22PM *  4 points [-]

Thanks for the response, Gregory. I was hoping to see more questions along these lines in the AMA, so I'm glad you followed up.

Open Phil's grant write-up is definitely quite critical, and not an endorsement. One of Open Phil's main criticisms of MIRI is that they don't think our agent foundations agenda is likely to be useful for AI alignment; but their reasoning behind this is complicated, and neither Open Phil nor MIRI has had time yet to write up our thoughts in any detail. I suggest pinging me to say more about this once MIRI and Open Phil have put up more write-ups on this topic, since the hope is that the write-ups will also help third parties better evaluate our research methods on their merits.

I think Open Phil's assessment that the papers they reviewed were ‘technically unimpressive’ is mainly based on the papers "Asymptotic Convergence in Online Learning with Unbounded Delays" and (to a lesser extent) "Inductive Coherence." These are technically unimpressive, in the sense that they're pretty easy results to get once you're looking for them. (The proof in "Asymptotic Convergence..." was finished in less than a week.) From my perspective the impressive step is Scott Garrabrant (the papers’ primary author) getting from the epistemic state (1) ‘I notice AIXI fails in reflection tasks, and that this failure is deep and can't be easily patched’ to:

  • (2) ‘I notice that one candidate for “the ability AIXI is missing that would fix these deep defects” is “learning mathematical theorems while respecting patterns in whether a given theorem can be used to (dis)prove other theorems.”’
  • (3) ‘I notice that another candidate for “the ability AIXI is missing that would fix these deep defects” is “learning mathematical theorems while respecting empirical patterns in whether a claim looks similar to a set of claims that turned out to be theorems.”’
  • (4) ‘I notice that the two most obvious and straightforward ways to formalize these two abilities don't let you get the other ability for free; in fact, the obvious and straightforward algorithm for the first ability precludes possessing the second ability, and vice versa.’

In contrast, I think the reviewers were mostly assessing how difficult it would be to get from 2/3/4 to a formal demonstration that there’s at least one real (albeit impractical) algorithm that can actually exhibit ability 2, and one that can exhibit ability 3. This is a reasonable question to look at, since it's a lot harder to retrospectively assess how difficult it is to come up with a semiformal insight than how difficult it is to formalize the insight; but those two papers weren't really chosen for being technically challenging or counter-intuitive. They were chosen because they help illustrate two distinct easy/straightforward approaches to LU that turned out to be hard to reconcile, and also because (speaking with the benefit of hindsight) conceptually disentangling these two kinds of approaches turned out to be one of the key insights leading to "Logical Induction."

I confess scepticism at this degree of inferential distance, particularly given the Open Phil staff involved in this report involved several people who previously worked with MIRI.

I wasn't surprised that there's a big inferential gap for most of Open Phil's technical advisors -- we haven't talked much with Chris/Dario/Jacob about the reasoning behind our research agenda. I was surprised by how big the gap was for Daniel Dewey, Open Phil's AI risk program officer. Daniel's worked with us before and has a lot of background in alignment research at FHI, and we spent significant time trying to understand each other’s views, so this was a genuine update for me about how non-obvious our heuristics are to high-caliber researchers in the field, and about how much background researchers at MIRI and FHI have in common. This led to a lot of wasted time: I did a poor job addressing Daniel's questions until late in the review process.

I'm not sure what prior probability you should have assigned to ‘the case for MIRI's research agenda is too complex to be reliably communicated in the relevant timeframe.’ Evaluating how promising basic research is for affecting the long-run trajectory of the field of AI is inherently a lot more complicated than evaluating whether AI risk is a serious issue, for example. I don't have as much experience communicating the former, so the arguments are still rough. There are a couple of other reasons MIRI's research focus might have more inferential distance than the typical alignment research project:

  • (a) We've been thinking about these problems for over a decade, so we've had time to arrive at epistemic states that depend on longer chains of reasoning. Similarly, we've had time to explore and rule out various obvious paths (that turn out to be dead ends).
  • (b) Our focus is on topics we don't expect to jibe well with academia and industry, often because they look relatively intractable and unimportant from standard POVs.
  • (c) ‘High-quality nonstandard formal intuitions’ are what we do. This is what put us ahead of the curve on understanding the AI alignment problem, and the basic case for MIRI (from the perspective of people like Holden who see our early analysis and promotion of the alignment problem as our clearest accomplishment) is that our nonstandard formal intuitions may continue to churn out correct and useful insights about AI alignment when we zero in on subproblems. MIRI and FHI were unusual enough to come up with the idea of AI alignment research in the first place, so they're likely to come up with relatively unusual approaches within AI alignment.

Based on the above, I think the lack of mutual understanding is moderately surprising rather than extremely surprising. Regardless, it’s clear that we need to do a better job communicating how we think about choosing open problems to work on.

I note the blogging is by people already in MIRI's sphere of influence/former staff, and MIRI's previous 'blockbuster result' in decision theory has thus far underwhelmed)

I don't think we've ever worked with Scott Aaronson, though we're obviously on good terms with him. Also, our approach to decision theory stirred up a lot of interest from professional decision theorists at last year's Cambridge conference; expect more about this in the next few months.

is not a promissory note that easily justifies an organization with a turnover of $2M/year, nor fundraising for over a million dollars more.

I think this is a reasonable criticism, and I'm hoping our upcoming write-ups will help address this. If your main concern is that Open Phil doesn't think our work on logical uncertainty, reflection, and decision-theoretic counterfactuals is likely to be safety-relevant, keep in mind that Open Phil gave us $500k expecting this to raise our 2016 revenue from $1.6-2 million (the amount of 2016 revenue we projected absent Open Phil's support) to $2.1-2.5 million, in part to observe the ROI of the added $500k. We've received around $384k in our fundraiser so far (with four days to go), which is maybe 35-60% of what we'd expect based on past fundraiser performance. (E.g., we received $597k in our 2014 fundraisers and $955k in our 2015 ones.) Combined with our other non-Open-Phil funding sources, that means we've so far received around $1.02M in 2016 revenue outside Open Phil, which is solidly outside the $1.6-2M range we've been planning around.

There are a lot of reasons donors might be retracting; I’d be concerned if the reason is that they're expecting Open Phil to handle MIRI's funding on their own, or that they're interpreting some action of Open Phil's as a signal that Open Phil wants broadly Open-Phil-aligned donors to scale back support for MIRI.

(In all of the above, I’m speaking only for myself; Open Phil staff and advisors don’t necessarily agree with the above, and might frame things differently.)

Comment author: Gregory_Lewis 27 October 2016 07:20:09PM *  2 points [-]

Nate, my thanks for your reply. I regret I may not have expressed myself well enough for your reply to precisely target the worries I expressed; I also regret insofar as you reply overcomes my poor expression, it make my worries grow deeper.

If I read your approach to the Open Phil review correctly, you submitted some of the more technically unimpressive papers for review because they demonstrated the lead author developing some interesting ideas for research direction, and that they in some sense lead up to the 'big result' (Logical Induction). If so, this looks like a pretty surprising error: one of the standard worries facing MIRI given its fairly slender publication record is the technical quality of the work, and it seemed pretty clear that was the objective behind sending them out for evaluation. Under whatever constraints Open Phil provided, I'd have sent the 'best by academic lights' papers I had.

In candour, I think 'MIRI barking up the wrong tree' and/or (worse) 'MIRI not doing that much good research)' is a much better explanation for what is going on than 'inferential distance'. I struggle to imagine a fairer (or more propitious-to-MIRI) hearing than the Open Phil review: it involved two people (Dewey and Christiano) who previously worked with you guys, Dewey spent over 100 hours trying to understand the value of your work, they comissioned external experts in the field to review your work.

Suggesting that the fairly adverse review that results may be a product of lack of understanding makes MIRI seem more like a mystical tradition than a research group. If MIRI is unable to convince someone like Dewey, the prospects of it making the necessary collaborations or partnerships with the wider AI community look grim.

I don't think we've ever worked with Scott Aaronson, though we're obviously on good terms with him. Also, our approach to decision theory stirred up a lot of interest from professional decision theorists at last year's Cambridge conference; expect more about this in the next few months.

I had Aaronson down as within MIRI's sphere of influence, but if I overstate I apologize (I am correct in that Yuan previously worked for you, right?)

I look forward to seeing MIRI producing or germinating some concrete results in decision theory. The 'underwhelming blockbuster' I referred to above was the TDT/UDT etc. stuff, which MIRI widely hyped but has since then languised in obscurity.

There are a lot of reasons donors might be retracting; I’d be concerned if the reason is that they're expecting Open Phil to handle MIRI's funding on their own, or that they're interpreting some action of Open Phil's as a signal that Open Phil wants broadly Open-Phil-aligned donors to scale back support for MIRI.

It may simply be the usual (albeit regrettable) trait of donors jockeying to be 'last resort' - I guess it would depend what the usual distribution of donations are with respect to fundraising deadlines.

If donors are retracting, I would speculate Open Phil's report may be implicated. One potential model would be donors interpreting Open Phil's fairly critical support to be an argument against funding further growth by MIRI, thus pulling back so MIRIs overall revenue hovers at previous year levels (I don't read in the Open Phil a report a particular revenus target they wanted you guys to have). Perhaps a simpler explanation would be having a large and respected org do a fairly in depth review and give a fairly mixed review makes previously enthusiastic donors update to be more tepid, and perhaps direct their donations to other players in the AI space.

With respect, I doubt I will change my mind due to MIRI giving further write-ups, and if donors are pulling back in part 'due to' Open Phil, I doubt it will change their minds either. It may be that 'High quality non-standard formal insights' is what you guys do, but the value of that is pretty illegible on its own: it needs to be converted into tangible accomplishments (e.g. good papers, esteem from others in the field, interactions in industry) first to convince people there is actually something there, but also as this probably the plausible route to this comparative advantage having any impact.

Thus far this has not happened to a degree commensurate with MIRI's funding base. I wrote four-and-a-half years ago that I was disappointed in MIRI's lack of tangible accomplishments: I am even more disappointed that I find my remarks now follow fairly similar lines. Happily it can be fixed - if the logical induction result 'takes off' as I infer you guys hope it does, it will likely fix itself. Unless and until then, I remain sceptical about MIRI's value.

View more: Next