Comment author: RobBensinger 27 February 2017 08:02:25PM 3 points [-]
Comment author: Sean_o_h 09 March 2017 04:31:10PM 0 points [-]

Murray will be remaining involved with CFI, albeit at reduced hours. The current intention is that there will still be a postdoc in trust/transparency/interpretability based out of Imperial, although we are looking into the possibility of having a colleague of Murray's supervising or co-supervising.

Comment author: RobBensinger 25 February 2017 11:15:11PM 4 points [-]

One of the spokes of the Leverhulme Centre for the Future of Intelligence is at Imperial College London, headed by Murray Shanahan.

Comment author: Sean_o_h 26 February 2017 04:33:19PM 5 points [-]

There will be a technical AI safety-relevant postdoc position opening up with this CFI spke shortly, looking at trust/transparency/interpretability in AI systems.

Comment author: Sean_o_h 15 December 2016 01:45:54PM 13 points [-]

Thanks for a very detailed and informative post.

As I have before, I would encourage EAs to think about supporting GCRI, whether on AI safety or (especially) GCR more broadly. (1) As you note, they've done some useful analyses on a very limited budget. (2) It's my view that a number of their members (Seth Baum and Dave Denkenberger in particular in my case) have been useful and generous information and expertise resources to this community over the last couple of years (Seth has both provided useful insights, and made very useful networking connections, for me and others in FHI and CSER re: adjacent fields that we could usefully engage with, including biosecurity, risk analysis, governance etc). (3) They've been making good efforts to get more relevant talent into the field - e.g. one of their research associates, Matthias Maas, gave one of the best-received contributed talks at our conference this week (on nuclear security and governance). (4) They're less well-positioned to secure major funding from other sources than some of the orgs above. (5) As a result of (4), they've never really had the opportunity to "show what they can do" so to speak - I'm quite curious as to what they could achieve with a bigger budget and a little more long-term security.

So I think there's an argument to be made on the grounds of funding opportunity constraints, scaleability, and exploration value. The argument is perhaps less strong on the grounds of AI safety alone, but stronger for GCR more broadly.

Comment author: Joey 08 December 2016 07:02:51PM 6 points [-]

Given all the interest in this (fairly unrelated to top post) topic I wonder if it makes sense to do a different post/survey on what would be the ideal posting frequency for EA orgs on the EA forum. I know CS would be very responsive to information on this and I suspect all the other EA orgs would be as well.

It also seems a bit hard to deal with criticism that falls along somewhat contradicting lines of a) you're not being transparent enough, I want more things like the monthly update and b) you're too spammy, I want to see less things like the monthly update. (I do know there is a difference between number of posts and given information, but limiting number of posts does make it harder).

Comment author: Sean_o_h 08 December 2016 09:05:19PM *  1 point [-]

Yes. I agree with those who have pointed out that this derailed an important CEA conversation (and regret, in hindsight, contributing to this - my apologies), but the questions Joey raises are ones that it would be v useful to have more info on, in the context of a separate discussion.

Comment author: Sean_o_h 07 December 2016 01:17:00PM *  4 points [-]

As a representative of an org (CSER, and previously FHI) who has periodically posted updates on these orgs on the EA forum and previously LW, it's very helpful to hear opinions (both positive and negative) on desirability and frequency of updates. I would be grateful for more opinions while it's under discussion.

Thank you Michael for raising the question.

Comment author: Sean_o_h 07 December 2016 01:40:16PM *  3 points [-]

If enough people feel the same as Michael, is there a case for having a forum subsection where e.g. updates/fundraising/recruitment calls for EA orgs could live?

Disadvantages I could see

  • 'branching' the forum complicates and clutters it at a point where there still isn't a huge amount of volume to support/justify such structures.

  • these updates might get less visibility.

Advantages (in addition to the forum being kept more free for discussion of EA ideas)

  • These updates would then all be clustered in one place, making it easier for a reader to do an overview of orgs without digging through the forum's history.
Comment author: MichaelDickens  (EA Profile) 07 December 2016 05:10:38AM 1 point [-]

I don't believe organizations should post fundraising documents to the EA Forum. As a quick heuristic, if all EA orgs did this, the forum would be flooded with posts like this one and it would pretty much kill the value of the forum.

It's already the case that a significant fraction of recent content is CEA or CEA-associated organizations talking about their own activities, which I don't particularly want to see on the EA Forum. I'm sure some other people will disagree but I wanted to contribute my opinion so you're aware that some people dislike these sorts of posts.

Comment author: Sean_o_h 07 December 2016 01:17:00PM *  4 points [-]

As a representative of an org (CSER, and previously FHI) who has periodically posted updates on these orgs on the EA forum and previously LW, it's very helpful to hear opinions (both positive and negative) on desirability and frequency of updates. I would be grateful for more opinions while it's under discussion.

Thank you Michael for raising the question.

Comment author: Pablo_Stafforini 30 November 2016 05:45:07PM *  5 points [-]

The claim that it's natural to donate to one's employer given one's prior decision to become an employee assumes that EAs—or at least those working for EA orgs—should spend all their altruistic resources (i.e. time and money) in the same way. But this assumption is clearly false: it can be perfectly reasonable for me to believe that I should spend my time working for some organization, and that I should spend my money supporting some other organization. Obviously, this will be the case if the organization I work for, but not the one I support, lacks room for more funding. But it can also be the case in many other situations, depending on the relative funding and talent constraints of both the organization I work for and the organizations I could financially support.

Comment author: Sean_o_h 30 November 2016 06:18:27PM 4 points [-]

And the impact of one's skillset in different orgs/focus areas.

Comment author: So8res 27 October 2016 04:26:22PM *  4 points [-]

Thanks for the response, Gregory. I was hoping to see more questions along these lines in the AMA, so I'm glad you followed up.

Open Phil's grant write-up is definitely quite critical, and not an endorsement. One of Open Phil's main criticisms of MIRI is that they don't think our agent foundations agenda is likely to be useful for AI alignment; but their reasoning behind this is complicated, and neither Open Phil nor MIRI has had time yet to write up our thoughts in any detail. I suggest pinging me to say more about this once MIRI and Open Phil have put up more write-ups on this topic, since the hope is that the write-ups will also help third parties better evaluate our research methods on their merits.

I think Open Phil's assessment that the papers they reviewed were ‘technically unimpressive’ is mainly based on the papers "Asymptotic Convergence in Online Learning with Unbounded Delays" and (to a lesser extent) "Inductive Coherence." These are technically unimpressive, in the sense that they're pretty easy results to get once you're looking for them. (The proof in "Asymptotic Convergence..." was finished in less than a week.) From my perspective the impressive step is Scott Garrabrant (the papers’ primary author) getting from the epistemic state (1) ‘I notice AIXI fails in reflection tasks, and that this failure is deep and can't be easily patched’ to:

  • (2) ‘I notice that one candidate for “the ability AIXI is missing that would fix these deep defects” is “learning mathematical theorems while respecting patterns in whether a given theorem can be used to (dis)prove other theorems.”’
  • (3) ‘I notice that another candidate for “the ability AIXI is missing that would fix these deep defects” is “learning mathematical theorems while respecting empirical patterns in whether a claim looks similar to a set of claims that turned out to be theorems.”’
  • (4) ‘I notice that the two most obvious and straightforward ways to formalize these two abilities don't let you get the other ability for free; in fact, the obvious and straightforward algorithm for the first ability precludes possessing the second ability, and vice versa.’

In contrast, I think the reviewers were mostly assessing how difficult it would be to get from 2/3/4 to a formal demonstration that there’s at least one real (albeit impractical) algorithm that can actually exhibit ability 2, and one that can exhibit ability 3. This is a reasonable question to look at, since it's a lot harder to retrospectively assess how difficult it is to come up with a semiformal insight than how difficult it is to formalize the insight; but those two papers weren't really chosen for being technically challenging or counter-intuitive. They were chosen because they help illustrate two distinct easy/straightforward approaches to LU that turned out to be hard to reconcile, and also because (speaking with the benefit of hindsight) conceptually disentangling these two kinds of approaches turned out to be one of the key insights leading to "Logical Induction."

I confess scepticism at this degree of inferential distance, particularly given the Open Phil staff involved in this report involved several people who previously worked with MIRI.

I wasn't surprised that there's a big inferential gap for most of Open Phil's technical advisors -- we haven't talked much with Chris/Dario/Jacob about the reasoning behind our research agenda. I was surprised by how big the gap was for Daniel Dewey, Open Phil's AI risk program officer. Daniel's worked with us before and has a lot of background in alignment research at FHI, and we spent significant time trying to understand each other’s views, so this was a genuine update for me about how non-obvious our heuristics are to high-caliber researchers in the field, and about how much background researchers at MIRI and FHI have in common. This led to a lot of wasted time: I did a poor job addressing Daniel's questions until late in the review process.

I'm not sure what prior probability you should have assigned to ‘the case for MIRI's research agenda is too complex to be reliably communicated in the relevant timeframe.’ Evaluating how promising basic research is for affecting the long-run trajectory of the field of AI is inherently a lot more complicated than evaluating whether AI risk is a serious issue, for example. I don't have as much experience communicating the former, so the arguments are still rough. There are a couple of other reasons MIRI's research focus might have more inferential distance than the typical alignment research project:

  • (a) We've been thinking about these problems for over a decade, so we've had time to arrive at epistemic states that depend on longer chains of reasoning. Similarly, we've had time to explore and rule out various obvious paths (that turn out to be dead ends).
  • (b) Our focus is on topics we don't expect to jibe well with academia and industry, often because they look relatively intractable and unimportant from standard POVs.
  • (c) ‘High-quality nonstandard formal intuitions’ are what we do. This is what put us ahead of the curve on understanding the AI alignment problem, and the basic case for MIRI (from the perspective of people like Holden who see our early analysis and promotion of the alignment problem as our clearest accomplishment) is that our nonstandard formal intuitions may continue to churn out correct and useful insights about AI alignment when we zero in on subproblems. MIRI and FHI were unusual enough to come up with the idea of AI alignment research in the first place, so they're likely to come up with relatively unusual approaches within AI alignment.

Based on the above, I think the lack of mutual understanding is moderately surprising rather than extremely surprising. Regardless, it’s clear that we need to do a better job communicating how we think about choosing open problems to work on.

I note the blogging is by people already in MIRI's sphere of influence/former staff, and MIRI's previous 'blockbuster result' in decision theory has thus far underwhelmed)

I don't think we've ever worked with Scott Aaronson, though we're obviously on good terms with him. Also, our approach to decision theory stirred up a lot of interest from professional decision theorists at last year's Cambridge conference; expect more about this in the next few months.

is not a promissory note that easily justifies an organization with a turnover of $2M/year, nor fundraising for over a million dollars more.

I think this is a reasonable criticism, and I'm hoping our upcoming write-ups will help address this. If your main concern is that Open Phil doesn't think our work on logical uncertainty, reflection, and decision-theoretic counterfactuals is likely to be safety-relevant, keep in mind that Open Phil gave us $500k expecting this to raise our 2016 revenue from $1.6-2 million (the amount of 2016 revenue we projected absent Open Phil's support) to $2.1-2.5 million, in part to observe the ROI of the added $500k. We've received around $384k in our fundraiser so far (with four days to go), which is maybe 35-60% of what we'd expect based on past fundraiser performance. (E.g., we received $597k in our 2014 fundraisers and $955k in our 2015 ones.) Combined with our other non-Open-Phil funding sources, that means we've so far received around $1.02M in 2016 revenue outside Open Phil, which is solidly outside the $1.6-2M range we've been planning around.

There are a lot of reasons donors might be retracting; I’d be concerned if the reason is that they're expecting Open Phil to handle MIRI's funding on their own, or that they're interpreting some action of Open Phil's as a signal that Open Phil wants broadly Open-Phil-aligned donors to scale back support for MIRI.

(In all of the above, I’m speaking only for myself; Open Phil staff and advisors don’t necessarily agree with the above, and might frame things differently.)

Comment author: Sean_o_h 27 October 2016 05:29:58PM 3 points [-]

"Also, our approach to decision theory stirred up a lot of interest from professional decision theorists at last year's Cambridge conference; expect more about this in the next few months." A quick note to say that comments that have made their way back to me from relevant circles agree with this. Also, my own impression - from within academia, but outside decision theory and AI - is that the level of recognition of, and respect for, MIRI's work is steadily rising in academia, although inferential gaps like what nate describes certainly exist, plus more generic cultural gaps. I've heard positive comments about MIRI's work from academics I wouldn't have expected even to have heard of MIRI. And my impression, from popping by things like Cambridge's MIRIx discussion group, is that they're populated for the most part by capable people with standard academic backgrounds who have become involved based on the merits of the work rather than any existing connection to MIRI (although I imagine some are or were lesswrong readers).

Comment author: Sean_o_h 14 September 2016 01:36:40PM *  9 points [-]

A personal comment (apologies that this is neither feedback nor criticism): I switched from a career plan that was pointing me towards neglected tropical disease genomics and related topics to x-risk/gcr after reading Martin Rees' Our Final Century (fun fact: I showed up at FHI without any real idea who Nick Bostrom was or why a philosopher was relevant to x-risk research).

6 years later, there's still a nagging voice in the back of my mind that worries about issues related to what you describe. (Admittedly, the voice worries more so that we don't end up doing the right work - if we do the right work but it's not the Xrisk that emerges [but was a sensible and plausible bet], or we are doing the right thing but someone else plays the critical role in averting it due to timing and other factors, I can live with that even if it technically means a wasted career and a waste of funds). I'm hoping that the portfolio of research activities I'm involved in setting up here in Cambridge is broad enough to us a good shot, directly or indirectly, of making some difference in the long run. But it's not totally clear to me I'll ever know for certain (i.e. a clean demonstration of a catastrophe that was clearly averted because of work I was involved with at CSER/CFI/FHI seems unlikely). I try to placate that voice by donating to global disease charities (e.g. SCI) despite working in xrisk.

So basically, I'm saying I empathise with these feelings. While it perhaps conflicts with some aspects of really dedicated cause prioritisation, I think a donor ecosystem in which some people are taking the long, high payoff bets, and don't mind a higher probability that their funds don't directly end up saving lives in the long run, while others are more conservative and are supporting more direct and measurable do-the-most-gooding, makes for a good overall EA 'portfolio' (and one in which the different constituents help to keep each other both open-minded and focused).

While I can't comment on whether this is selfish or narcissistic, if the end result is an ecosystem with a level of diversity in the causes it supports, that seems to be good given the level of uncertainty we have to have about the long-run importance of many of these things - provided, of course, we have high confidence that the causes within this diversity remain orders of magnitude more important than other causes we are choosing not to support (i.e. the majority of causes in the world).

Comment author: John_Maxwell_IV 11 September 2016 08:58:43AM *  23 points [-]

Organizations become dysfunctional when employees have reasons to act in a way that's incompatible with the goals of the organization. See the principal-agent problem. Aligning the incentives of employees with the goals of the organization is nontrivial. You can see this play out in the business world:

  • One reason Silicon Valley wins is because companies in Silicon Valley offer meaningful equity to their employees. Peter Thiel recommends against compensating startup CEOs with lots of cash or using consultants.

  • Y Combinator companies are most interested in hiring programmers motivated by building a great product. If you're building an app for skiers, it may be wise to skip the programming genius in favor of a solid developer who's passionate about skiing.

  • For a non-Silicon Valley example, Koch Industries has grown 2000x since 1967. Rather than using breakthrough technology, connections, or cheap capital, Charles Koch credits his success to the culture he built based on "crazy ideas drawing from my interest in the philosophy of science, the scientific method, and my studies of how people can best live and work together." On the topic of hiring, he says: "most [companies] hire first on talent and then they hope that the values are compatible with their culture... We hire first on values."

The principal-agent problem is especially relevant in large organizations. An employee's incentives are set up by their boss, so each additional layer of hierarchy is an opportunity for incentive problems to compound. (Real-life example from my last job: boss says I need to work on project x, even though we both know it's not very important for the company, because finishing it makes our department look good.) This is the stuff out of which Dilbert comics are made.

But incentives are still super important in small organizations. You'd think that a company's CEO, of all people, would be immune, but Thiel observed that high CEO salaries predict startup failure because it makes the company's culture not "equity-focused".

There are also benefits associated with having a team that's unified culturally.

Using non-EA employees seems fine when incentives are easy to set up and the work is not core to the organization's mission, e.g. ditch-digging type work.

Comment author: Sean_o_h 12 September 2016 08:28:36AM *  5 points [-]

I agree with this. Organisational culture matters a lot. I would suggest that a good strategy is hiring a mix, where there are enough people motivated by the right thing to set and maintain the culture, and those new to the culture will, for the most part, adopt it (provided the culture is based around sound principles, as EA is). This provides the benefit of (a) allowing the flexibility to hire in people with specialist skills outside current EA (b) encouraging the development of more EAs (c) providing outside perspectives that can, where appropriate, be used to improve implementation of EA principles (or refinement of principles).

(Note that what you're most likely to be looking at in many of these cases (new people) is not "people who are opposed to EA" but more likely "people who haven't previously encountered, thought deeply about, or had a lot of exposure to the best thinking/arguments around EA".

(This is obviously relevant to generic culture-building, not necessarily EA-specific)

View more: Next