R

RyanCarey

9102 karmaJoined Aug 2014

Bio

Researching Causality and Safe AI at Oxford

Previously, founder (with help from Trike Apps) of the EA Forum.

Discussing research etc at https://twitter.com/ryancareyai.

Comments
1248

Topic contributions
6

There is one caveat: if someone acting on behalf on an EA organisation truly did something wrong which contributed to this fraud, then obviously we need to investigate that. But I am not aware of any evidence to suggest that happened. 

I tend to think EA did. Back in September 2023, I argued:

EA contributed to a vast financial fraud, through its:

  • People. SBF was the best-known EA, and one of the earliest 1%. FTX’s leadership was mostly EAs. FTXFF was overwhelmingly run by EAs, including EA’s main leader, and another intellectual leader of EA. 
  • Resources. FTX had some EA staff and was funded by EA investors.
  • PR. SBF’s EA-oriented philosophy on giving, and purported frugality served as cover for his unethical nature.
  • Ideology. SBF apparently had an RB ideology, as a risk-neutral act-utilitarian, who argued a decade ago why stealing was not in-principle wrong, on Felicifia. In my view, his ideology, at least as he professed it, could best be understood as an extremist variant of EA.

Of course, you can argue that contributing (point 1) people-time and (2) resources is consistent with us having just been victims, although I think that glosses over the extent to which EA folks at FTX had bought into Sam's vision, and folks at FTXFF might have more mildly lapsed in judgment. And we could regard (3) the PR issue as minor. But even so, (4) the ideology is important. FTX wasn't just any scam. It was one that a mostly-EA group was motivated to commit, to some degree or other, based on EA-style/consequentialist reasoning. There were several other instances of crypto-related crimes in and around the EA community. And the FTX implosion shared some characteristics with those events, and with other EA scandals. As I argued:

Other EA scandals, similarly, often involve multiple of these elements:

[Person #1]: past sexual harassment issues, later reputation management including Wiki warring and misleading histories. (norm-violation, naive conseq.)
[Person #2]: sexual harassment (norm-violation? naive conseq?)
[Person #3] [Person #4] [Person #5]: three more instances of crypto crimes (scope sensitivity? Norm-violation, naive conseq.? naivete?)
Intentional Insights: aggressive PR campaigns (norm-violation, naive conseq., naivete?)
Leverage Research, including partial takeover of CEA (risk appetite, norm-violation, naive conseq, unilateralism, naivete)
(We’ve seen major examples of sexual misbehaviour and crypto crimes in the rationalist community too.)

You could argue still that some of these elements are things that are shared with all financial crime. But then why have EAs committed >10% of the largest financial frauds of all-time, while consisting of about one millionth of the world's population, and less than 0.1% and perhaps 0.01% of its startups? You can suppose that we were just unlucky, but I don't find this particularly convincing.

I think that at this point, you should want to concede that EA appears to have contributed to FTX in quite a number of ways, and not all of them can be dismissed easily. That's why I think a more thorough investigation is needed.

As for PR, I simply think that shouldn't be the primary focus, and that it far from the most important consideration on the current margin. First, we need to get the facts in order. And then we need to describe the strategy. And then based on what kind of future EA deserves to have, we could decide how and whether to try to defend its image.

Their suggestions are relatively abstract, but you might consider reading Katja and Robin on the general topic of whether to focus on contributing money vs other things when you're young.

Yes, that's who I meant when I said "those working for the FTX Future Fund"

This is who I thought would be responsible too, along with the CEO of CEA, that they report to, (and those working for the FTX Future Fund, although their conflictedness means they can't give an unbiased evaluation). But since the FTX catastrophe, the community health team has apparently broadened their mandate to include "epistemic health" and "Special Projects", rather than narrowing it to focus just on catastrophic risks to the community, which would seem to make EA less resilient in one regard, than it was before.

Of course I'm not necessarily saying that it was possible to put the pieces together ahead of time, just that if there was one group responsible for trying, they were it.

^In summarising Why They Do It, Will says that usually, that most fraudsters aren't just "bad apples" or doing "cost-benefit analysis" on their risk of being punished. Rather, they fail to "conceptualise what they're doing as fraud". And that may well be true on average, but we know quite a lot about the details of this case, which I believe point us in a different direction.

In this case, the other defendants have said they knew what they're doing was wrong, that they were misappropriating customers' assets, and investing them. That weighs somewhat against the misconceptualisation hypothesis, albeit without ruling it out as a contributing factor.

On the other hand, we have some support for the bad apples idea. SBF has said:

In a lot of ways I don't really have a soul. This is a lot more obvious in some contexts than others. But in the end there's a pretty decent argument that my empathy is fake, my feelings are fake, my facial reactions are fake.

So I agree with Spencer, that SBF was at least deficient in affective experience, whether or not he was psychopathic.

Regarding cost-benefit analysis, I would tend to agree with Will that it's unlikely that SBF and company made a detailed calculation of the costs and benefits of their actions (and clearly they calculated incorrectly if they did), although the perceived costs and benefits could also be a contributing factor.

So based on the specific knowledge of the case, I think that the bad apples hypothesis makes more sense than the cost-benefit hypothesis and misconceptualisation hypotheses.

There is also a fourth category worth considering - whether SBF's views on side constraints were a likely factor - and I think overwhelmingly yes. Sure, as Will points out, SBF may have commented approvingly about a recent article on side constraints. But more recently, he referred to ethics as "this dumb game we woke Westerners play where we say all say the right shibboleths and so everyone likes us." Furthermore, if we're doing Facebook archaeology, we should also consider his earlier writing. In May 2012, SBF wrote about the idea of stealing to give:

I'm not sure I understand what the paradox is here. Fundamentally if you are going to donate the money to [The Humane League] and he's going to buy lots of cigarettes with it it's clearly in an act utilitarian's interest to keep the money as long as this doesn't have consequences down the road, so you won't actually give it to him if he drives you. He might predict this and thus not give you the ride, but then your mistake was letting Paul know that you're an act utilitarian, not in being one. Perhaps this was because you've done this before, but then not giving him money the previous time was possibly not the correct decision according to act utilitarianism, because although you can do better things with the money than he can, you might run in to problems later if you keep in. Similarly, I could go around stealing money from people because I can spend the money in a more utilitarian way than they can, but that wouldn't be the utilitarian thing to do because I was leaving out of my calculation the fact that I may end up in jail if I do so.

... 

As others have said, I completely agree that in practice following rules can be a good idea. Even though stealing might sometimes be justified in the abstract, in practice it basically never is because it breaks a rule that society cares a lot about and so comes with lots of consequences like jail. That being said, I think that you should, in the end, be an act utilitarian, even if you often think like a rule utilitarian; here what you're doing is basically saying that society puts up disincentives for braking rules and those should be included in the act utilitarian calculation, but sometimes they're big enough that a rule utilitarian calculation approximates it pretty well in a much simpler fashion.

I'm sure people will interpret this passage in different ways. But it's clear that, at least at this point in time, he was a pretty extreme act utilitarian.

Taking this and other information on balance, it seems clear in retrospect that a major factor is that SBF didn't take side constraints that seriously.

Of course, most of this information wasn't available or wasn't salient in 2022, so I'm not claiming that we should have necessarily worried based on it. Nor am I implying that improved governance is not a part of the solution. Those are further questions.

There is also the theoretical possibility of disbursing a larger number of $ per hour of staff capacity.

I think you can get closer to dissolving this problem by considering why you're assigning credit. Often, we're assigning some kind of finite financial rewards. 

Imagine that a group of n people have all jointly created $1 of value in the world, and that if any one of them did not participate, there would only be $0 units of value. Clearly, we can't give $1 to all of them, because then we would be paying $n to reward an event that only created $0 of value, which is inefficient. If, however, only the first guy (i=1) is an "agent" that responds to incentives, while the others (1<=i<=n) are "environment" whose behaviour is unresponsive to incentives, then it is fine to give the first guy a reward of $1.

This is how you can ground the idea that agents who cooperate should share their praise (something like a Shapley Value approach), whereas rival agents who don't buy into your reward scheme should be left out of the shapley calculation.

Answer by RyanCareyApr 08, 202417
6
1

Hi Polkashell,

There are indeed questionable people in EA, as in all communities. EA may be worse in some ways, because of its utilitarian bent, and because many of the best EAs have left the community in the last couple of years.

I think it's common in EA for people to:

  • have high hopes in EA, and have them be dashed, when their preferred project is defunded, when a scandal breaks, and so on. 
  • burn out, after they give a lot of effort to a project. 

What can make such events more traumatic is if EA has become the source of their livelihood, meaning, friendships, etc., i.e. their whole life. 

I think this risk can be reduced by expecting less from EA, and being less invested in it.

The fact that you're already noticing discomfort with your local group suggests that it might be good to step away from EA, or at least hedge your bets in some way. That does not necessarily mean shying away from a cause area X just because there are some EA assholes in it. There are assholes everywhere, after all. But rather to figure out what kind of work, and what kind of life makes sense for you, rather than just from an EA perspective. Also, to maintain connections and support structures outside of EA.

I hope that helps

Julia tells me "I would say I listed it as a possible project rather than calling for it exactly."]

It actually was not just neutrally listed as a "possible" project, because it was the fourth bullet point under "Projects and programs we’d like to see" here.

It may not be worth becoming a research lead under many worldviews. 

I'm with you on almost all of your essay, regarding the advantages of a PhD, and the need for more research leads in AIS, but I would raise another kind of issue - there are not very many career options for a research lead in AIS at present. After a PhD, you could pursue:

  1. Big RFPs. But most RFPs from large funders have a narrow focus area - currently it tends to be prosaic ML, safety, and mechanistic interpretability. And having to submit to grantmakers' research direction somewhat defeats the purpose of being a research lead.
  2. Joining an org working on an adjacent research direction. But they may not exist, depending on what you're excited to work on.
  3. Academia. But you have to be willing to travel, teach a lot, and live on well below the salary for a research contributor.
  4. Little funders (like LTFF). But they may take 3+ months to apply for, and only last a year at a time, and they won't respond to your emails for an explanation of this.
  5. Get hired by as a researcher at OpenPhil? But very few will be hired and given research autonomy here.

For a many research leads, these options won't be very attractive, and I find it hard to feel positive about convincing people to become research leads until better opportunities are in place. What would make me excited? I think we should have:

A. Research agenda agnostic RFPs. There needs to be some way for experienced AI safety researchers to figure out whether AI safety is actually a viable long-term career for them. Currently, there's no way to get OpenPhil's opinion on this - you simply have to wait years until they notice you. But there aren't very many AI safety researchers, and there should be a way for them to run this test so that they can decide which way to direct their lives.

Concrete proposal: OpenPhil should say "we want applications from AIS researchers who we might be excited about as individuals, even if we don't find their research exciting" and should start an RFP along these lines.

B. MLGA (Make LTFF great again). I'm not asking much here, but they should be faster, be calibrated on their timelines, respond to email in case of delays, offer multi-year grants.

Concrete proposal: LTFF should say "we want to fund people for multiple years at a time, and we will resign if we can't get our grantmaking process work properly

C. At least one truly research agenda-agnostic research organisation, that will hire research leads to pursue their own research interests.

Concrete proposal: Folks should found an academic department-style research organisation that hires research leads, gets them office space and visas, and gives them a little support to apply for grants to support their teams. Of course this requires a level of interest from OpenPhil and other grantmakers in supporting this organisation. 

Finally, I conclude on a personal note. As Adam knows, and other readers may deduce, I myself am a research lead underwhelmed with options (1-5). I would like to fix C (or A-B) and am excited to talk about ways of achieving this, but a big part of me just wants to leave AIS for a while, as these options are so much stronger, from a selfish perspective. Given that AIS has been this way for years, I suspect many others might leave before these issues are fixed.

Load more