This is in response to Sarah Constantin's recent post about intellectual dishonesty within the EA community.
I roughly agree with Sarah's main object level points, but I think this essay doesn't sufficiently embody the spirit of cooperative discourse it's trying to promote. I have a lot of thoughts here, but they are building off a few existing essays. (There's been a recent revival over on Less Wrong attempting to make it a better locus for high quality discussion. I don't know if it's especially succeeded, but I think the concepts behind that intended revival and very important)
- Why Our Kind Can't Cooperate (Eliezer Yudkowsky)
- A Return to Discussion (Sarah Constantin)
- The Importance of [Less Wrong, OR another Single Conversational Locus] (Emphasis mine) (Anna Salamon)
- The Four Layers of Intellectual Conversation (Eliezer Yudkowsky)
I think it's important to have all three concepts in context before delving into: - EA has a lying problem (Sarah Constantin)
I recommend reading all of those. But here's a rough summary of what I consider the important bits. (If you want to actually argue with these bits, please read the actual essays before doing so, so you're engaging with the full substance of the idea)
- Intellectuals and contrarians love to argue and nitpick. This is valuable - it produces novel insights, and keeps us honest. BUT it makes it harder to actually work together to achieve things. We need to understand how working-together works on a deep enough level that we can do so without turning into another random institution that's lost it's purpose. (See Why Our Kind... for more)
- Lately, people have tended to talk on social media (Facebook, Tumblr, etc) rather than in formal blogs or forums that encourage longform discussion. This has a few effects. (See A Return to Discussion for more)
- FB discussion is fragmented - it's hard to find everything that's been said on a topic. (And tumblr is even worse)
- It's hard to know whether OTHER people have read a given thing on a topic.
- A related point (not necessarily in "A Return to Discussion" is that social media incentives some of the worst kinda of discussion. People share things quickly, without reflection. People read and respond to things in 5-10 minute bursts, without having time to fully digest them.
- Having a single, long form discussion area that you can expect everyone in an intellectual community to have read, makes it much easier to building knowledge. (And most of human progress is due, not to humans being smart, but being able to stand on the shoulders of giants). Anna Salamon's "Importance of a Single Conversational Locus" is framed around x-risk, but I think it applies to all aspects of EA: the problems the world faces are so huge that they need a higher caliber of thinking and knowledge-building than we currently have in order to solve.
- In order to make true intellectual progress, you need people to be able to make critiques. You also need those critics to expect their criticism to in turn be criticized, so that the criticism is high quality. If a critique turns out to be poorly thought out, we need shared, common knowledge of that so that people don't end up rehashing the same debates.
- And finally, (one of) Sarah's points in "EA has a lying problem" is that, in order to be different from other movements and succeed where they failed, EA needs to hold itself to a higher standard than usual. There's been much criticism of, say, Intentional Insights for doing sketchy, truth-bendy things to gain prestige and power. But that plenty of "high status" people within the EA community do things that are similar, even if to a different degree. We need to be aware of that.
I would not argue as strongly as Sarah does that we shouldn't do it at all, but it's worth periodically calling each other out on it.
Cooperative Epistemology
So my biggest point here, is that we need to be more proactive and mindful about how discussion and knowledge is built upon within the EA community.
To succeed at our goals:
- EA needs to hold itself to a very high intellectual standard (higher than we currently have, probably. In some sense anyway)
- Factions within EA needs to be able to cooperate, share knowledge. Both object level knowledge (i.e. how cost effective is AMF?) and meta/epistemic knowledge like:
- How do we evaluate messy studies
- How do we discuss things online so that people actually put effort into reading and contributing the discussion.
- What kinds of conversational/debate norms lead people to be more transparent.
- We need to be able to apply all the knowledge to go out and accomplish things, which will probably involve messy political stuff.
I have specific concerns about Sarah's post, which I'll post in a comment when I have a bit more time.
I at least would say that I care about doing the most good that I can, but am also mindful of the fact that I run on corrupted hardware, which makes ends justifying means arguments unreliable, per EY's classic argument (http://lesswrong.com/lw/uv/ends_dont_justify_means_among_humans/)
""The end does not justify the means" is just consequentialist reasoning at one meta-level up. If a human starts thinking on the object level that the end justifies the means, this has awful consequences given our untrustworthy brains; therefore a human shouldn't think this way. But it is all still ultimately consequentialism. It's just reflective consequentialism, for beings who know that their moment-by-moment decisions are made by untrusted hardware."
This doesn't mean I think there's never a circumstance where you need to breach a deontological rule; I agree with EY when they say "I think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort.". This is the reason under Sarah's definition of absolutely binding promises, I would simply never make such a promise- I might say that I would try my best and to the best of my knowledge there was nothing that would prevent me from doing a thing, or something like that- but I think the universe can be amazingly inconvenient and don't want to be a pretender at principles I would not actually in extremis live up to.
The theory I tend to operate under I think of as "biased naive consequentialism", where I do naive consequentialism- estimating out as far as I can see easily- then introduce heavy bias against things which are likely to have untracked bad consequences, e.g. lying, theft. (I kind of am amused by how all the adjectives in the description are negative ones.). But under a sufficiently massive difference, sure, I'd lie to an axe murderer. This means there is a "price", somewhere. This is probably most similar to the concept of "way utilitarianism", which I think is way better than either act or rule utilitarianism, and is discussed as a sort of steelman of Mohist ideas (https://plato.stanford.edu/entries/mohism/).
One of the things I take from the thinking around the non-central fallacy aka the worst argument in the world (http://lesswrong.com/lw/e95/the_noncentral_fallacy_the_worst_argument_in_the/) is that one should smoothly reduce the strength of such biases for examples which are very atypical of the circumstances the bias was intended for, so as to not have weird sharp edges near category borders.
All this is to say that in weird extreme edge cases, under conditions of perfect knowledge, I think what people do is not important. It's okay to have a price. But in the central cases, in actual life, I think they should have either a very strong bias against deception and for viewing deceptive behaviour poorly, or an outright deontological prohibition if they can't reliably maintain that.
If I was to say one thing I think is a big problem, it's that in practice some people's price seems to be only able to be infinite or zero (or even negative- a lot of people seem to get tempted by cool "ends justify means" arguments which don't even look prima facie like they'd actually have positive utility. I mean, trading discourse and growth for money in a nascent movement is, um, even naive utilitarianism can track far enough out to see the problems there, you have to have an intuitive preference for deception to favour it).
I disagree with you in that I think infinite works fine almost always, so it wouldn't be a big problem if everyone had that- I'd be very happy if all the people who had their price to cheat at around zero moved it to infinite. But I agree with you in that infinite isn't actually the correct answer for an ideal unbiased reasoner, just not that this should affect how humans behave while under the normal circumstances that are the work of the EA movement.
The alarming part for me is that I think in general these debates do, because people erroneously jump from "a hypothetical scenario with a hypothetical perfect reasoner would not behave deontologically" to sketchiness in practice.
Let me first clarify that I see the goal of doing the most good as my end goal, and YMMV - no judgment on anyone who cares more about truth than doing good. This is just my value set.
Within that value set, using "insufficient" means to get to EA ends is just as bad as using "excessive" means. In this case, being "too honest" is just as bad as "not being honest enough." The correct course of actions is to correctly calibrate one's level of honesty to maximize for positive long-term impact for doing the most good.
Now... (read more)