Comment author: Wei_Dai 07 September 2017 08:10:51AM 2 points [-]

This is a bit tangential, but do you know if anyone has done an assessment of the impact of HPMoR? Cousin_it (Vladimir Slepnev) recently wrote:

The question then becomes, how do we set up a status economy that will encourage research? Peer review is one way, because publications and citations are a status badge desired by many people. Participating in a forum like LW when it's "hot" and frequented by high status folks is another way, but unfortunately we don't have that anymore. From that perspective it's easy to see why the massively popular HPMOR didn't attract many new researchers to AI risk, but attracted people to HPMOR speculation and rational fic writing. People do follow their interests sometimes, but mostly they try to find venues to show off.

Taking this one step further, it seems to me that HPMoR may have done harm by directing people's attentions (including Eliezer's own) away from doing the hard work of making philosophical and practical progress in AI alignment and rationality, towards discussion/speculation of the book and rational fic writing, thereby contributing to the decline of LW. Of course it also helped bring new people into the rationalist/EA communities. What would be a fair assessment of its net impact?

Comment author: lukeprog 09 September 2017 10:14:14PM 2 points [-]

Back in ~2014, I remember doing a survey of top-contributing MIRI donors over the previous 3 years and a substantial fraction (1/4th?) had first encountered MIRI or EA or whatever through HPMoR. Malo might have the actual stats. It might even be in a MIRI blog post footnote somewhere.

But w.r.t. to research impact, someone could make a list of the 25 most useful EA researchers, or the 15 most useful "AI safety" researchers, or whatever kind of research you most care about, and find out what fraction of them were introduced to x-risk/EA/rationality/whatever through HPMoR.

I don't have a good sense for the what the net impact is.

Comment author: lukeprog 07 September 2017 04:00:23AM 3 points [-]

We got close to doing this when I was at MIRI but just didn't have the outreach capacity to do it. The closest we got was to print a bunch of paperback copies of (the first 17 chapters of) just one book, HPMoR, and we shipped copies of that to contacts at various universities etc. I think we distributed 1000-2000 copies, not sure if more happened after I left.

Comment author: lukeprog 30 August 2017 10:37:43PM 2 points [-]

Without taking the time to reply to the post as a whole, a few things to be aware of…

Efforts to Improve the Accuracy of Our Judgments and Forecasts

Tetlock forecasting grants 1 and 2

What Do We Know about AI Timelines?

Some AI forecasting grants: 1, 2, 3.

Comment author: kbog  (EA Profile) 29 June 2017 06:06:45AM *  1 point [-]

The report aims to be a "direct inquiry into moral status," but because it does so from an anti-realist perspective,

Why? It's not more widely accepted than realism, and arguably it's decision-irrelevant regardless of its plausibility as per Ross's deflationary argument (http://www-bcf.usc.edu/~jacobmro/ppr/deflation-ross.pdf).

a certain notion of idealized preferences comes into play. In other words: if you don't think "objective values" are "written into the fabric of the universe," then (according to one meta-ethical perspective) all that exists are particular creatures that value things, and facts about what those creatures would value if they had more time to think about their values and knew more true facts and so on.

But there are plenty of accounts of anti-realist ethics, and they don't all make everything reducible to preferences or provide this account of what we ought to value. I still don't see what makes this view more noteworthy than the others and why Open Phil is not as interested in either a general overview of what many views would say or a purely empirical inquiry from which diverse normative conclusions can be easily drawn.

And in fact, as I understand it, the people involved in making Open Phil grantmaking decisions about farm animal welfare do have substantial disagreements with my own moral intuitions, and are not making their grantmaking decisions solely on the basis of my own moral intuitions, or even solely on the basis of guesses about what my "idealized" moral intuitions would be.

I don't see what reason they have to take them into account at all, unless they accept ideal advisor theory and are using your preferences as a heuristic for what their preferences would be if they did the same thing that you are doing. Is ideal advisor theory their view now? And is it also the case for other moral topics besides animal ethics?

Also: can we expect a reduced, empirical-only version of the report to be released at some point?

Comment author: lukeprog 08 July 2017 10:22:33PM 1 point [-]

These are reasonable questions, and I won't be able to satisfactorily address them in a short comment reply. Nevertheless, I'll try to give you a bit more of a sense of "where I'm coming from" on the topic of meta-ethics.

As I say in the report,

I suspect my metaethical approach and my moral judgments overlap substantially with those of at least some other Open Philanthropy Project staff members, and also with those of many likely readers, but I also assume there will be a great deal of non-overlap with my colleagues at the Open Philanthropy Project and especially with other readers. My only means for dealing with that fact is to explain as clearly as I can which judgments I am making and why, so that others can consider what the findings of this report might imply given their own metaethical approach and their own moral judgments.

We don't plan to release an "empirical-only" version of the report, but I think those with different meta-ethical views will be able to read the empirical sections of the report — e.g. most of section 3, appendices C-E, and some other sections — and think for themselves about what those empirical data imply given their own meta-ethical views.

However, your primary question seems to be about why Open Phil is willing to make decisions that are premised on particular views about meta-ethics that we find plausible (e.g. ideal advisor theory) rather than a broader survey of (expert?) views about meta-ethics. I'll make a few comments about this.

First: the current distribution of expert opinion is a significant input to our thinking, but we don't simply defer to it. This is true with respect to most topics that intersect with our work, not just meta-ethics. Instead, we do our best to investigate decision-relevant topics deeply enough ourselves that we develop our own opinions about them. Or, as we said in our blog post on Hits-based giving (re-formatted slightly):

We don’t defer to expert opinion or conventional wisdom, though we do seek to be informed about them… following expert opinion and conventional wisdom is likely to cut against our goal of seeking neglected causes… We do think it would be a bad sign if no experts… agreed with our take on a topic, but when there is disagreement between experts, we need to be willing to side with particular ones. In my view, it’s often possible to do this productively by learning enough about the key issues to determine which arguments best fit our values and basic epistemology.

Second: one consequence of investigating topics deeply enough to form our own opinions about them, rather than simply deferring to what seems to be the leading expert opinion on the topic (if any exists), is that (quoting again from "Hits-based giving") "we don't expect to be able to fully justify ourselves in writing." That is why, throughout my report, I repeat that my report does not really "argue" for the assumptions I make and the tentative conclusions I come to. But I did make an effort to refer the reader to related readings and give some sense of "where I'm coming from."

Third: even if I spent an entire year writing up my best case for (e.g.) ideal advisor theory, I don't think it would convince you, and I don't think it would be thoroughly convincing to myself, either. We can't solve moral philosophy. All we can do is take pragmatic steps of acceptable cost to reduce our uncertainty as we aim to (as I say in the report) "execute our mission to 'accomplish as much good as possible with our giving' without waiting to first resolve all major debates in moral philosophy."

In the end, anyone who is trying to do as much "good" as possible — or even just "more good than bad" — must either (1) wrestle with the sorts of difficult issues we're wrestling with (or the similarly unsolved problems of some other moral framework), and come to some "best guesses for now," or (2) implicitly make assumptions about ~all those same fraught issues anyway, but without trying to examine and question them. (At least, this is true so long as "good" isn't just defined with respect to domain-narrow, funder-decided "goods" like "better scores by American children on standardized tests.")

We don't think it's possible for Open Phil or any other charitable project to definitively answer such questions, but we do prefer to act on questioned/examined assumptions rather than on largely unexamined assumptions. Hence our reports on difficult philosophical questions summarize what we did to examine these questions and what our best-guess conclusions are for the moment, but those reports do not convincingly argue for any solid "answers" to these questions. (Besides the moral patienthood report, see also e.g. here and here.)

Of course, you might think the above points are reasonable, but still want to know more about why I find ideal advisor theory particularly compelling among meta-ethical views. I can't think of anything especially brief to say, other than "that's the family of views I find most plausible after having read, thought, and argued about meta-ethics for several years." I haven't personally written a defense of ideal advisor theory, and I'm not aware of a published defense of ideal advisor theory that I would thoroughly endorse. If you're curious to learn more about my particular views, perhaps the best I can do is point you to Pluralistic moral reductionism, Mixed Reference: The Great Reductionist Project, and ch. 9 of Miller (2013), and Extrapolated volition (normative moral theory).

Another question you seem to be asking is why Open Phil chose to produce a report with this framing first, as opposed to "a general overview of what many [meta-ethical] views would say." I think this is because ideal advisor theory is especially popular among the people at Open Phil who engage most deeply with the details of our philosophical framework for giving. As far I know, all these people (myself included) have substantial uncertainty over meta-ethical views and normative moral theories (see footnote 12 on normative uncertainty), but (as far I know) we put unusually high "weight" on ideal advisor theories — either as a final "theory" of normative morality, or as a very important input to our moral thinking. Because of this, it seemed likely to be more informative (to our decision-making about grants) per unit effort to conduct an investigation that was a mix of empirical data (not premised on any meta-ethical theory) and moral philosophy (premised on some kind of ideal advisory theory), rather than to produce a more neutral survey of the implications of a large variety of meta-ethical theories, most of which we (the people at Open Phil who engage most deeply with the details of our philosophical framework for giving) have considered before and decided to give little or no weight to (again, as far as I know).

One more comment on ideal advisor theory: What I mean by ideal advisor theory might be less narrow than what you're thinking of. For example, on my meaning, ideal advisor theory could (for all I know) result in reflective equilibria as diverse as contractarianism, deontological ethics, hedonic utilitarianism, egoism, or a thorough-going nihilism, among other views.

That said, as I say in the report, I don't think my tentative moral judgments in the report depend on my meta-ethical views, and the empirical data I present don't depend on them either. Also, we continue to question and examine the assumptions behind our current philosophical framework for giving, and I expect that framework to evolve over time as we do so.

A final clarification: another reason I discuss my meta-ethical views so much (albeit mostly in the appendices) is that I suspect one's ethical views unavoidably infect one's way of discussing the relevant empirical data, and so I chose to explain my ethical views in part so that people can interpret my presentation of the empirical data while having some sense of what biases I may bring to that discussion as a result of my ethical views.

Comment author: kbog  (EA Profile) 29 June 2017 06:06:45AM *  1 point [-]

The report aims to be a "direct inquiry into moral status," but because it does so from an anti-realist perspective,

Why? It's not more widely accepted than realism, and arguably it's decision-irrelevant regardless of its plausibility as per Ross's deflationary argument (http://www-bcf.usc.edu/~jacobmro/ppr/deflation-ross.pdf).

a certain notion of idealized preferences comes into play. In other words: if you don't think "objective values" are "written into the fabric of the universe," then (according to one meta-ethical perspective) all that exists are particular creatures that value things, and facts about what those creatures would value if they had more time to think about their values and knew more true facts and so on.

But there are plenty of accounts of anti-realist ethics, and they don't all make everything reducible to preferences or provide this account of what we ought to value. I still don't see what makes this view more noteworthy than the others and why Open Phil is not as interested in either a general overview of what many views would say or a purely empirical inquiry from which diverse normative conclusions can be easily drawn.

And in fact, as I understand it, the people involved in making Open Phil grantmaking decisions about farm animal welfare do have substantial disagreements with my own moral intuitions, and are not making their grantmaking decisions solely on the basis of my own moral intuitions, or even solely on the basis of guesses about what my "idealized" moral intuitions would be.

I don't see what reason they have to take them into account at all, unless they accept ideal advisor theory and are using your preferences as a heuristic for what their preferences would be if they did the same thing that you are doing. Is ideal advisor theory their view now? And is it also the case for other moral topics besides animal ethics?

Also: can we expect a reduced, empirical-only version of the report to be released at some point?

Comment author: lukeprog 02 July 2017 06:20:29PM *  2 points [-]

(Just FYI, I'm drafting a reply to this, but it might be a while before it's ready to post.)

Comment author: lukeprog 28 June 2017 11:41:38PM 1 point [-]

Cross-posted here from a comment on the announcement post, a question from Evan_Gaensbauer:

Do you think science or philosophy can meaningfully separate the capacity to experience suffering or pain from however else consciousness is posited to be distributed across species? What would be fruitful avenues of research for effective altruists to pursue if it's possible to solve the problem in the first question, without necessarily addressing whatever remains of consciousness?

(To preserve structure, I'll reply in a comment reply.)

Comment author: lukeprog 29 June 2017 12:09:49AM *  2 points [-]

I'll use the terms "nociception" and "pain" as defined in Appendix D:

nociception is the encoding and processing of noxious stimuli, where a noxious stimulus is an actually or potentially body-damaging event (either external or internal, e.g. cutaneous or visceral)… Pain, in contrast to mere nociception, is an unpleasant conscious experience associated with actual or potential body damage (or akin to unpleasant experiences associated with noxious stimuli).

…Nociception can occur without pain, and pain can occur without nociception. Loeser & Treede (2008) provide examples: “after local anesthesia of the mandibular nerve for dental procedures, there is peripheral nociception without pain, whereas in a patient with thalamic pain [a kind of neuropathic pain resulting from stroke], there is pain without peripheral nociception.”

Often, it's assumed that if animals are conscious, then it's likely that they experience pain in conjunction with the types of nociceptive processing that, in humans, would be accompanied by conscious pain — or at least, this seems likely for the animals that are fairly similar to us in their neural architecture (e.g. mammals, or perhaps all vertebrates). And likewise, it seems that if we limit the nociception that occurs in their bodies, then this will also limit the conscious pain they experience if they are conscious at all.

Unfortunately, humans also experience pain without nociception, e.g. (as my report says) "neuropathic pain, and perhaps also… some cases of psychologically-created experiences of pain, e.g. when a subject is hallucinating or dreaming a painful experience." Assuming whichever animals are conscious are also capable of non-nociceptive pain, this means that conscious animals may be capable of suffering in ways that are difficult for us to detect.

Even still, I think we can study the mechanisms of nociception (and other triggers for conscious pain) independently of studying consciousness, and this could lead to interventions that (probably) address the usual causes of conscious pain even if we don't know whether a given animal is conscious or not.

However, I'm not sure this is quite what you meant to be asking; let me know if there was a somewhat different question you were hoping I'd answer.

Comment author: Evan_Gaensbauer 25 June 2017 07:40:09AM 0 points [-]

Hey, that sounds great to me. Thanks. Here's my question.

Do you think science or philosophy can meaningfully separate the capacity to experience suffering or pain from however else consciousness is posited to be distributed across species? What would be fruitful avenues of research for effective altruists to pursue if it's possible to solve the problem in the first question, without necessarily addressing whatever remains of consciousness?

Comment author: lukeprog 28 June 2017 11:41:54PM 0 points [-]

Hi Evan, your question has now been posted to the AMA here.

Comment author: lukeprog 28 June 2017 11:41:38PM 1 point [-]

Cross-posted here from a comment on the announcement post, a question from Evan_Gaensbauer:

Do you think science or philosophy can meaningfully separate the capacity to experience suffering or pain from however else consciousness is posited to be distributed across species? What would be fruitful avenues of research for effective altruists to pursue if it's possible to solve the problem in the first question, without necessarily addressing whatever remains of consciousness?

(To preserve structure, I'll reply in a comment reply.)

Comment author: thebestwecan 28 June 2017 09:30:13PM *  2 points [-]

I think Tomasik's essay is a good explanation of objectivity in this context. The most relevant brief section.

Type-B physicalists maintain that consciousness is an actual property of the world that we observe and that is not merely conceptually described by structural/functional processing, even though it turns out a posteriori to be identical to certain kinds of structures or functional behavior.

If you're Type A, then presumably you don't think there's this sort of "not merely conceptually described" consciousness. My concern then is that some of your writing seems to not read like Type A writing, e.g. in your top answer in this AMA, you write:

I'll focus on the common fruit fly for concreteness. Before I began this investigation, I probably would've given fruit fly consciousness very low probability (perhaps <5%), and virtually all of that probability mass would've been coming from a perspective of "I really don't see how fruit flies could be conscious, but smart people who have studied the issue far more than I have seem to think it's plausible, so I guess I should also think it's at least a little plausible." Now, having studied consciousness a fair bit, I have more specific ideas about how it might turn out to be the case that fruit flies are conscious, even if I think they're relatively low probabilitiy, and of course I retain some degree of "and maybe my ideas about consciousness are wrong, and fruit flies are conscious via mechanisms that I don't currently find at all plausible." As reported in section 4.2, my current probability that fruit flies are conscious (as loosely defined in section 2.3.1 is 10%.

Speaking of consciousness in this way seems to imply there is an objective definition, but as I speculated above, maybe you think this manner of speaking is still justified given a Type A view. I don't think there's a great alternative to this for Type A folks, but what Tomasik does is just frequently qualifies that when he says something like 5% consciousness for fruit flies, it's only a subjective judgment, not a probability estimate of an objective fact about the world (like whether fruit flies have, say, theory of mind).

I do worry that this is a bad thing for advocating for small/simple-minded animals, given it makes people think "Oh, I can just assign 0% to fruit flies!" but I currently favor intellectual honesty/straightforwardness. I think the world would probably be a better place if Type B physicalism were true.

Makes sense about the triviality objection, and I appreciate that a lot of your writing like that paragraph does sound like Type A writing :)

Comment author: lukeprog 28 June 2017 11:35:00PM 2 points [-]

My hope was that the Type A-ness / subjectivity of the concept of "consciousness" I'm using would be clear from section 2.3.1 and 2.3.2, and then I can write paragraphs like the one above about fruit fly consciousness, which refers back to the subjective notion of consciousness introduced in section 2.3.

But really, I just find it very cumbersome to write in detail and at length about consciousness in a way that allows every sentence containing consciousness words to clearly be subjective / Type A-style consciousness. It's similar to what I say in the report about fuzziness:

given that we currently lack such a detailed decomposition of “consciousness,” I reluctantly organize this report around the notion of “consciousness,” and I write about “which beings are conscious” and “which cognitive processes are conscious” and “when such-and-such cognitive processing becomes conscious,” while pleading with the reader to remember that I think the line between what is and isn’t “conscious” is extremely “fuzzy” (and as a consequence I also reject any clear-cut “Cartesian theater.”)

But then, throughout the report, I make liberal use of "normal" phrases about consciousness such as what's conscious vs. not-conscious, "becoming" conscious or not conscious, what's "in" consciousness or not, etc. It's just really cumbersome to write in any other way.

Another point is that, well, I'm not just a subjectivist / Type A theorist about consciousness, but about nearly everything. So why shouldn't we feel fine using more "normal" sentence structures to talk about consciousness, if we feel fine talking about "living things" and "mountains" and "sorting algorithms" and so on that way? I don't have any trouble talking about the likelihood that there's a mountain in such-and-such city, even though I think "mountain" is a layer of interpretation we cast upon the world.

Comment author: Benito 28 June 2017 04:46:37PM 1 point [-]

I was confused by the issue regarding diet qualia. Does the argument reduce to answering this question: “Is it the case that explaining away all the individuals properties of conscious experience could ever add up to a completed explanation-away of consciousness”? (In my understanding, the weak illusionists say that it wouldn’t, the strong illusionists say that it would, and the not-illusionists say that this process can’t even get started.)

Comment author: lukeprog 28 June 2017 11:20:02PM *  1 point [-]

I'm not sure whether the thing you're trying to say is compatible with what I'd say or not. The way I'd say it is this:

The 'weak illusionist' says that while many features of conscious experience can be 'explained away' as illusions, the 'something it's like'-ness of conscious experience is not (and perhaps cannot be) an illusion, and thus must be "explained" rather than "explained away." In contrast, the "strong illusionist" says that even the 'something it's like'-ness of conscious experience is an illusion.

But what might it be for the 'something it's like'-ness to be an illusion? Basically, that it seems to us that there is more to conscious experience than 'zero qualia', but in fact there are only zero qualia. E.g. it seems to us that there are 'diet qualia' that are more than 'zero qualia', but in fact 'diet qualia' have no distinctive features beyond 'zero qualia.'

Now in fact, I think there probably is more to 'zero qualia' than Frankish's "properties of experiences that dispose us to judge that experiences have introspectable qualitative properties that are intrinsic, ineffable, and subjective," but I don't think those additional properties will be difficult for the strong illusionst to adopt, and I don't think they'll vindicate a position according to which there is a distinctive 'diet qualia' option. Speaking of diet qualia vs. zero qualia is very rough: the true form of the answer will be classes of cognitive algorithms (on my view).

View more: Next