5

JesseClifton comments on “Just take the expected value” – a possible reply to concerns about cluelessness - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (18)

You are viewing a single comment's thread. Show more comments above.

Comment author: Owen_Cotton-Barratt 21 December 2017 07:52:45PM 6 points [-]

By now it should be clear that simply following the expected value is not a sufficient response to concerns of cluelessness.

I was pretty surprised by this sentence. Maybe you could say more precisely what you mean?

I take the core concern of cluelessness to be that perhaps we have no information about what options are best. Expected value gives a theoretical out to that (with some unresolved issues around infinite expectations for actors with unbounded utility functions). Approximations to expected value that humans can implement are as you point out kind of messy and opaque, but that's a feature of human reasoning in general, and doesn't seem particularly tied to expected value. Is that what you're pointing at?

Comment author: JesseClifton 22 December 2017 05:20:48PM 3 points [-]

I can’t speak for the author, but I don’t think the problem is the difficulty of “approximating” expected value. Indeed, in the context of subjective expected utility theory there is no “true” expected value that we are trying to approximate. There is just whatever falls out of your subjective probabilities and utilities.

I think the worry comes more from wanting subjective probabilities to come from somewhere — for instance, models of the world that have a track-record of predictive success. If your subjective probabilities are not grounded in such a model, as is arguably often the case with EAs trying to optimize complex systems or the long-run future, then it is reasonable to ask why they should carry much epistemic / decision-theoretic weight.

(People who hold this view might not find the usual Dutch book or representation theorem arguments compelling.)

Comment author: RomeoStevens 24 December 2017 07:48:19AM 3 points [-]

I'll second this. In double cruxing EV calcs with others it is clear that they are often quite parameter sensitive and that awareness of such parameter sensitivity is rare/does not come for free. Just the opposite, trying to do sensitivity analysis on what are already fuzzy qualitative->quantitative heuristics is quite stressful/frustrating. Results from sufficiently complex EV calcs usually fall prey to ontology failures, ie key assumptions turned out wrong 25% of the time in studies on analyst performance in the intelligence community, and most scenarios have more than 4 key assumptions.

Comment author: kbog  (EA Profile) 26 December 2017 08:55:56PM 0 points [-]

I think the worry comes more from wanting subjective probabilities to come from somewhere — for instance, models of the world that have a track-record of predictive success. If your subjective probabilities are not grounded in such a model, as is arguably often the case with EAs trying to optimize complex systems or the long-run future, then it is reasonable to ask why they should carry much epistemic / decision-theoretic weight.

But that just means that people are making estimates that are insufficiently robust to unknown information and are therefore vulnerable to the optimizer's curse. It doesn't imply that taking the expected value is not the right solution to the idea of cluelessness.

Comment author: JesseClifton 27 December 2017 07:46:00AM 0 points [-]

But that just means that people are making estimates that are insufficiently robust to unknown information and are therefore vulnerable to the optimizer's curse.

I'm not sure what you mean. There is nothing being estimated and no concept of robustness when it comes to the notion of subjective probability in question.

Comment author: kbog  (EA Profile) 27 December 2017 05:20:30PM 0 points [-]

The expected value of your actions is being estimated. Those estimates are based on subjective probabilities and can be well or poorly supported by evidence.

Comment author: JesseClifton 27 December 2017 08:38:33PM 3 points [-]

For a Bayesian, there is no sense in which subjective probabilities are well or poorly supported by the evidence, unless you just mean that they result from calculating the Bayesian update correctly or incorrectly.

Likewise there is no true expected utility to estimate. It is a measure of an epistemic state, not a feature of the external world.

I am saying that I would like this epistemic state to be grounded in empirical reality via good models of the world. This goes beyond subjective expected utility theory. As does what you have said about robustness and being well or poorly supported by evidence.

Comment author: kbog  (EA Profile) 07 January 2018 07:38:10AM *  0 points [-]

For a Bayesian, there is no sense in which subjective probabilities are well or poorly supported by the evidence

Yes, whether you are Bayesian or not, it means that the estimate is robust to unknown information.

I am saying that I would like this epistemic state to be grounded in empirical reality via good models of the world. This goes beyond subjective expected utility theory.

No, subjective expected utility theory is perfectly capable of encompassing whether your beliefs are grounded in good models. I don't see why you would think otherwise.

As does what you have said about robustness and being well or poorly supported by evidence.

No, everything that has been written on the optimizer's curse is perfectly compatible with subjective expected utility theory.

Comment author: JesseClifton 07 January 2018 10:22:15PM 0 points [-]

whether you are Bayesian or not, it means that the estimate is robust to unknown information

I’m having difficulty understanding what it means for a subjective probability to be robust to unknown information. Could you clarify?

subjective expected utility theory is perfectly capable of encompassing whether your beliefs are grounded in good models.

Could you give an example where two Bayesians have the same subjective probabilities, but SEUT tells us that one subjective probability is better than the other due to better robustness / resulting from a better model / etc.?

Comment author: kbog  (EA Profile) 08 January 2018 05:57:15PM *  0 points [-]

It means that your credence will change little (or a lot) depending on information which you don't have.

For instance, if I know nothing about Pepsi then I may have a 50% credence that their stock is going to beat the market next month. However, if I talk to a company insider who tells me why their company is better than the market thinks, I may update to 55% credence.

On the other hand, suppose I don't talk to that guy, but I did spend the last week talking to lots of people in the company and analyzing a lot of hidden information about them which is not available to the market. And I have found that there is no overall reason to expect them to beat the market or not - the info is good just as much as it is bad. So I again have a 50% credence. However, if I talk to that one guy who tells me why the company is great, I won't update to 55% credence, I'll update to 51% or not at all.

Both people here are being perfect Bayesians. Before talking to the one guy, they both have 50% credence. But the latter person has more reason to be surprised if Pepsi diverges from the mean expectation.

Comment author: JesseClifton 08 January 2018 10:21:40PM *  0 points [-]

It sounds to me like this scenario is about a difference in the variances of the respective subjective probability distributions over future stock values. The variance of a distribution of credences does not measure how “well or poorly supported by evidence” that distribution is.

My worry about statements of the form “My credences over the total future utility given intervention A are characterized by distribution P” does not have to do with the variance of the distribution P. It has to do with the fact that I do not know whether I should trust the procedures that generated P to track reality.