15

RobBensinger comments on Multiverse-wide cooperation in a nutshell - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (10)

You are viewing a single comment's thread. Show more comments above.

Comment author: RobBensinger 13 November 2017 03:47:37AM *  1 point [-]

After the boxes have already been placed in front of me, however, I can no longer influence their contents, so it would be good if I two-boxed

You would get more utility if you were willing to one-box even when there's no external penalty or opportunity to bind yourself to the decision. Indeed, functional decision theory can be understood as a formalization of the intuition: "I would be better off if only I could behave in the way I would have precommitted to behave in every circumstance, without actually needing to anticipate each such circumstance in advance." Since the predictor in Newcomb's problem fills the boxes based on your actual action, regardless of the reasoning or contract-writing or other activities that motivate the action, this suffices to always get the higher payout (compared to causal or evidential decision theory).

There are also dilemmas where causal decision theory gets less utility even if it has the opportunity to precommit to the dilemma; e.g., retro blackmail.

For a fuller argument, see the paper "Functional Decision Theory" by Yudkowsky and Soares.

Comment author: JamesDrain 14 November 2017 12:10:45AM 0 points [-]

Ha, I think the problem is just that your formalization of Newcomb's problem is defined so that one-boxing is always the correct strategy, and I'm working with a different formulation. There are four forms of Newcomb's problem that jibe with my intuition, and they're all different from the formalization you're working with.

  1. Your source code is readable. Then the best strategy is whatever the best strategy is when you get to publicly commit e.g. you should tear off the wheel when playing chicken if you have the opportunity to do so before your opponent.
  2. Your source code is readable and so is your opponent's. Then you get mathy things like mutual simulation and lob's theorem.
  3. We're in the real world, so the only information the other player has to guess your strategy is information like your past behavior and reputation. (This is by far the most realistic situation in my opinion.)
  4. You're playing against someone who's an expert in reading body language, say. Then it might be impossible to fool them unless you can fool yourself into thinking you'll one-box. But of course, after the boxes are actually in front of you, it would be great for you if you had a change of heart.

Your version is something like 5. Your opponent can simulate you with 100% accuracy, including unforeseen events like something unexpected causing you to have a change of mind.

If we're creating AIs that others can simulate, then I guess we might as well make them immune to retro blackmail. I still don't see the implications for humans, who cannot be simulated with 100% fidelity and already have ample intuition about their reputations and know lots of ways to solve coordination problems.