11

Jeff_Kaufman comments on Independent re-analysis of MFA veg ads RCT data - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (6)

You are viewing a single comment's thread. Show more comments above.

Comment author: MichaelDickens  (EA Profile) 20 February 2016 05:01:53AM 4 points [-]

This is a very different way of interpreting the study results than any of the writeups I've seen. Edge's Report, Mercy for Animals, and Animal Charity Evaluators all conclude that there was basically no effect.

From your results it still would be reasonable to conclude that there's "no effect" since the p-value is >0.05, but the p-value is low enough that I would give a follow-up study a reasonably high chance of getting a statistically significant result.

Also: it looks like you're using a one-sided t-test to get your p-value. I don't know much about significance testing but wouldn't it be better to use a two-sided t-test? My understanding is that one-sided tests are sort of cheating by making your p-value half of what it really should be.

Comment author: Jeff_Kaufman 20 February 2016 06:08:47AM *  3 points [-]

it looks like you're using a one-sided t-test to get your p-value.

I agree that a two-sided test would be the right thing to use here, and p-value calculations aren't something I fully understand. Is this calculation one-sided or two-sided?

Comment author: MichaelDickens  (EA Profile) 20 February 2016 04:33:18PM *  4 points [-]

It looks like the NORMDIST function on your sheet is taking the integral from 0 to z_score, which is one-sided. A two-sided test would take

(integral from 0 to z_score) + (integral from -z_score to infinity)
Comment author: Dan_Keys 20 February 2016 06:50:27AM 2 points [-]

I can't tell what's being done in that calculation.

I'm getting a p-value of 0.108 from a Pearson chi-square test (with cell values 55, 809; 78, 856). A chi-square test and a two-tailed t-test should give very similar results with these data, so I agree with Michael that it looks like your p=0.053 comes from a one-tailed test.

Comment author: Jeff_Kaufman 20 February 2016 02:46:41PM *  1 point [-]

Yes, you're right. Sorry! I redid it computationally and also got 0.108. Post updated.