RandomEA comments on Students for High-Impact Charity: 2018 Update - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (17)

You are viewing a single comment's thread.

Comment author: RandomEA 09 August 2018 04:32:59AM 2 points [-]

Two significant limitations are high rates of respondent attrition and the likely influence of social desirability bias and/or demand effects, as it was likely clear (post-workshop) which were the desired responses.

It seems to me one indication of social desirability bias and/or selective attrition is that there is a nearly half point shift in the average response to "I currently eat less meat than I used to for ethical reasons." On the other hand, it's possible students interpreted it as "I currently plan on eating less meat than I used to for ethical reasons."

Comment author: Peter_Hurford  (EA Profile) 09 August 2018 01:59:26PM *  1 point [-]

I wonder if it is possible to add a check for this in a future survey. Maybe ask them if they intentionally conserve their water usage to save the environment? There should be no reason for that to change from pre- to post- without a change in social desirability or attrition.

Comment author: cafelow  (EA Profile) 10 August 2018 02:21:56AM 4 points [-]

My guess is the main problem occurs when it is very clear to students what the instructors want them to say. Since we don't talk about water usage, students may not change their answer for the water usage question, but still change their answer to the questions relating to the content (whether or not they have been impacted by the program), so there may not be a shift in the water question, but still a shift in the other questions.

We did test out some different social desirability scales in the surveys, which is a common method. The preliminary analysis suggested that social desirability is a factor, but we haven't finished that analysis on the full data yet.

Comment author: baxterb 09 August 2018 02:01:55PM 0 points [-]

Definitely true on both counts. I suspect that many answers are signalling intentions, but social desirability certainly has a role to play, as we mentioned above. This is one of the reasons we are now placing less emphasis on the future collection of quantitative survey data.