Derek comments on Proposed methodology for leafleting study - Effective Altruism Forum

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (17)

You are viewing a single comment's thread. Show more comments above.

Comment author: Derek 11 October 2018 09:15:40PM 0 points [-]

It's been a long time since I wrote the comment, but I think I was under the impression the allocation had to happen at the point of distribution, using only the names on the pigeonholes. But if you could get a list of students that exactly matched the names on the pigeonholes in advance of distribution, then I agree randomising hashes would be ideal. I doubt you'd get this from administrators due to data protection issues, but presumably you could go round and manually record the names. That would be very time-consuming for a large study, but perhaps worth it to avoid any doubts about the randomisation.

Note that this would not remove all risk of selection bias because the allocation would still not be concealed, i.e. the people putting the leaflets in the pigeonholes would know who was assigned to each group. It is possible they would consequently give some of the leaflets to the wrong people, e.g. if they wanted to increase the effectiveness of the intervention they might be influenced by personal knowledge of individuals (better to give a leaflet to someone who is not already vegan) or presumed characteristics (gender, ethnicity, etc) that are correlated with likelihood of being influenced by the leaflets. This may seem far-fetched, but unconcealed allocation is associated with higher effect sizes in medical trials so we shouldn't be too quick to assume it wouldn't matter here.



One solution is to give every student a leaflet inside an opaque sealed envelope, with some getting the 'treatment' and some a 'control'. But this introduces additional complexity, e.g. it could cause (further) 'contamination' as students compare what they got in that weird envelope, it reduces external validity (assuming leaflets would not normally be in envelopes), and the control leaflet would have to be very carefully designed so as not to affect the outcomes of interest while being very similar in every other respect (e.g. a leaflet promoting exercise may indirectly influence dietary choices).