I've been iterating back and forth with Jon Behar at TLYCS, and we've finally honed in on a plan for the pamphleting pilot. The document was rendered with TeX and won't cut and paste nicely (sorry about that), but you can view and download the PDF here. I would love to get some feedback on our approach and (especially) our planned metrics. Thanks!
Have you looked at the history of your 4 metrics (Visitors, Subscribers, Donors, Pledgers) to see how much noise there is in the baseline rates? The noisier they are, the more uncertainty you'll have in the effect size of your intervention.
Could you have the pamphlets only give a url that no one else goes to, and then directly track how many new subscribers/donors/pledgers have been to that url?
I think we'll be able to get a standard deviation, as well as mean, for the baseline values we compare against, which should be helpful to determine if the individual distribution results are significantly different than the baseline rates. I don't think we'll have enough distribution days in the pilot to be able to get the same for the pilot numbers (e.g. we won't be able to tell if the results of individual distributions is typical of all distributions), but that seems like something we could accumulate over time if we proceed with the program.
There is ... (read more)