Comment author: Dan_Keys 19 February 2016 02:09:45AM 4 points [-]

A quick search into the academic research on this topic roughly matches the claims in this post.

Meta-analyses by Allen (1991) (pdf, blog post summary) and O'Keefe (1999) (pdf, blog post summary) defined "refutational two-sided arguments" as arguments that include 1) arguments in favor of the preferred conclusion, 2) arguments against the preferred conclusion, and 3) arguments which attempt to refute the arguments against the preferred conclusion. Both meta-analyses found that refutational two-sided arguments were more persuasive than one-sided arguments (which include only the first of those 3 types of arguments), which in turn were more persuasive than nonrefutational two-sided arguments (which include the first 2 of those 3 types of arguments).

So: surveying both sides of the argument, and making the case for why one side holds more weight than the other, does seem to lead to more convincing writing.

These results are at a fairly broad level of generality. I don't know if any research has looked at questions like whether it matters if you include the strongest arguments against the preferred conclusion (vs. only including straw man arguments) or if it matters if you act as if the arguments against the preferred conclusion have been completely refuted (vs. somewhat outweighed by the arguments in favor of the preferred conclusion).

A quick skim through the list of articles citing Allen and O'Keefe's papers turned up some studies which look for additional sources of variability which might moderate this effect, but I didn't notice any that challenge the general pattern or which get into really good detail on whether normatively good arguments (e.g., non-straw-man, measured conclusions) are more convincing.

Comment author: Dan_Keys 30 January 2015 06:20:27PM 3 points [-]

Have you looked at the history of your 4 metrics (Visitors, Subscribers, Donors, Pledgers) to see how much noise there is in the baseline rates? The noisier they are, the more uncertainty you'll have in the effect size of your intervention.

Could you have the pamphlets only give a url that no one else goes to, and then directly track how many new subscribers/donors/pledgers have been to that url?

View more: Prev