Comment author:Halstead
09 June 2018 12:26:05PM
*
4 points
[-]

The assumptions here about the persistence of the effect seem over-optimistic.

You measure the effect after one month and then assume that it will persist for 1 to 12 years (90% CI). So, you assign a less than 10% chance that the effect will fade out within a year. You made this decision "arbitrarily" on the basis of an ACE meta-analysis investigating how long people who say they don't eat meat have not eaten meat without interruption. The first to say is that this is testing a very different population and so is of questionable relevance to the Animal Equality intervention. In the ACE study, the sample is people who say they have made the commitment to be vegetarian. In yours, it is people who have been shown a video who say they haven't eaten pork a month on.

Given that we are working with fairly arbitrary intuitions here, I find it highly surprising that the 90% CI doesn't include fade out of the effect within a year. My median estimate is that the effect fades out within a year. I'd be curious to hear what other people think about this.

But you think there is around a 10% chance that the effect will fade out after 12 years. The claim is that there is a 10% chance that being shown an animal advocacy video on one day will have an effect on consumption decisions 12 years down the line. I would put the chance of this at ~0%.

If I am right and a more reasonable estimate of persistence seems to be closer to 6 months (I actually think I'm being conservative here - I'd guess closer to 2-3 months), this suggests you should revise your cost-effectiveness estimate down by an order of magnitude.

Comment author:Flodorner
11 June 2018 07:49:32AM
*
0 points
[-]

The claim does not seem to be exactly, that there is a 10% chance of an animal advocacy video affecting consumption decisions after 12 years for a given individual.

I'd interpret it as: there is a 5% chance of the mean duration of reduction, conditioned on the participant reporting to change their behaviour based on the video being higher than 12 years.

This could for example also be achieved by having a very long term impact on very few participants. This interpretation seems a lot more plausible, although i am not certain at all, wheter that claim correct. Long term follow up data would certainly be very helpful.

Comment author:Halstead
11 June 2018 08:01:08AM
1 point
[-]

Yes I was speaking somewhat loosely. It is nevertheless in my view very implausible that the intervention would sustain its effect for that long - we're talking about the effect of one video here. Do you think the chance of fade-out within a year is less than 10%? What is your median estimate?

Comment author:Flodorner
11 June 2018 08:55:19AM
0 points
[-]

Are you talking about the individual level, or the mean? My estimate would be, that for the median individual, the effect will have faded out after at most 6 months. However, the mean might be influenced by the tails quite strongly.

Thinking about it for a bit longer, a mean effect of 12 years does seem quite implausible, though. In the limiting case, where only the tails matter, this would be equivalent to convincing around 25% of the initially influenced students to stop eating pork for the rest of their lives.

The upper bound for my 90% confidence interval for the mean seems to be around 3 years, while the lower bound is at 3 months. The probability mass within the interval is mostly centered to the left.

## Comments (22)

Best*4 points [-]The assumptions here about the persistence of the effect seem over-optimistic.

You measure the effect after one month and then assume that it will persist for 1 to 12 years (90% CI). So, you assign a less than 10% chance that the effect will fade out within a year. You made this decision "arbitrarily" on the basis of an ACE meta-analysis investigating how long people who say they don't eat meat have not eaten meat without interruption. The first to say is that this is testing a very different population and so is of questionable relevance to the Animal Equality intervention. In the ACE study, the sample is people who say they have made the commitment to be vegetarian. In yours, it is people who have been shown a video who say they haven't eaten pork a month on.

Given that we are working with fairly arbitrary intuitions here, I find it highly surprising that the 90% CI doesn't include fade out of the effect within a year. My

medianestimate is that the effect fades out within a year. I'd be curious to hear what other people think about this.But you think there is around a 10% chance that the effect will fade out after 12 years. The claim is that there is a 10% chance that being shown an animal advocacy video on one day will have an effect on consumption decisions 12 years down the line. I would put the chance of this at ~0%.

If I am right and a more reasonable estimate of persistence seems to be closer to 6 months (I actually think I'm being conservative here - I'd guess closer to 2-3 months), this suggests you should revise your cost-effectiveness estimate down by an order of magnitude.

*0 points [-]The claim does not seem to be exactly, that there is a 10% chance of an animal advocacy video affecting consumption decisions after 12 years for a given individual.

I'd interpret it as: there is a 5% chance of the mean duration of reduction, conditioned on the participant reporting to change their behaviour based on the video being higher than 12 years.

This could for example also be achieved by having a very long term impact on very few participants. This interpretation seems a lot more plausible, although i am not certain at all, wheter that claim correct. Long term follow up data would certainly be very helpful.

Yes I was speaking somewhat loosely. It is nevertheless in my view very implausible that the intervention would sustain its effect for that long - we're talking about the effect of one video here. Do you think the chance of fade-out within a year is less than 10%? What is your median estimate?

Are you talking about the individual level, or the mean? My estimate would be, that for the median individual, the effect will have faded out after at most 6 months. However, the mean might be influenced by the tails quite strongly.

Thinking about it for a bit longer, a mean effect of 12 years does seem quite implausible, though. In the limiting case, where only the tails matter, this would be equivalent to convincing around 25% of the initially influenced students to stop eating pork for the rest of their lives.

The upper bound for my 90% confidence interval for the mean seems to be around 3 years, while the lower bound is at 3 months. The probability mass within the interval is mostly centered to the left.