Comment author: erikaalonso 13 January 2017 12:38:41AM *  20 points [-]

Hi everyone! I’m here to formally respond to Sarah’s article, on behalf of ACE. It’s difficult to determine where the response should go, as it seems there are many discussions, and reposting appears to be discouraged. I’ve decided to post here on the EA forum (as it tends to be the central meeting place for EAs), and will try to direct people from other places to this longer response.

Firstly, I’d like to clarify why we have not inserted ourselves into the discussion happening in multiple Facebook groups and fora. We have recently implemented a formal social media policy which encourages ACE staff to respond to comments about our work with great consideration, and in a way that accurately reflects our views (as opposed to those of one staff member). We are aware that this might come across as “radio silence” or lack of concern for the criticism at hand—but that is not the case. Whenever there are legitimate critiques about our work, we take it very seriously. When there are accusations of intent to deceive, we do not take them lightly. The last thing we want to do is respond in haste only to realize that we had not given the criticism enough consideration. We also want to allow the community to discuss amongst themselves prior to posting a response. This is not only to encourage discussion amongst individual members of the community, but also so that we can prioritize responding to the concerns shared by the greatest number of community members.

It is clear to us now that we have failed to adequately communicate the uncertainty surrounding the outcomes of our leafleting intervention report. We absolutely disagree with claims of intentional deception and the characterization of our staff as acting in bad-faith—we have never tried to hide our uncertainty about the existing leafleting research report, and as others have pointed out, it is clearly stated throughout the site where leafleting is mentioned. However, our reasoning that these disclaimers would be obvious was based on the assumption that those interested in the report would read it in its entirety. After reading the responses to this article, it’s obvious that we have not made these disclaimers as apparent as they should be. We have added a longer disclaimer to the top of our leafleting report page, expressing our current thoughts and noting that we will update the report sometime in 2017.

In addition, we have decided to remove the impact calculator (a tool which included an ability to enter donations directed to leafleting and receive estimates of high and low bounds of animals spared) from our website entirely until we feel more confident that it is not misleading to those unfamiliar with cost effectiveness calculations and/or an understanding of how the low/best/high error bounds exemplify the uncertainty regarding those numbers. It is not typical for us to remove content from the site, but we intend to operate with abundant caution. This change seems to be the best option, given that people believe we are being intentionally deceptive in keeping them online.

Finally, leadership at ACE all agree it has been too long since we have updated our Mistakes page, so we have added new entries concerning issues we have reflected upon as an organization.

We also notice that there is concern among the community that our recommendations are suspect due to the weak evidence supporting our cost-effectiveness estimates of leafleting. The focus on leafleting for this criticism is confusing to us, as our cost-effectiveness estimates address many interventions, not only leafleting, and the evidence for leafleting is not much weaker than other evidence available about animal advocacy interventions. On top of that, cost-effectiveness estimates are only a factor in one of the seven criteria used in our evaluation process. In most cases, we don’t think that they have changed the outcome of our evaluation decisions. While we haven’t come up with a solution for clarifying this point, we always welcome and are appreciative of constructive feedback.

We are committed to honesty, and are disappointed that the content we've published on the website concerning leafleting has caused so much confusion as to lead anyone to believe we are intentionally deceiving our supporters for profit. On a personal note, I’m devastated to hear that our error in communication has led to the character assassination not only of ACE, but of the people who comprise the organization—some of the hardest working, well-intentioned people I’ve ever worked with.

Finally, I would like everyone to know that we sincerely appreciate the constructive feedback we receive from people within and beyond the EA movement.

*Edited to add links

Comment author: Jeff_Kaufman 14 January 2017 04:30:27PM 1 point [-]

We have recently implemented a formal social media policy which encourages ACE staff to respond to comments about our work with great consideration, and in a way that accurately reflects our views (as opposed to those of one staff member).

Is this policy available anywhere? Looking on your site I'm finding only a different Social Media Policy that looks like maybe it's intended for people outside ACE considering posting on ACE's fb wall?

Comment author: Peter_Hurford  (EA Profile) 05 January 2017 07:25:49PM 0 points [-]

That sounds pretty awesome, who do you think would be a good person to reach out to when I'm ready?

Comment author: Jeff_Kaufman 06 January 2017 09:07:28PM 0 points [-]

Ben Kuhn maybe?

Comment author: Peter_Hurford  (EA Profile) 04 January 2017 07:43:12PM 1 point [-]

Thanks Jeff!

-

1) ... I think the chance of failure could be significantly higher.

Possibly, but they are already starting to operate in the country in question, and my understanding is that's been going pretty well. My impression is that they're much more competent than Safaricom. My inside view is much higher than 50%, and getting down to 50% was a discount from there.

Sounds like you definitely have inside info that I don't have, so for now I'd have to rely on my outside view, but I can work to acquire that inside info if I look into this more.

-

2) ... I expect future roll-outs will take place in countries with higher base consumption

I'm confused. I was trying to talk about the counterfactual for a specific very poor country if Wave were not working there. So if future mobile money rollouts by other organizations happen first in countries with higher base consumption then that increases the counterfactual impact of Wave choosing to come into a country with very low consumption.

I don't know what country Wave is looking at or how they are doing what they do because you have inside info that I don't have. If it has consumption comparable to Kenya than my point is invalid. I just was concerned that it wouldn't.

-

3) ... I expect them to continue at AMF levels (or greater) for at least a few more years

See http://www.jefftk.com/p/leaving-google-joining-wave#fb-835897806972_835943804792

Cool. Sounds like this isn't a disagreement between us then.

-

4) ... I really don't know how many staff years it would take

That, combined with estimating marginal impact, makes this pretty awkward. I figure something like 40 person years?

Agreed that it is pretty awkward to estimate. I modified my model to use some of your inputs -- such as a 40% chance of 1-10M subscribers and a 10% chance of >10M subscribers and 40 person years -- and it comes out to $383/hr (95%: $145/hr to $834/hr). The new mean is still in my old 95% interval which is about the best I can hope for with this level of uncertainty.

-

5) ... I'm confused about why GiveDirectly is stated to be 5x more cost-effective than AMF

This comes from cell F31 of the "Results" tab. I haven't put time into understanding how that's calculated, but it looked like the relevant bottom line number.

Oh, I see that now. I suppose this is a question for GiveWell and not you. I'll ask them.

Comment author: Jeff_Kaufman 05 January 2017 06:29:29PM 1 point [-]

Sounds like you definitely have inside info that I don't have, so for now I'd have to rely on my outside view, but I can work to acquire that inside info if I look into this more.

If you're interested in working for Wave, or are advising other people on whether it's a good idea for them, I could imagine they'd be quite interested in talking to you!

if it has consumption comparable to Kenya than my point is invalid. I just was concerned that it wouldn't.

It's poorer than Kenya.

Comment author: Peter_Hurford  (EA Profile) 04 January 2017 06:13:00PM 1 point [-]

I spent about two hours looking at this in further depth and made an initial stab at modeling out the impact. I estimate an effectiveness of $200/hr (95% interval: $50/hr to $511/hr), not taking into account the value of donating the salary earned from working at Wave.

Some places where I notice we disagree or I am confused:

1.) I disagree with you here (footnote 1) that there is a 50% chance of failure (or success). I think the chance of failure could be significantly higher. From https://en.wikipedia.org/wiki/M-Pesa: M-Pesa expanded to Kenya (>10M subscribers), Tanzania (5M), South Africa (100K in a year, 1M in five years), India (???), Mozambique (???), and Lesotho (???).

Also, a 2016 Vodaphone press-release suggested M-Pesa seems to have 25M customers worldwide after 10 years of effort.

Based on this, I model that a Kenya-level success (>10M subscribers) thus looks like it would have a less than 1/10 chance and a South Africa-level success (1-10M subscribers) looks like it would have a ~3/10 chance. However, I think this success figure could be lower due to diminishing marginal returns since M-Pesa has already plucked low hanging fruit. It's possible better technology could increase this chance. I'd have to know more specifically about what problems M-Pesa runs into and how these are addressed.

-

2.) I think your estimate that getting M-Pesa a year earlier is only 66x worse than getting a $288 transfer from GiveDirectly is an overestimate because I expect future roll-outs will take place in countries with higher base consumption. However, as you point out, that estimate is also already an underestimate due to misunderstanding the study. I don't know how to correct for this either way, so I used the 66x number literally in my calculation.

-

3.) I disagree with you here (footnote 1a) that marginal ETG donations are at GiveDirectly levels of cost-effectiveness. I expect them to continue at AMF levels (or greater) for at least a few more years, for reasons OpenPhil mentioned and Carl mentioned. I did an AMF-adjustment in my model for this reason.

-

4.) I really don't know how many staff years it would take to either complete a roll-out or know that it's not going to happen and this is an important part of the model. I currently guess 5-10 full-time staff for 2-5 years, or 10-50 total staff years. This does not count field agents or other hired locals. I couldn't find any information on M-Pesa's total staff count anywhere. I note that Wave has at least 44 staff (from counting faces on the about page), but I don't know if they're all full-time or all focused on expanding cash transfers.

-

5.) I'm confused about why GiveDirectly is stated to be 5x more cost-effective than AMF when from GiveWell's cost-effectiveness estimate, GiveDirectly has a median of $7702 per life saved, ranging from $2200 to $16000, excluding outliers. AMF has a median of $3282 per life saved, ranging from $2200 to $4800, excluding outliers. Together, this implies a comparison centered around 2.35x but ranging from 1x to 3.3x. Maybe I misread the sheet -- I haven't invested that much time in making sure I fully understand it yet.

Comment author: Jeff_Kaufman 04 January 2017 07:23:54PM 1 point [-]

1) ... I think the chance of failure could be significantly higher.

Possibly, but they are already starting to operate in the country in question, and my understanding is that's been going pretty well. My impression is that they're much more competent than Safaricom. My inside view is much higher than 50%, and getting down to 50% was a discount from there.

2) ... I expect future roll-outs will take place in countries with higher base consumption

I'm confused. I was trying to talk about the counterfactual for a specific very poor country if Wave were not working there. So if future mobile money rollouts by other organizations happen first in countries with higher base consumption then that increases the counterfactual impact of Wave choosing to come into a country with very low consumption.

3) ... I expect them to continue at AMF levels (or greater) for at least a few more years

See http://www.jefftk.com/p/leaving-google-joining-wave#fb-835897806972_835943804792

4) ... I really don't know how many staff years it would take

That, combined with estimating marginal impact, makes this pretty awkward. I figure something like 40 person years?

5) ... I'm confused about why GiveDirectly is stated to be 5x more cost-effective than AMF

This comes from cell F31 of the "Results" tab. I haven't put time into understanding how that's calculated, but it looked like the relevant bottom line number.

Comment author: Sindy_Li 25 December 2016 07:22:24PM *  3 points [-]

Because it is helpful to think about exactly what intervention is needed to help mobile money expand (which may differ by country), I'm throwing here a few potential barriers (mostly based on my own experience in Kenya and Myanmar):

  • Regulatory barriers (India allowed it only recently because of this; in Myanmar it's still ongoing)

  • Network effects: in Kenya I heard that an important reason it took off was that Safaricom had a very high market share (maybe near 70%?); in Nigeria I heard that the fragmentation of the telecommunication market is one reason it didn't take off. I'm not sure if more countries are similar to Kenya or Nigeria, also these are all anecdotal. One interesting thing though is that lack of competition in Kenya may have contributed to the high charges (though there is more competition now including from mobile carriers and banks).

  • Lack of trust: people may not trust mobile carriers or mobile money agents. Probably less of a problem in a close knit community where agents are shopkeepers. Also, lack of trust in banks is a common problem in developing countries but I have no idea about trust on mobile carriers/agents.

  • Existing alternatives already good enough: this has been mentioned to me in Myanmar, that the traditional "hundi" system of money transfer works well and is cheap which may dampen adoption of mobile money. If that's true then mobile money wouldn't contribute much anyway, but I'm skeptical since mobile money is really much more convenient. (Also it can be used as a savings tool like a checking account, and the poor often face savings constraint too, but I'm not sure how effective that will be; interventions that tackle "self-control" have worked well on this so such elements might need to be bundled in order for mobile money to help with saving)

Comment author: Jeff_Kaufman 03 January 2017 09:35:19PM *  0 points [-]

The intervention I had in mind when writing this post was joining a start-up that has been working on this and has been seeing good results so far: http://www.jefftk.com/p/leaving-google-joining-wave

Comment author: rohinmshah  (EA Profile) 20 December 2016 08:00:32PM 0 points [-]

Thanks!

Comment author: Jeff_Kaufman 21 December 2016 07:18:46PM 0 points [-]

I just realized: there's no way that rss feed can work, because it needs to be authenticated with your cookies. Sorry!

Comment author: HaukeHillebrandt 21 December 2016 02:57:42PM 1 point [-]

Very cool post.

Just saw that the transaction costs for m-pesa are quite high - the company makes ~20% profit... so there might be something that a Wave-like startup could do: https://en.wikipedia.org/wiki/M-Pesa#Cost.2C_transaction_charges.2C_statistics

maybe using crypocurrency - see here:

http://phys.org/news/2016-10-cryptocurrency-bottom-billion.html

Comment author: Jeff_Kaufman 21 December 2016 06:19:20PM *  2 points [-]

Just saw that the transaction costs for m-pesa are quite high - the company makes ~20% profit

The transaction costs listed on the wikipedia page you cite aren't trivial, but would average well less than 20% unless most transactions are (a) very small and (b) to unregistered users. I'm missing something.

EDIT: could it just be that their profit is 20% of expenses, as opposed to 20% of the money that flows through the M-Pesa network?

maybe using crypocurrency

That article doesn't really show that cryptocurrency helps here. Mostly they're unhappy with transaction fees on international remittances, but you can have low transaction fees just by automating interactions with the money transfer organization, without going to cryptocurrency. And with cryptocurrency generally you pay someone a fee to get your money into the cryptocurrency and then your recipient pays someone else a fee to get it into their local currency.

Comment author: Jeff_Kaufman 21 December 2016 04:59:21PM 0 points [-]

The study doesn't appear to control for cash transfers received through access to M-Pesa.

Good point! I hadn't thought about this at all. GiveDirectly's cash transfers were very large, enough that $9.5m would go to 33k people ($288/person). The population was 43M, so 1 in 1300 people received money from GiveDirectly. Their sample size is just 1593, so you expect 0-2 GiveDirectly recipients. I think they should be pretty visible in the data? Might be worth writing to the authors.

It seems like you're assuming that the GiveDirectly money would have gone only to the M-Pesa-access side of the (natural) experiment, but they categorized areas based on whether they had M-Pesa access in 2008-2010, not 2012-2014 when access was much higher.

GiveWell estimate each $ to GiveDirectly raises ln(consumption) by 0.0049

I didn't notice that GiveWell had an estimate for this, and checking now I still don't see it. Where's this estimate from?

(In my post I just took their average amount transferred, figured out what effect that had on the average recipient's income, and then discounted by .8 for GiveDirectly's overhead.)

Comment author: Jeff_Kaufman 21 December 2016 05:30:25PM 1 point [-]

(My method gives an estimate of 0.0022 per dollar to GiveDirectly, so if GiveWell is estimating 0.0049 then my bottom line numbers are roughly 2x too high.)

Comment author: JamesSnowden 21 December 2016 04:16:03PM *  2 points [-]

Thanks for this Jeff - a very informative post.

The study doesn't appear to control for cash transfers received through access to M-Pesa. I was thinking about how much of the 0.012 increase in ln(consumption) was due to GiveDirectly cash transfers.

Back of the envelope:

  • M-Pesa access raises ln(consumption) by 0.012 for 45% of population (c.20m people).
  • 0.012 * 20m = 234,000 unit increases in ln(consumption)

  • GiveDirectly gave c.$9.5m in cash transfers between 2012-14 to people with access to M-Pesa. [1]

  • GiveWell estimate each $ to GiveDirectly raises ln(consumption) by 0.0049
  • 9.5m * 0.0049 = 46,000 unit increases in ln(consumption)

So GiveDirectly accounted for (very roughly) a fifth of the 0.012 increase in ln(consumption) due to M-Pesa.

[1] this is an overestimate as it assumes all transfers went to Kenya and none to Uganda

(Done in haste - may have got my sums / methodology wrong)

Comment author: Jeff_Kaufman 21 December 2016 04:59:21PM 0 points [-]

The study doesn't appear to control for cash transfers received through access to M-Pesa.

Good point! I hadn't thought about this at all. GiveDirectly's cash transfers were very large, enough that $9.5m would go to 33k people ($288/person). The population was 43M, so 1 in 1300 people received money from GiveDirectly. Their sample size is just 1593, so you expect 0-2 GiveDirectly recipients. I think they should be pretty visible in the data? Might be worth writing to the authors.

It seems like you're assuming that the GiveDirectly money would have gone only to the M-Pesa-access side of the (natural) experiment, but they categorized areas based on whether they had M-Pesa access in 2008-2010, not 2012-2014 when access was much higher.

GiveWell estimate each $ to GiveDirectly raises ln(consumption) by 0.0049

I didn't notice that GiveWell had an estimate for this, and checking now I still don't see it. Where's this estimate from?

(In my post I just took their average amount transferred, figured out what effect that had on the average recipient's income, and then discounted by .8 for GiveDirectly's overhead.)

7

Estimating the Value of Mobile Money

In the US we have many options for sending money around: there's the traditional cash, check, credit card, bank transfer, etc., and then there's PayPal, Venmo, Square, Wallet, etc. In poorer countries, however, if you want to send money to someone in another city your best option is often to... Read More

View more: Next