Comment author: Evan_Gaensbauer 03 August 2018 10:03:28PM *  4 points [-]
  1. The CEA, the very organization you juxtaposed with Leverage and Paradigm in this comment has in the past been compared to a Ponzi scheme. Effective altruists who otherwise appreciated that criticism thought much of the value was lost in comparing it to a Ponzi scheme, and without it, the criticism may been better received. Additionally, LessWrong and the rationality community; CFAR and MIRI; and all of AI safety have been for years been smeared as a cult by their detractors. The rationality community isn't perfect. There is no guarantee interactions with a self-identified (aspiring) rationality community will "rationally" go however an individual or small group of people interacting with the community, online or in person, hope or expect. But the vast majority of effective altruists, even those who are cynical about these organizations or sub-communities within EA, disagree with how these organizations have been treated, for it poisons the well of good will in EA for everyone. In this comment, you stated your past experience with the Pareto Fellowship and Leverage left you feeling humiliated and manipulated. I've also been a vocal critic in person throughout the EA community of both Leverage Research and how Geoff Anders has led the organization. But that to elevate a personal opposition of them to a public exposure of opposition research in an attempt to tarnish an event they're supporting alongside many other parties in EA is not something I ever did, or will do. My contacts in EA and myself have followed Leverage. I've desisted in making posts like this myself, because digging for context I found Leverage has changed from any impression I've gotten of them. And that's why at first I was skeptical of attending the EA Summit. But upon reflection, I realized it wasn't supported by the evidence to conclude Leverage is so incapable of change that anything they're associated with should be distrusted. But what you're trying to do with Leverage Research is no different than what EA's worst critics do not in an effort to change EA or its members, but to tarnish them. From within or outside of EA, to criticize any EA organization in such a fashion is below any acceptable epistemic standard in this movement.

  2. If the post and comments here are stating facts about Leverage Research, and you're reporting impressions with no ability to remember specific details that Leverage is like a cult, those are barely facts. The only fact is some people perceived Leverage to be like a cult in the past, which are only anecdotes. And without details, they're only hearsay. Combined with the severity of the consequences if this hearsay was borne out, to be unable to produce actual facts invalidates the point you're trying to make.

Comment author: BenHoffman 05 August 2018 01:14:25AM 4 points [-]

"Compared to a Ponzi scheme" seems like a pretty unfortunate compression of what I actually wrote. Better would be to say that I claimed that a large share of ventures, including a large subset of EA, and the US government, have substantial structural similarities to Ponzi schemes.

Maybe my criticism would have been better received if I'd left out the part that seems to be hard for people to understand; but then it would have been different and less important criticism.

Comment author: gwern 22 June 2018 01:44:03AM *  2 points [-]

It's hard for me to believe that the effect of bednets is large enough to show an effect in RCTs, but not large enough to show up more often than not as a result of mass distribution of bednets.

You may find it hard to believe, but nevertheless, that is the fact: correlational results can easily be several times the true causal effect, in either direction. If you really want numbers, see, for example, the papers & meta-analyses I've compiled in https://www.gwern.net/Correlation on comparing correlations with the causal estimates from simultaneous or later conducted randomized experiments, which have plenty of numbers. Hence, it is easy for a causal effect to be swamped by any time trends or other correlates, and a followup correlation cannot and should not override credible causal results. This is why we need RCTs in the first place. Followups can do useful things like measure whether the implementation is being delivered, or can provide correlational data on things not covered by the original randomized experiments (like unconsidered side effects), but not retry the original case with double jeopardy.

Comment author: BenHoffman 25 July 2018 12:52:42PM 1 point [-]

retry the original case with double jeopardy

This sort of framing leads to publication bias. We want double jeopardy! This isn't a criminal trial, where the coercive power of a massive state is being pitted against an individual's limited ability to defend themselves. This is an intervention people are spending loads of money on, and it's entirely appropriate to continue checking whether the intervention works as well as we thought.

Comment author: gwern 22 June 2018 01:44:03AM *  2 points [-]

It's hard for me to believe that the effect of bednets is large enough to show an effect in RCTs, but not large enough to show up more often than not as a result of mass distribution of bednets.

You may find it hard to believe, but nevertheless, that is the fact: correlational results can easily be several times the true causal effect, in either direction. If you really want numbers, see, for example, the papers & meta-analyses I've compiled in https://www.gwern.net/Correlation on comparing correlations with the causal estimates from simultaneous or later conducted randomized experiments, which have plenty of numbers. Hence, it is easy for a causal effect to be swamped by any time trends or other correlates, and a followup correlation cannot and should not override credible causal results. This is why we need RCTs in the first place. Followups can do useful things like measure whether the implementation is being delivered, or can provide correlational data on things not covered by the original randomized experiments (like unconsidered side effects), but not retry the original case with double jeopardy.

Comment author: BenHoffman 25 July 2018 12:50:37PM *  1 point [-]

As I understand the linked page, it's mostly about retroactive rather than prospective observational studies, and usually for individual rather than population-level interventions. A plan to initiate mass bednet distribution on a national scale is pretty substantially different from that, and doesn't suffer from the same kind of confounding.

Of course it's mathematically possible that the data is so noisy relative to the effect size of the supposedly most cost-effective global health intervention out there, that we shouldn't expect the impact of the intervention to show up. But, I haven't seen evidence that anyone at GiveWell actually did the relevant calculation to check whether this was the case for bednet distributions.

Comment author: gwern 21 July 2017 09:24:48PM *  3 points [-]

The data was noisy, so they simply stopped checking whether AMF’s bed net distributions do anything about malaria.

This is an unfair gotcha. What would the point of this be? Of course the data is noisy. Not only is it noisy, it is irrelevant - if it was not, there would never be any need to have run randomized trials in the first place, you would simply dump the bed nets where convenient and check malaria rates. The whole point of randomized trials is realizing that correlational data is extremely weak and cannot give reliable causal inferences. (I can certainly imagine reasons why malaria rates might go up in regions that AMF does bed net distribution in, just as I can imagine reasons why death rates might be greater or increase over time in patients prescribed new drug X as compared to patients not prescribed X...) If they did the followups and malaria rates held stable or increased, you would not then believe that the bednets do not work; if it takes randomized trials to justify spending on bednets, it cannot then take only surveys to justify not spending on bed nets, as the causal question is identical. Since it does not affect any decisions, it is not important to measure. Or, if it did, what you ought to be criticizing Givewell & AMF for, as well as everyone else, is ever advocating & spending resources on highly unethical randomized trials, rather than criticizing them for not doing some followup surveys.

(A reasonable critique might be that they are not examining whether the intervention - which has been identified as causally effective and passing a cost-benefit - is being correctly delivered, the right people getting the nets, and using the nets. But as far as I know, they do track that...)

Comment author: BenHoffman 29 March 2018 03:34:34AM 1 point [-]

If they did the followups and malaria rates held stable or increased, you would not then believe that the bednets do not work; if it takes randomized trials to justify spending on bednets, it cannot then take only surveys to justify not spending on bed nets, as the causal question is identical.

It's hard for me to believe that the effect of bednets is large enough to show an effect in RCTs, but not large enough to show up more often than not as a result of mass distribution of bednets. If absence of this evidence really isn't strong evidence of no effect, it should be possible to show it with specific numbers and not just handwaving about noise. And I'd expect that to be mentioned in the top-level summary on bed net interventions, not buried in a supplemental page.

Comment author: Khorton 03 December 2017 02:07:32AM 0 points [-]

I'd find this pretty surprising based on my knowledge of the Canadian (Albertan) & British education systems. Does anyone have evidence for standardized exams decreasing "corruption"? (Ben, I'm not sure exactly what you meant by corruption here - do you mean grades that don't match ability, or lazy teaching, or something else?)

Comment author: BenHoffman 05 December 2017 06:31:01PM 1 point [-]

One simple example: https://en.wikipedia.org/wiki/Grade_inflation

More generally, things like the profusion of makework designed to facially resemble teaching, instead of optimizing for outcomes.

Comment author: jamie_cassidy 02 December 2017 02:01:41PM 1 point [-]

Very interesting and I like the central point that cash transfers aren't an automatic win and are therefore worth studying, which I hadn't considered to the same extent before. On the education stuff, it seems like a lot of these problems could be solved if jobs were allocated based on the results of a standardized exam rather than years of schooling or some similar metric. I'm not talking about one run by schools, because it's likely the process wouldn't be trustworthy, I'm talking about when you advertise a job that requires reading, writing, or filing skills, you test for these skills with a written exam. Encouraging governments and other large employers to act in this way would surely encourage students (and parents) to actually learn rather than simply attend school as a box-ticking exercise.

Comment author: BenHoffman 02 December 2017 09:58:43PM 2 points [-]

We should also expect this to mean that countries such as Australia and China that heavily weight a national exam system when advancing students at crucial stages will have less corrupt educational systems than countries like the US which weight locally assessed factors like grades heavily.

(Of course, there can be massive downsides to standardization as well.)

Comment author: [deleted] 02 December 2017 03:00:12PM 2 points [-]

Hmm. I wonder how to reconcile this post with this TED talk:

https://www.youtube.com/watch?v=C30bJBcM_0c

Basically, the TED talk argues that more bureaucracy is very important for economic development because it allows people to invest in various ways with higher confidence that they will actually see the returns. However, as this post describes, bureaucracy can also fall into various rent-seeking traps that waste resources without any real benefit.

Perhaps one way of reconciling is holding that meta-bureaucracy is important, such as standardized testing, performance evaluations, etc.?

Comment author: BenHoffman 02 December 2017 09:56:36PM 1 point [-]

I think the thing to do is try to avoid thinking of "bureaucracy" as a homogeneous quantity, and instead attend to the details of institutions involved. Of course, as a foreigner with respect to every country but one's own, this is going to be difficult to evaluate when giving abroad. This is one of the many reasons why giving effectively on a global scale is hard, and why it's so important to have information feedback of the kind GiveDirectly is working on. Long-term follow-up seems really important too, and even then there's going to be some substantial justified uncertainty.

Comment author: [deleted] 02 December 2017 03:13:18PM *  4 points [-]

Thanks for the post! There's actually a lot of existing literature on these topics.

Regarding the effect of cash on happiness, Haushofer, Reisinger & Shapiro (2015) have a paper called "Your Gain Is My Pain: Negative Psychological Externalities of Cash Transfers".

If you are in fact skeptical of the meaningfulness of your income as a metric, you should be similarly skeptical of the meaningfulness of variations in income of people in poor countries

Maybe I misunderstand you, but the whole point of cash transfers is that money matters more for quality of life when you are poor. See 80,000 Hours' review.

The (lack of) productivity effects of (primary) education in developing countries have also been studied, although I'm less familiar with that literature. However, Haushofer & Shapiro (2016) find that as a result of the cash transfers, "education expenditures increase by USD 1 PPP", while overall expenditure on non-durable goods increases by 36 USD PPP. (Table V).

Comment author: BenHoffman 02 December 2017 09:53:42PM 1 point [-]

There's an implied heuristic that if someone makes an investment that gives them an income stream worth $X, net of costs, then the real wealth of their society increases by at least $X. On this basis, you might assume that if you give a poor person cash, and they use it to buy education, which increases the present value of their children's earnings by $X, then you've thereby added $X of real wealth to their country.

I am saying that we should doubt the premise at least somewhat.

Comment author: BenHoffman 02 December 2017 09:50:20PM 6 points [-]

For some balance, see Kelsey Piper's comments here - it looks like empirically, the picture we get from GiveDirectly is encouraging.

Comment author: BenHoffman 09 November 2017 12:18:59AM *  2 points [-]

To support a claim that this applies in "virtually all" cases, I'd want to see more engagement with pragmatic problems applying modesty, including:

  • Identifying experts is far from free epistemically.
  • Epistemic majoritarianism in practice assumes that no one else is an epistemic majoritarian. Your first guess should be that nearly everyone else is iff you are, in which you should expect information cascades due to the occasional overconfident person. If other people are not majoritarians because they're too stupid to notice the considerations for it, then it seems a bit silly to defer to them. On the other hand, if they're not majoritarians because they're smarter than you are... well, you mention this, but this objection seems to me to be obviously fatal and the only thing left is to explain why the wisdom of the majority disagrees with the epistemically modest.
  • The vast majority of information available about other people's opinions does not differentiate clearly between their impressions and their beliefs after adjusting for their knowledge about others' beliefs.
  • People lie to maintain socially desirable opinions.
  • Control over others' opinions is a valuable social commodity, and apparent expertise gives one some control.

In particular, the last two factors (different sorts of dishonesty) are much bigger deals if most uninformed people copy the opinions of apparently informed people instead of saying "I have no idea".

Overall, I agree that when you have a verified-independent, verified-honest opinion from a peer, one should weight it equally to one's own, and defer to one's verified epistemic superiors - but this has little to do with real life, in which we rarely have that opportunity!

View more: Next