Surveying scientific experts seems like a promising way to gather informed opinions relevant for important decisions, and to improve common knowledge on important topics.
The IGM economic experts panel has been doing surveys for roughly 10 years, and I wonder how well both the surveyed consensus and the format of the platform hold up, as it might be a good idea to
- copy the general IGM platform to other scientific fields,
- offer ideas to improve the IGM panel, and more ambitiously to improve the economics consensus (e.g. by spotting biases), and
- use the survey results when they interact with topics relevant for EA.
Two ideas how to evaluate the survey's usefulness
- Look through IGM's survey questions and evaluate those which have something like a "correct" answer in retrospect. This requires somebody fairly informed about the surveyed issues.
- Interview someone from IGM, or a researcher knowledgeable about expert consensuses in economics or generally (suggestions?).
Two articles about the IGM survey
- Tyler Cowen evaluating some IGM polls in 2015: "But quite seriously, my opinion of the professional consensus — on topics outside an individual’s research specialty — really has gone down as a result of these polls." (link)
- discussion of biases in the economics elite, in the surveyed IGM questions and for which experts were selected (paper from 2016)
- IGM asking softball questions apparently aiming at eliciting textbook answers
- some schools of economics not being represented, e.g. the Austrian school
- clustering of responses according to political partisanship and departmental affiliations
Related questions that might be interesting
- what has been the impact of the IGM survey?
- did it influence policy makers? did it influence public discourse?
- does it have an impact on the future research topics of surveyed researchers?
- which fields are most promising candidates for a similar survey platform? are there already similar surveys in other fields?
- what influences the buy-in that the IGM experts panel enjoys from academics?
- what tradeoffs are involved in different formats of knowledge elicitation?
If you read the expert comments, very often they complain that the question is poorly phrased. It's typically about wording like "would greatly increase" where there's not even an attempt to define "greatly". So if you want to improve the panel or replicate it, that is my #1 recommendation.
...My #2 recommendation is to create a Metaculus market for every IGM question and see how it compares.
Additionally, sometimes the question seems to ask about one specific cost or benefit of a policy, and respondents are unsure how to answer if they think that issue is unimportant but disagree/agree for other reasons.
This does seem useful. At least one similar survey does exist for other fields: the TRIP survey for international relations scholars. I've found this somewhat useful for my research, though often the questions in IR seem less specific than the questions asked of economists.
There is also the PhilPapers Survey!
Thanks for this post.
Here's a copy of some quick notes I wrote earlier (and don't plan to take any next steps on), in case this is useful to anyone:
Replies from colleagues of mine included:
Thanks, that's useful to me! E.g. I didn't consider trying to convince those surveying institutions to ask about particularly important topics they might not have on their radar, or maybe paying them for the service.
As you don't plan considering it further, did you make a guess how useful work on this seems? For fields that are particularly relevant for EAs (like epidemiology, AI, maybe international relations) it might be very valuable to take the initiative so that at least some share of the surveys will be informative for the most important issues.
I don't have a particularly informed/informative stance on how useful it'd be, unfortunately. (I guess I can at least say it'd be nice if someone spent a few hours thinking further about it, but that's true of many things.)
I think the IGM panel is great, and expanding it within economics or extending it to other fields would be great. Some quick thoughts:
It's impossible to objectively retrospectively evaluate panel responses, because all of the interesting questions are about causal effects of X on Y, where the answer doesn't become clear over time because the counterfactual is never observed. Maybe the best you could do is run a retrospective survey where you ask panel members (or other economists) whether they think the panel responses were right in hindsight.
I'm not surprised that Tyler Cowen has a negative view of many of the panel results; he's a libertarian and the panel is broadly left-leaning, as is the broader economics profession. I think a better critique is one that Cowen expresses, e.g., in his interview on Spencer Greenberg's podcast: that the views of economists seem to covary pretty closely with the political beliefs of their demographic group (highly educated urban dwellers), suggesting that their views aren't purely reflective of professional expertise.
I also don't think it's concerning that the IGM forum "excludes certain schools of economics, like the Austrian school." The economics profession doesn't really have "schools" anymore, and I don't think the "Austrian school" has ever had much mainstream credibility. The first and third bullet points from the 2016 paper are more concerning.
I don't think the IGM panel influences future research topics, as the questions are typically about topics already highly salient to academic economists. I don't know whether it influences public discourse or policymakers. The panel has high buy-in from academics because its members are all highly regarded in the profession.
Thanks, that all is useful feedback.
Regarding the impossibility of resolving questions about causal effects: Take the example I gave in the screenshot. I would think that there are plausible observations in the coming years that make the causal claim "monopolists using their market power -> inflation" very unlikely, right? E.g. if the inflation decreases without observing changes in competitiveness or something like this. And yes, a retrospective survey might be the gold standard, though I suppose I'd feel somewhat comfortable trusting a careful economist to make a bunch of judgements by herself.
And regarding buy-in, do you have an idea how the members themselves initially got interested in participating? I guess if you want to replicate the IGM panel, you'd need to have some well connected people on board fairly early. Maybe due to econs proximity to politics it's also very natural for them to want to speak out about policy issues.
I'm skeptical that there'll be a consensus 5 or 10 years from now on whether market power was a key contributor to 2021 inflation; there are just too many confounding factors blocking an inference like "inflation decreased without changes in competitiveness, so market power must not have been causing inflation." It's true that retrospective consensus sometimes emerges (for example, that financial deregulation was at least partly responsible for the Global Financial Crisis), but this is pretty rare.
I'm not sure how the forum started. I assume a key factor in making the participants interested was that the panel was started by a reputable institution (Chicago Booth) that could guarantee a wide audience. I wouldn't guess that economists are more willing than academics from other fields to express views on policy-relevant issues; I suspect political scientists, sociologists, etc. would be similarly willing.
Thanks, that makes sense!