Comment author: Carl_Shulman 05 November 2017 05:11:17AM *  13 points [-]

"The p-value critique doesn't apply to many scientific fields." I agree with this, or at least that it is vastly weaker when overwhelming data are available to pin down results.

"As far as I can tell, it mostly applies to social science and maybe epidemiological research. "

I disagree with this.

For instance, p-value issues have been catastrophic in quantitative genetics. The vast bulk of candidate gene research in genetics was non-replicable p-hacking of radically underpowered studies. E.g. schizophrenia candidate genes replicate at chance levels in massive replications but had literatures of p-hacked and publication bias artifact studies. The field moved to requiring genome-wide significance of 5*10^-8 (i.e. Bonferroni corrections for multiple testing at all measured variants). Results obtained in huge genome-wide association studies that meet that criterion replicate reliably.

ETA: It isn't basic biological research, but medical and drug trials routinely have severe p-hacking issues. And there have been a lot of reproducibility problems reported with, e.g. preclinical cancer research, often lacking slam dunk evidence. The Reproducibility Project: Cancer is working on that.

Medical studies take up the bulk of biomedical research funds, and Eliezer's example is at the intersection of medicine and nutrition.

ETA2: I don't think issues of p-hacking would be solved just by using Bayesian statistics: people can instead selectively report Bayes factors, i.e. posterior hacking. It's the selective use of analytic and reporting degrees of freedom that's central. Here's Daryl Bem and coauthors' Bayesian meta-analysis purporting to show psi in Bem's p-hacked experiments.

Comment author: surfergirl 08 November 2017 09:59:11PM *  2 points [-]

medical and drug trials routinely have severe p-hacking issues. And there have been a lot of reproducibility problems reported with, e.g. preclinical cancer research, often lacking slam dunk evidence.

Due to my medical problems I have been reading medical literature for 25 years, and indeed it is a catastrophe of p-hacking and the like, incompetent statistical analysis, ven very often there is a basic misunderstanding of what p-values mean. You routinely see researchers claiming "no effect" when the p value is slightly over 0.05.

Usually, medical papers are misleading in some serious way. The best you can hope for is that they waste the vast majority of the value in the data.

People who read abstracts only and thing they are learning something are deluding themselves. You can to go through the methods section carefully and even then not all the shenanigans are disclosed, and look very closely at sponsorship of the parties to the study (researchers, journal editors, institutions etc) to pick up the extreme biases that result from sponsorship.

Comment author: surfergirl 06 November 2017 02:06:07AM *  0 points [-]

John Ott, who popularized and maybe invented time lapse photography (https://www.youtube.com/watch?v=m3Foz1ZJGlU ), wrote a book called "Health and Light". His observations of plant growth led him to an interest in the health effects of light.

https://www.amazon.com/Health-Light-extraordinary-Affects-emotional/dp/0898040981/ref=sr_1_1?ie=UTF8&qid=1509933857&sr=8-1&keywords=health+and+light

He discusses many of the same issues in that book. His view was that while sunlight was much brighter than internal lights, you did not need very bright lights as long as they were full spectrum e.g. included soft UV light. I don't know if this is true or not.

He ran into the meme of "sunlight causes cancer" among other things. Also he started attacking TV set manufacturers for the levels of x-rays they were emitting and thus made powerful enemies. Also of course he was an outsider.

EY's comments on the limitations of market efficiency are worth drawing out. Highly liquid stocks in the short term are efficient relative to one another. Most stocks are illiquid, expensive to trade and hard to short so mispricings can persist. If the market as a whole, even just the large liquid stocks, are too cheap/expensive it is hard to arbitrage away this mispricing because as Keynes put it "The market can remain irrational longer than you can remain solvent".

Due to agency issues, fund managers are entirely focused on short term performance relative to the market. Any agent who tries to take the long term view, and who therefore suffers from "tracking error" will lose all or almost all his funds under management. This has happened to people with stellar multi-decade track records.

Comment author: Robert_Wiblin 29 October 2017 10:55:34PM *  11 points [-]

This is not really my area, but how strong is the evidence that Japan's monetary policy has been terribly misguided?

Japan's central bank has had to face the uncommon and challenging situation of a declining working age population. This is sufficiently unusual that they can't learn a lot about how to best to handle it from the historical record.

During Japan's “lost decade” (1992-2007), GDP per working-age adult grew at 1.4%/yr, which isn't much slower than the 2.0% rate in the US (the end point of a financial bubble for the US, arguably spurred by excessively low interest rates).

Japan's unemployment rate has been the envy of developed countries, never rising above 5.6% even during the peak of their recessions: https://tradingeconomics.com/japan/unemployment-rate

We can add to this list of partial successes that while they've certainly undershot their inflation target, they haven't had a repeat of their inflation breakout in the early 70s (https://tradingeconomics.com/japan/inflation-cpi). Lost control of inflation expectations is not a pleasant disease to treat. Their GDP per working hour has simply roughly tracked the OECD average - again, half points for that: https://data.oecd.org/lprdty/gdp-per-hour-worked.htm

While they may well have done better than this with more experimental or heterodox monetary policy, central banks tend to be quite risk averse and slow to change. Historically the downsides of messing up your monetary policy have been very large, so they are inclined to accept the 'devil they know' rather than shoot for the ideal policy. Whether being so risk-averse is the right approach just seems unclear to me.

Furthermore, if they changed their targets, methods or processes too frequently, their forward guidance could lose credibility.

I don't mean to say that Japanese monetary policy was correct ex post or even ex ante, just that if this was chosen as a slam dunk instance of 'experts' clearly messing up, it doesn't seem so strong to me.

Comment author: surfergirl 05 November 2017 11:09:30PM 3 points [-]

Excellent points.

To add to this, we should also bear in mind that GDP growth is a bad metric for comparing countries with low population growth with countries with high population growth.

For one western country with high population growth, I calculated that 20-25% of GDP is devoted just to catering for the population growth. New roads, hospitals, houses, offices, power stations, phone lines, offices, factories etc etc. So for Japan with hardly any population growth, GDP overstates the goods available for consumption versus the US, with high population growth. In effect the US has to produce a lot just to stand still.

As Japan has transitioned to low population growth, its effective 'consumption-available' GDP growth has been far higher than it looks.