Statistical-significance thinking is not just a bad way to publish, it’s also a...

Eric Loken writes:

The table below wason your blog a few days ago, with the clear point about p-values (and even worse the significance versus non-significance) being a poor summary of data. The thought I’ve had lately, working with various groups of really smart and thoughtful researchers, is that Table 4 is also a model of their mental space as they think about their research and as they do their initial data analyses.

It’s getting much easier to make the case that Table 4 is not acceptable to publish . But I think it’s also true that Table 4 is actually the internal working model for a lot of otherwise smart scientists and researchers. That’s harder to fix!

I agree. One problem with all this discussion of forking paths, publication bias, etc., is that this focus on the process of publication/criticism/replication/etc can distract us from the value of thinking clearly when doing research: avoiding the habits of wishful thinking and discretization that lead us to draw strong conclusions from noisy data.

Not long ago we discussed a noisy study produced a result in the opposite direction of the original hypothesis, leading the researcher to completely change the scientific story. Changing your model of the world in response to data is a good thing—but not if the data are essentially indistinguishable from noise. Actually, in that case the decision was based on p-value that did not reach the traditional level of statistical significance, but the general point still holds.

Whether you’re studying voting, or political attitudes, or sex ratios, or whatever, it’s ultimately not about what it takes, or should take, to get a result published, but rather how we as researchers can navigate through uncertainty and not get faked out by noise in our own data.