This weekend I finished reading the new book Science Fictions, by Stuart Ritchie. It’s about the replication crisis in science, the underlying causes and ways to make science more reliable.

In this article I’ll review it and ponder some of the issues it raises. I’ll also suggest some topics that it didn’t cover.

Not got time to read a book? No problem! There’s a comic version of it too!

Summary

Overall conclusion: buy it. It’s a great book about a vitally important topic.

Especially if you’ve enjoyed my essays on unreliable science then you’ll enjoy this, because Science Fictions is probably the most approachable and mainstream text on the topic that’s available. Ritchie starts by laying out a clear thesis, spends the bulk of the book making the case for the prosecution and ends with practical suggestions for improvement.

He doesn’t pull his punches or weigh the text down with attempts to please his academic colleagues. It’s written in plain English and you get exactly what it claims to be, quite literally so: the subtitle is “Exposing fraud, bias, negligence and hype in Science” and the bulk of the book is in four chapters titled “Fraud”, “Bias”, “Negligence” and “Hype”. As you’d expect given this structure, it isn’t the work of someone who is tip-toeing around.

Brits might be able to judge the character of the work by the way it opens with a quote from the science episode of Brass Eye, a 1997 hit comedy show that featured celebrities fooled into talking obvious nonsense under the impression that they were helping good causes.

Overview

Science Fictions is made up of three parts.

Part 1 covers the basics of how science works (at least in theory): the grant process, data gathering, paper writing, peer review, publication, citation.

Part 2 is the meat of the book. In each chapter it lays out examples of problematic research practices, explaining clearly not only what went wrong but also exploring the motives of the guilty. This is the fun part.

Part 3 explains the incentive structures in academia and why they so often lead to bad behaviour. Then it lays out a variety of practical suggestions for ways the institutions of science can improve themselves.

Finally there’s an appendix titled, “How to read a scientific paper”. It’s a useful guide to spotting the signs of problems.

Review

Firstly, who is this book for?

If you’re a seasoned follower of the replication crisis, or a ‘sceptic’ of one form or another, then honestly it’s probably not for you. Not because it’s bad but because it’s competently telling a story you already know. I’m nowhere even close to being obsessive about this topic, I just read a lot of stuff online, but even so I found I already recognised about 70%-80% of the stories of misconduct that the book used. The cases I didn’t know about were interesting, but didn’t fundamentally change the strength or weakness of the overall argument.

Yet seeing it all laid out made me realise I’ve been reading about scientific misconduct for perhaps a decade now, and without a book like this that tied it all together I’d have no chance of ever adequately communicating the rationale behind my overall dim view of academic research. There are many people in the same boat: years of reading leave conclusions formed from a wealth of evidence, but it’s scattered across so many long-form news articles, blogs, meta-science papers, forum comments and so on that it can’t be quickly or concisely communicated. Often you can’t even re-find something you remember reading!

So is the book for someone with no prior exposure to the problems of science?

Without a doubt, the intelligent layman is the target audience. For the right reader it will be highly convincing, perhaps even worldview shaking. If you have doubts about some scientific field but aren’t quite sure how to convince your friends or family, start by buying them this book as a gift. As you’d expect given that it’s written by a psychologist with many published papers and impeccable media credentials, it carefully avoids hot-button areas that would cause some readers to automatically reject it out of hand. For example it almost entirely ignores climatology or vaccine safety, preferring psychology, nutrition and medicine for the bulk of its examples. At the same time it builds a case that applies to all scientific fields and the book is quite clear on that — its thesis is that science is broken, not merely specific sub-fields.

On the other hand, the book does make a few assumptions about the reader:

1. Ritchie isn’t afraid of talking about maths. Whilst assuming no particular mathematical strength, due to the structure of the book he packs almost all the human interest stories into the first and last chapters of Part 2 (“fraud”, “hype”) and puts a chapter devoted to statistical issues (“bias”) in the middle.
   The topic-oriented structure of the book makes this unavoidable, but readers who are kept awake by juicy stories of scientists led astray may struggle with this chapter, as it’s a 40 page discussion of p-values, effect sizes, sampling bias, funnel plots and so on. It’s all very approachable and easy to read, but must inherently require that you find these things interesting enough to bother. Readers who can’t quite make it through might be left with the the take-home message that “scientists can commit fraud too”, which isn’t actually the main argument.
2. It assumes the reader is not a scientist. If the reader is in academia and concerned enough with integrity to buy the book, they are probably already aware of the problems and can skip straight to the last chapter which discusses potential solutions. But although I found this chapter thought provoking, with many possible improvements I hadn’t heard about, it’s probably not enough by itself to make the book worth buying, especially as some of the solutions it talks about have been already been promoted for decades.
3. It’s not for people who are interested in the topical intersections of science and law, policy or journalism. It’s not even for university administrators, journal editors or anyone else who can directly affect change. The advice for how to improve science is entirely inward focused on researchers themselves. The appendix would be useful for non-scientists who want to do a basic sanity check of a paper, but the book offers no profession-specific advice.

Author worldview

In a Sowellian framework, Ritchie has a fairly constrained vision. He sees human nature as flawed, unchanging and consistent between people. In this understanding of humanity incentives are very important. Academia on the other hand is dominated by a highly unconstrained vision, in which human nature is widely varying, perfectible and (especially amongst academics) essentially good. This conflict leads to occasionally jarring moments, like when he admits that many of the improvements he suggests either probably won’t work or indeed haven’t worked, as they’re pushing against deep and bad reward structures.

At the same time the book is fundamentally optimistic. The many creative and convincing ideas for how science can be improved stand in contrast to the paucity of ideas for self-improvement found in other fields. How many journalists are writing books like this one? Oh sure, a few make some coin off entertaining trashings of their profession, like Glenn Greenwald or Matt Taibbi, but they don’t have even one concrete suggestion for systematic improvement … let alone the long list found here. Ritchie’s message is that science, despite its many flaws, is able to use its own tools to improve itself.

Yet people who have a worldview in which scientists are noble or even heroic, in which they are battling to save the world from ignorant and villainous politicians/capitalists, may still find Ritchie’s fundamental outlook alien and his total demolition of scientific certainty repellant, or even dangerous.

Stumbles at the last hurdle

Indeed, one gets the strong impression that this problem may have started to come up as he finished the text and asked his friends to review it. The final pages are an epilogue which feels quite different to the rest of the book.

Normally an epilogue is meant to summarise, or perhaps add a few thoughts by the author of a reflective nature. This epilogue is essentially a defence against the charge that he shouldn’t have written the book at all. As the author tries to convince his friends he’s not an evil conservative, he loses the focus and clarity that typifies the rest of the work.

As Ritchie admits on page 241:

The usual reaction I received when I told my friends about this book was a broader concern regarding trust in science: “Isn’t it irresponsible to write something like that? Won’t you just encourage a free-for-all, where people use your arguments to justify their disbelief in evolution, or in the safety of vaccines, or in man-man global warming? After all, if mainstream science is so biased, and its results so hyped, why should the average person believe what scientists are telling them?”

Given the thesis of the book this concern is both entirely valid and entirely irrelevant. After all, Ritchie has just spent 240+ pages arguing that modern science is filled with nonsense, so people will indeed use his arguments to justify disbelief in various scientific outputs. And they’re right to do so, because science isn’t a religion and “belief” isn’t meant to enter into it, so if science is systematically unreliable that should rationally lead to higher levels of scepticism.

Yet what follows is a remarkably shambolic attempt at a counter-argument, quite unlike the confident and convincing prose in the rest of the book.

He argues that it doesn’t matter because trust in science is very high according to polls, so hearing about problems in it will only reduce scientific trust a little bit and thus isn’t to worry about. His next argument is that the drumbeat of obviously untrustworthy findings being reported in the media will reduce faith in science much more than his exposé will. Yet such stories have been regularly published for decades, perhaps centuries, and as he just pointed out, trust in scientists is still very high, so there’s no reason to believe that. Clearly, the fact that everything we eat is associated with cancer hasn’t rubbed off on scientists at all. Probably people assume it’s the journalist’s fault rather than the scientists (incidentally the “Hype” chapter demolishes this idea with, ironically, meta-scientific studies which show that scientists themselves are responsible for most hype surrounding papers).

This sort of internal contradiction is jarring because the rest of the book is free of it.

Unfortunately what follows next is far weaker still. I’m not myself big into climate change scepticism: as a quick perusal of my blog archives will show I’ve never discussed it. I’ve blogged positively about energy efficiency, written a series about unreliable ancient history for my own amusement (part one, two and three if you’re interested), I’ve blogged about unreliable epidemiology in recent months, and I did a takedown of a fraudulent economics paper about Twitter bots biasing referendums. But global warming, no.

Ritchie clearly understands that his book applies to climatology just as much as any research field. Indeed nothing in the book is really about psychology or biology, except that’s where most of the examples come from. But in these final pages he quotes a “science historian” who argues that global warming sceptics “have invoked a fairy-tale image of scientific publishing as a bedrock of consensus, only to profess outrage when it failed to live up to this fantasy … Evidence [the ClimateGate emails] that scientific life behind the printed pages of journals was not a precise reflection of the better behaved public face was seized upon by commentators to argue that the bottom had fallen out”.

But this is exactly what the book has just spent 242 pages arguing: that scientific life behind the journals is not at all a reflection of its better behaved public face, and that the bottom has in fact fallen out. And of course the notion that global warming is scientific consensus isn’t some fantasy fairy tale told by sceptics; it’s the standard rebuttal to them! This seemed totally unfair to the sceptic’s position.

Ritchie goes on to make a whiplash inducing argument in which he tries to take all sides simultaneously. Claiming that climatology has come under a “particularly subtle form of attack”, he cites the US DoA requiring papers to be published with a disclaimer that results are preliminary. Although admitting that “at face value, that appears to be exactly what I’ve been recommending”, he says “nobody thinks this was driven by an innocent desire to improve people’s interpretation of research”, but rather it was because the Trump administration is “pro-fossil-fuels”. Yet a few sentences after attacking the DoA he’s criticising the scientists who “over-compensated” in attacking the DoA, saying they have an “idealised, prettified view of science we’ve seen and dismissed throughout this book”. Then even though “politicians use concerns about replicability as a disingenuous pretext for their scepticism of climate change” that “doesn’t justify scientists over-stating how much confidence we have in our results”.

From a logic perspective this is disastrous. If there are in fact replicability or other problems in climatology then anyone who knows that can’t help but be relatively pro-fossil-fuel compared to someone who doesn’t. Indeed, it would be irrational not to care less about the policy recommendations of scientists who are doing non-replicable work. This is a circular argument, which is why Ritchie ends up contradicting himself so much: in this framing you can’t tell if someone’s “pro fossil fuel” position is the cause of, or caused by, their belief that the science is unreliable. Politicians here can’t win — they must believe even if they have concerns about replicability, otherwise they’re being “disingenuous”. What’s the message here, that only academics may doubt academia?

This section gave me such a bad taste in my mouth and runs so counter to everything the rest of the book is about, I went to find out what climate change sceptics actually did think about the ClimateGate emails. Not surprisingly they’re actually climatology sceptics rather than climate change sceptics (“the climate changes” is a weak statement that almost everyone agrees with). Most of their arguments are about scientific methodologies; the same as in this book. This 10 year retrospective is pretty damning and includes a lot of material that would have fit perfectly in the earlier chapters of the book, like the way academics illegally conspired to destroy emails discussing various manipulations of the scientific process. That wasn’t prosecuted due to a six month statute of limitations, despite the UK government saying it was “hard to imagine more cogent prima facie evidence” of a FOIA offence being committed — and despite university officials repeatedly lying to a parliamentary committee about it.

Heck, that paper is pretty good. Maybe I should become a climatology sceptic too ;)

But after this wobble, Ritchie does get things back on track.

Bravo and well said.

I understand why Ritchie felt the need to pander to his friend’s beliefs here, many of whom are probably academics themselves. But the book would have been stronger if he’d simply left this section out. It’s hard to notice something that’s missing and most readers would reach the end without pondering the list of sub-fields that don’t appear in it. The sudden, lurching introduction of climatology simply raises the question of why it doesn’t appear anywhere else in the book.

Upgrade ideas

My suggestion for a second edition: either lose the epilogue discussion entirely, or man up and include examples of fraud, hype, bias and negligence in climate science. Certainly there will be some if the book’s thesis is correct.

But the epilogue is ultimately a small thing. I’d like to see a second edition of the book explore some additional themes:

• Like almost everyone he conflates science with academia. Despite dwelling heavily on the issue of incentives, he doesn’t investigate corporate science where the incentives are totally different. For companies research papers are means to ends, not ends in and of themselves, so this would have been a fascinating investigation. But corporate science barely gets a mention. I suspect this is because companies don’t really fund psychology, which is the author’s background. In computer science many of the best papers come out of corporate R&D labs so it felt like a big omission.
• In the same vein, patents are a form of scientific publication in which there’s an examination process to ensure the content is actually usable by other people (i.e. replicable) - at least in theory. Comparing patents vs publications would be interesting.
• Modelling isn’t discussed at all. It’s hard to imagine a more relevant topic given that lockdown policies originated in the predictions of by now totally discredited models. The release of the Imperial College London epidemiology software revealed that many scientists are self taught programmers who can’t write code correctly, who routinely evaluate models in circular ways (i.e. against other models), they lie about whether their models are reproducible, and worse, they don’t actually recognise that these are legitimate problems in the first place. Enormous amounts of so-called “science” appears to in reality be the output of computer simulations that assume the correctness of the underlying theories, rather than experiments that could disprove them. Again, I suspect this omission is because modelling isn’t used much in psychology, but it’s of such great relevance any serious discussion of bad science should include it.

Conclusions

Science Fictions is one for the bookshelf of any intellectually curious person.

In some ways it’s a strange book. Viewed from one angle, it’s not for the faint of heart. Although it outlines possible improvements to the academic process, in many ways Ritchie builds the case for the abolition of it. It will leave you feeling like the world is worse than you thought it was.

But viewed from another, it’s the type of psychological self-help book the author sometimes criticises. The contents could be a valuable reality check in all kinds of situations. Perhaps you won’t feel a need to give up that food you love just because of a weak statistical “association with cancer”, or perhaps there’ll be a more substantial benefit — for example, sleeping better at night after reading an expert prediction of catastrophe, or choosing not to invest in a company that’s built on science hype.

Whatever the impact, being more clear eyed about the realities of academia can’t actually hurt you. In the end, it can only help.