A study is underpowered when its sample is too small to give a reasonable chance of detecting an effect of the size that would matter if it existed. The consequences are well understood. True effects are missed more often than not, and the effects that do reach significance are overestimated, sometimes with the wrong sign (Gelman and Carlin, 2014). When most studies in a field have low power, a significant finding is weaker evidence than its p-value suggests, a point central to the argument of Ioannidis (2005) about the reliability of published research.

None of this is news to statisticians or to most working biologists. What needs explaining is the persistence of the practice in the face of that knowledge, and in particular the role of peer review. Every underpowered paper in the literature was read before publication by an editor and by reviewers chosen for their expertise, and it was accepted. This article argues that review fails here for structural reasons and not through carelessness, that the reasons are especially strong in quantitative biology, and that they point to remedies different from another exhortation to reviewers to check the sample size.

The first systematic audit of power was carried out by Cohen (1962), who worked through a volume of a psychology journal and found that the studies in it had roughly even odds of detecting an effect of medium size. A generation later Sedlmeier and Gigerenzer (1989) repeated the exercise on a later volume of its successor journal and found no improvement, and asked in their title whether studies of power have any effect on the power of studies. Button et al. (2013) brought the question to neuroscience, using published meta-analyses to estimate the power of the contributing studies, and concluded that it was typically far below the conventional target of 0.80.

Laboratory biology has its own evidence. A survey of published animal experiments found that the basis for the chosen sample size was hardly ever reported, alongside frequent omissions in the description of design and analysis (Kilkenny et al., 2009). When the reasoning behind a sample size is absent from the record, nobody downstream can evaluate it. The stability of these findings across decades and disciplines suggests that whatever maintains small samples is not addressed by pointing the problem out.

Four features of the review process work against the detection of inadequate power. They are summarised in Table 1, together with two further mechanisms that operate before a manuscript ever reaches a reviewer.

Two mechanisms act before review and determine what reviewers see. Small studies that find nothing tend not to be written up, the file drawer effect described by Rosenthal (1979), and flexibility in analysis raises the proportion of small studies that find something (Simmons et al., 2011). The submissions that arrive are thus a filtered set in which small samples and significant results go together.

The published effects are then used to plan new work. An investigator who bases a sample size on the effect reported in the literature, as textbooks advise, will calculate that a small sample is sufficient, because the reported effect is inflated. The loop in Figure 1 closes without anyone in it behaving unreasonably by local standards. Modelling work has formalised the wider dynamic. Smaldino and McElreath (2016) showed that when laboratories are rewarded for the number of positive findings, methods that generate such findings cheaply spread through a population of laboratories without any intent to deceive, and Higginson and Munafò (2016) found that the rational strategy for a scientist under such incentives is to run many small studies.

Several objections deserve an answer. The first is ethical. Animal experiments are bound by the principle of reduction, and demanding larger groups appears to conflict with it. The conflict is only apparent. An experiment too small to answer its question uses animals to no purpose, and the ARRIVE 2.0 guidelines accordingly ask authors to explain how the sample size was decided, not to minimise it (Percie du Sert et al., 2020).

The second objection is that much laboratory work is exploratory, and that requiring power calculations for exploration is a category mistake. This has force. Exploration needs no prior hypothesis about effect size. The difficulty is that exploratory studies are seldom reported as such: they are written up with confirmatory language and tested against confirmatory thresholds. A study presented as hypothesis generating, with estimates and intervals in place of claims of discovery, is not open to the criticism made here.

The third is that biological effects are often large, so small samples suffice. Sometimes they are, and a knockout that abolishes a signal needs few replicates to show it. But the claim has to be made and defended in advance for the effect in question, which is what a sample size justification is. A fourth objection holds that meta-analysis will eventually correct individual small studies. It cannot do so when the studies that enter it have been selected for significance.

If the analysis above is right, asking reviewers to be more vigilant will achieve little, because the information they need is missing and the cues they have point the wrong way. Three changes address the mechanisms directly. The first is to move review of the design ahead of the results. In the Registered Reports format introduced by Chambers (2013), the rationale, methods and planned sample are reviewed and provisionally accepted before data are collected, so that the sample size is examined when it can still be changed and when no result exists to make it look adequate. The format is now offered by many journals and was among the measures recommended in the manifesto of Munafò et al. (2017).

The second change is to alter what a conventional submission must contain. A justification of sample size need not be a formal power analysis. Lakens (2022) describes several legitimate forms, including justification by resource constraints, provided that the consequences for what the study can show are stated. Coupled with a required statement of the experimental unit and the number of independent replicates, this puts the missing information on the page. The third change is to shift the question reviewers ask of a significant result from whether an effect exists to how well it has been estimated, which makes a wide interval from a small sample a visible limitation. These are modest demands, and each answers one of the mechanisms in Table 1.

Editors are better placed than reviewers to make such requirements stick. A reviewer who asks for a sample size justification in one manuscript is making a personal demand that the next reviewer may not repeat, and authors learn to treat it as the luck of the draw. A journal that asks for it in every submission, in a fixed place and a fixed form, turns it into a property of the venue. The same holds for the experimental unit: a mandatory line in the methods stating what was independently treated, and how many such units there were, costs authors a sentence and spares reviewers the task of reconstructing it from figure legends. None of these measures removes the incentives that favour small studies, and journals acting alone cannot do that. They would, however, stop review from certifying designs it has had no means of examining.

Underpowered studies pass peer review because review, as ordinarily conducted, is not positioned to stop them. The reviewer arrives after the design is fixed, is shown a result that appears to vindicate it, shares the conventions that produced it, and in quantitative biology may not be told plainly how many independent observations there were. Sixty years of audits have shown that naming the problem does not shift it. Changing when the design is reviewed and what authors must state about it is more likely to, and both changes are within the power of journals to make.

To be declared by the author(s) before publication.

To be declared by the author(s) before publication.