The dose-response curve is the basic quantitative object of pharmacology and toxicology. From it come the potency of a compound, its maximal effect, the steepness of the transition between them, and the doses taken forward into later studies. In early research the curve is nearly always estimated from little data. The principle of reduction, one of the three Rs set out by Russell and Burch (1959), obliges investigators to use no more animals than the question requires, and practical constraints on compound supply and time push in the same direction. A typical in vivo experiment has a vehicle group and a few dose groups with a small number of animals in each, and the in vitro equivalents are often not much larger once the true unit of replication is counted.

Small experiments are not in themselves a fault, but they strain the statistical methods routinely applied to them. Low power and inflated effect estimates have been documented across the preclinical sciences (Button et al., 2013), and nonlinear curve fitting adds difficulties of its own. Bayesian hierarchical modelling has become the main proposal for analysing such data more stably, because it lets related curves inform one another and admits external information in a controlled way. This article reviews that proposal: what the models are, what they require of the analyst, and where their limits lie.

The article is a narrative review. It draws on standard texts in Bayesian and multilevel modelling (Gelman and Hill, 2007; Gelman et al., 2013), on methodological papers concerning priors, historical data and computation, and on guidance from the toxicological risk assessment community. The literature was not searched systematically, and the selection favours sources that a practising analyst can act on. Clinical dose-finding designs are outside the scope except where their methods have been carried over to laboratory work.

Two separate problems arise when a curve is fitted to a small experiment. One concerns the curve, the other the structure of the data.

The central idea is older than its current popularity. Lindley and Smith (1972) showed that when the parameters of several similar regressions are regarded as exchangeable draws from a common distribution, the estimate for each one is pulled towards the common mean by an amount that depends on how precisely it is determined by its own data. Units with little data are pulled strongly and units with plentiful data hardly at all. This behaviour is called partial pooling or shrinkage. It sits between two extremes that are both unsatisfactory in small studies: fitting every curve separately, which is unstable, and fitting one curve to everything, which ignores real differences.

In a dose-response setting the model is built in levels, as Figure 1 shows. At the bottom, each observation is the value of the Hill function at the given dose plus residual error. The parameters of that function belong to an experimental unit, and unit-level parameters vary around the values for their experiment or batch. Those in turn vary around population values. Not every parameter needs to vary at every level. A common and defensible choice lets the midpoint, on the logarithmic scale, and the maximal effect differ between experiments while the slope is shared or allowed only modest variation, since the slope is the parameter least well determined by sparse data. The textbook bioassay example in Gelman et al. (2013), in which a few animals are tested at each of four dose levels, shows in miniature how much a single small experiment can and cannot say.

The gain is practical. A plate in which the top concentration failed, or an animal cohort that did not reach the plateau, still yields a usable curve, because the missing information about the asymptote is borrowed from sister experiments. Uncertainty about the population parameters is carried through to every derived quantity, so the interval for a potency ratio or a predicted response at an untested dose reflects all sources of variation in the model. The same structure extends naturally to several compounds tested in a common assay, where compound-level potencies are themselves given a distribution.

A hierarchical model needs prior distributions at the top level, and in small studies these matter. The main components of a model and the concern each raises are summarised in Table 1.

Fitting these models once required custom code. General-purpose software now does the work. Stan implements Hamiltonian Monte Carlo with adaptive tuning and has become a standard engine for hierarchical models (Carpenter et al., 2017), and the brms package allows nonlinear multilevel models, Hill curves included, to be specified in R formula syntax and passed to Stan (Bürkner, 2017). Small hierarchical data sets are, perhaps unexpectedly, among the harder cases computationally. When the between-group standard deviation is poorly determined, the posterior has a funnel shape that samplers explore badly, and the software reports divergent transitions. The standard remedy is the non-centred parameterisation, in which group effects are expressed as scaled standard deviates (Papaspiliopoulos et al., 2007). Sampler warnings should be resolved and not ignored, since they indicate that the reported intervals may be wrong.

A fitted model then has to be checked. Simulating data from the prior alone shows whether the priors imply sensible curves before any data are seen, and simulating from the posterior shows whether the model can reproduce the observed pattern of responses across doses; both are part of the graphical workflow described by Gabry et al. (2019). Competing structures, for example shared versus varying slopes, can be compared by approximate leave-one-out cross-validation (Vehtari et al., 2017), though with few experimental units such comparisons are themselves uncertain and should not be over-read. A sensitivity analysis that refits the model under a wider and a narrower prior for the between-group scale is the single most informative check in a small study.

Toxicological risk assessment offers the most developed regulatory use of these ideas. Crump (1984) proposed replacing the no-observed-adverse-effect level, which depends heavily on the doses chosen and the group size, with a benchmark dose defined as the dose producing a specified change in response, reported with a lower confidence bound. Because several curve shapes usually fit toxicity data about equally well and give different benchmark doses, averaging across models was introduced to carry that uncertainty into the result (Wheeler and Bailer, 2007), following the general framework of Bayesian model averaging (Hoeting et al., 1999). The current guidance of the European Food Safety Authority recommends a Bayesian approach with model averaging as the preferred method for deriving a reference point (EFSA Scientific Committee, 2022). Hierarchical structure enters when data from several studies, sexes or related endpoints are analysed together, so that each sparse data set is supported by the others.

The method has costs and conditions. Partial pooling assumes that the units being pooled are exchangeable, meaning that nothing known beforehand distinguishes one from another. If one experiment used a different strain or one compound belongs to a different chemical class, that assumption fails unless the difference is put into the model as a covariate. Shrinkage also pulls unusual units towards the centre, and an unusual unit is sometimes the discovery. A compound that is truly far more potent than its relatives will be estimated as somewhat less so, and the analyst should look at unpooled estimates alongside pooled ones before concluding that nothing stands out.

Dependence on the prior cannot be removed when groups are few, only made visible. Reports should state every prior, give the reasoning for the between-group scale, show the sensitivity analysis and identify the experimental unit, in the spirit of the reporting recommendations long made for Bayesian analyses in health research (Spiegelhalter et al., 2004). Where a decision rule will be applied repeatedly, for instance a criterion for advancing compounds, its long-run error rates under plausible scenarios can be examined by simulation before the method is adopted. Published evaluations of that kind for preclinical dose-response settings are still uncommon, and guidance on default priors for particular assay types is thinner than for clinical trials. These are the areas in which further methodological work would be most useful to laboratories.

Hierarchical Bayesian models suit the conditions under which preclinical dose-response data are produced: few units, incomplete curves, nested measurements and a body of related experiments. They stabilise estimation by sharing information, represent the experimental design honestly and express uncertainty in the quantities that matter for decisions. Their results in small studies depend on assumptions about between-group variation and exchangeability that the data cannot fully test. Used with stated priors, sensitivity analysis and proper checks, they are a sound replacement for curve-by-curve fitting. Used as a black box, they move the fragility of small studies from the optimiser to the prior.

To be declared by the author(s) before publication.

To be declared by the author(s) before publication.