The seven stages are shown in Figure 1. They are presented in sequence, but the early stages are iterative: piloting almost always sends the researcher back to the codebook, and poor reliability should do so again.
Stages of the proposed coding procedure, with the return loop from reliability testing to the codebook
Good coding projects begin with a claim that could turn out to be false. "How do courts treat proportionality?" is a topic. "Courts that cite the structured proportionality test apply all of its limbs in most cases" is a claim with observable implications, which is what Epstein and King (2002) ask of any empirical legal argument. The scholar should write down the claim, the features of an opinion that would count for and against it, and any rival explanation worth recording. Questions about frequency, co-occurrence and change over time suit the method. Questions about what the law ought to be do not, though coded results may inform them.
The next task is to say which opinions the claim is about and how they will be found: the courts, the period, the database and the exact search terms or citation rule. Where the population is manageable, all of it should be coded; where it is not, a random sample drawn from a complete list is preferable to a selection by prominence (Epstein and Martin, 2014). Two selection effects need to be acknowledged in every project. Reported opinions are not a random draw from decisions, and their differences from unreported ones can be large (Siegelman and Donohue, 1990). Decided cases are not a random draw from disputes either, because parties settle the cases whose outcome is predictable (Priest and Klein, 1984). Conclusions should be stated as conclusions about the population actually coded.
The codebook is the central document of the project. For each variable it gives a name, a definition, the permitted values, a decision rule for hard cases and at least one worked example. Categories must be exhaustive and mutually exclusive, which usually means including "other" and "unclear" values and watching how often they are used (Neuendorf, 2017). Variables differ in how much judgement they demand, and Table 1 groups the kinds a doctrinal project typically needs. A sound rule is to code what the opinion states and to keep inference to a minimum: whether a court cited a given authority is easier to code reliably than whether it "really" relied on it. Outcome variables should be coded separately from reasoning variables, ideally at a different time, so that knowledge of the result does not colour the reading of the reasons.
Types of variable in a codebook for judicial opinions, with typical difficulties and safeguards
| Type of variable | Example | Typical difficulty | Safeguard |
|---|---|---|---|
| Identifying | Court, date, citation, panel size | Clerical error | Take from database fields; spot-check |
| Party and procedure | Type of claimant; stage of proceedings | Mixed or multiple parties | Rule for the dominant category |
| Outcome | Claim allowed, dismissed or allowed in part | Partial success; remittal | Define success from one party's side |
| Authority cited | Whether a named precedent or provision is cited | String citations; indirect reference | Code express citation only |
| Doctrinal step | Whether each limb of a test is addressed | Limbs merged or implicit | Worked examples; an "implicit" value |
| Reasoning style | Reliance on purpose, text or policy | High inference; coder disagreement | Two coders; report agreement |
Before any coding counts, the researcher applies the draft codebook to a small number of opinions taken from the population but excluded from the later reliability test. Piloting shows which definitions are ambiguous, which categories are never used and which facts cannot be found in the text. The codebook is revised and the changes are logged. Scholars working alone should still pilot, and should ask a colleague to code a few opinions cold, since the author of a codebook is the person least able to see its gaps.
Reliability is the extent to which different coders, working independently from the same instructions, record the same values. It is tested by having at least two people code the same subsample without conferring and comparing the results variable by variable. Simple percentage agreement flatters, because some agreement arises by chance; the usual corrections are the kappa coefficient of Cohen (1960) for two coders and nominal categories and the alpha coefficient of Krippendorff, which accommodates several coders, missing values and different levels of measurement (Hayes and Krippendorff, 2007). The benchmarks most often quoted are those of Landis and Koch (1977), who described kappa between 0.61 and 0.80 as substantial agreement, and of Krippendorff (2018), who advises relying on variables with alpha of at least 0.80 and drawing only tentative conclusions from those between 0.667 and 0.80. These are conventions and should be reported as such.
Where a variable falls short, the remedy is to clarify the definition, retrain and test again on fresh opinions, or to drop the variable. It is not to discuss disagreements until they disappear and then report the agreed figures. The coefficient, the size of the subsample and the procedure should appear in the published article, an expectation long standard in communication research (Lombard et al., 2002). Coders should not be told the hypothesis where that can be avoided.
Once reliability is acceptable the codebook is frozen and the full population is coded, with values entered in a spreadsheet or database in which each row is an opinion and each column a variable. The unit of analysis should be settled beforehand: the case, the majority opinion, each separate opinion, or each issue within an opinion. Long projects should include a second reliability check partway through, because coders drift. Every departure from the codebook, and every opinion excluded after retrieval, is recorded with reasons.
For most doctrinal purposes the analysis is modest: frequencies, cross-tabulations and trends over time, presented in tables a lawyer can read. Regression has its place when several factors must be considered together, but it requires care and usually a collaborator, and it adds nothing if the underlying coding is unsound. The report should describe the search, the population, the codebook and the reliability results in enough detail for another scholar to repeat the work, and the codebook and coded data should be made available, which is the replication standard Epstein and King (2002) urged on law reviews. Counts should then be read alongside the opinions themselves. The strongest studies quote and discuss cases to show what a category looks like in practice.