Definitions overlap without coinciding. Alexander (2008) characterises dialogic teaching by principles: it is collective, reciprocal, supportive, cumulative and purposeful. Nystrand et al. (1997) operationalised dialogic instruction through authentic questions, to which the teacher does not hold a prescribed answer, and uptake, in which a question incorporates what a student has just said. Mercer and Littleton (2007) describe exploratory talk among students, in which reasons are given and challenged. The accountable talk framework specifies accountability to the learning community, to standards of reasoning and to knowledge (Michaels et al., 2008). A review of four decades of research concluded that the field has described patterns of dialogue far more than it has measured their consequences (Howe and Abedin, 2013). A rubric designer must therefore choose among accounts and say which has been chosen.

In a large class the arithmetic of turn-taking sets a ceiling on whole-class dialogue. If one person speaks at a time, most students will not speak in a given lesson however well the teacher questions. Teachers respond with choral answers, with brief pair talk, and with group tasks followed by reports from a few groups. Studies of classroom interaction in sub-Saharan Africa, where large classes are common, describe lessons dominated by recitation and chorus and argue that realistic improvement has to start from those conditions (Hardman et al., 2008). An instrument that rewards only extended whole-class exchanges will place nearly every large-class lesson at the bottom of its scale and will fail to distinguish a teacher who has students reasoning in pairs from one who lectures.

An observation score varies with the teacher, and also with the lesson, the observer and the segment observed. Hill et al. (2012) showed that agreement between raters is an incomplete test of an observation system and that a generalisability study is needed to learn how many lessons and raters a dependable score requires. Validity is treated in current thinking as an argument: the intended use of scores is stated, and evidence is assembled for each inference on which that use depends (Kane, 2006). Bell et al. (2012) applied this approach to observation protocols. Reviews of scoring rubrics add that analytic rubrics with clear descriptors, exemplars and rater training are more reliable than holistic judgements (Jonsson and Svingby, 2007).

The designer writes a definition of one paragraph and derives from it a small number of dimensions. Four are suggested here, shown with illustrative indicators in Figure 1: teacher moves, student contributions, participation structure and classroom climate. The third dimension is the one that conventional instruments lack. It asks whether the lesson is organised so that many students have the chance to talk about the subject matter, and whether what is said in pairs and groups is brought back into the public discussion. Including it makes structured peer talk part of the construct, consistent with the place of exploratory talk in the research (Mercer and Littleton, 2007).

Classical interaction analysis codes classroom events at intervals of a few seconds, which yields fine-grained data and exhausts a live observer. A rubric is better served by longer segments, each rated once. The proposal is to divide the lesson into segments of roughly ten minutes and to label each by its organisation: whole-class teaching, pair or group work, or individual seatwork. Dimensions are rated only where they apply. This prevents a lesson with substantial group work from being penalised on indicators that presuppose whole-class exchange, and it produces a simple profile of how lesson time was used.

Each indicator must pass a practical test: an observer standing at the side or back of a crowded room, without a transcript, has to be able to tell whether it occurred. Indicators concerning the teacher usually pass, since the teacher is audible. Whether a question is open, whether a follow-up builds on the student's answer, and whether the teacher restates a contribution so that the class can respond to it, the move described as revoicing by O'Connor and Michaels (1993), can all be judged live. Indicators concerning students need rewording. "Students give extended reasons" becomes observable if it is tied to the contributions made publicly and, during group work, to the groups the observer is sampling. Indicators that depend on hearing every student should be removed, and the omission recorded as a limit of the instrument.

For each dimension the designer writes three or four levels, each a short description of what the observer would see. Two cautions apply. Levels should describe quality and spread and should avoid raw counts, because the number of students who speak is bounded by class size and lesson length. A descriptor such as "contributions come from different parts of the room and include students who did not volunteer" captures spread without a count. And the top level should describe practice that a capable teacher can achieve in a large class. If no lesson in the intended setting could reach it, the scale has a level nobody will use, and the remaining levels will discriminate poorly.

Three sampling decisions are written into the observer's manual. The first is how many lessons are observed for each teacher and across which topics, since a single lesson is a weak basis for any statement about a teacher's practice. The second is where the observer is placed during whole-class teaching, ideally where both the teacher and a cross-section of students can be heard. The third, particular to large classes, is how groups are chosen during group work. Left to choice, observers drift towards the nearest or liveliest group. A rule fixed in advance, for example the group nearest each of three predetermined points in the room for a set number of minutes each, gives a defensible sample. An audio recorder on a sampled desk can supplement live notes where consent and resources allow.

Observer training uses video or live lessons scored independently and then discussed until descriptors are applied consistently. Agreement is reported with a chance-corrected statistic such as kappa (Cohen, 1960), and a small generalisability study estimates how much score variance is due to teachers, lessons, raters and segments (Hill et al., 2012). The validity argument follows Kane (2006). Experts in classroom dialogue review whether the indicators represent the construct; rubric scores for a subset of lessons are compared with detailed coding of the same lessons from recordings; and the relation between scores and class size is examined, because a rubric whose scores fall steadily as classes grow is measuring conditions more than teaching.