Coding Voting Alignment in Multilateral Assemblies With Open Text-as-Data Tools

1. Introduction

Scholars have counted votes in the General Assembly almost since it first met. The appeal is plain: nearly every state takes a position on the same question at the same moment, year after year, and the record is public. Lijphart (1963) criticised the early studies of bloc voting and proposed an index of agreement that is still in use. Later work asked whether the alignments of the Cold War survived its end (Kim and Russett, 1996; Voeten, 2000), and voting similarity became a routine control variable in studies of aid, trade and conflict.

The difficulty is that the measure a researcher downloads embodies decisions that someone else made: which votes to include, how to treat an abstention, whether to correct for agreement that would occur by chance. Those decisions affect conclusions (Häge, 2011). They also tie the researcher to one assembly, when the Human Rights Council, the specialised agencies and regional parliaments raise the same questions (Hug and Lukács, 2014). And the standard measures say nothing about subject matter unless resolutions are first sorted by topic, a task that used to require reading each one.

Open tools for treating text as data have changed what a single researcher can do. This article proposes a stepwise procedure that joins roll-call analysis to automated text classification so that alignment can be coded transparently for any assembly that publishes its votes and documents. The procedure is described and justified; it is not applied here.

2. Background

The procedure draws on two established bodies of method. One concerns how similarity between voting records should be measured; the other, how political texts can be analysed at scale.

2.1. Measuring similarity in votes

The simplest measure is the share of votes on which two states make the same choice. Lijphart's index refines it by counting a pairing of an abstention with a yes or a no as half an agreement (Lijphart, 1963). Signorino and Ritter (1999) showed that rank correlation, then common in the field, confuses similarity with association, and proposed the S score, which is based on the distance between two states' positions scaled by the maximum possible distance. Häge (2011) pointed out that S and raw agreement are inflated when most states vote the same way on most resolutions, as they do in the General Assembly, and recommended chance-corrected coefficients of the kind used to assess agreement between coders.

A different family of methods estimates each state's position on an underlying dimension. Spatial models of roll-call voting were developed for legislatures (Poole and Rosenthal, 1985). Bailey et al. (2017) adapted the approach to the General Assembly with a dynamic model that uses resolutions repeated in identical form across sessions to make positions comparable over time, which dyadic measures cannot guarantee when the agenda changes. An overview of the data and of its uses is given by Voeten (2013).

2.2. Text as data

Grimmer and Stewart (2013) set out the principles that govern automated content analysis in political science: all quantitative models of language are wrong, some are useful, and none can be trusted without validation. Methods fall into those that assign documents to known categories, with dictionaries or supervised learning, and those that discover categories, such as topic models. Scaling methods place texts on a dimension using reference documents (Laver et al., 2003) or word frequencies alone (Slapin and Proksch, 2008). The corpus of General Debate statements assembled by Baturo et al. (2017) made such methods usable for the United Nations and offers a source of state positions independent of votes. Open software has matured alongside: the quanteda package handles corpus management and document features (Benoit et al., 2018), and the stm package fits structural topic models (Roberts et al., 2019).

3. The Procedure

Figure 1 shows the seven steps and the point at which the analyst may need to return to earlier choices. Each step is described below with the decision it requires.

Figure 1: The seven steps of the proposed procedure for coding voting alignment

3.1. Step 1: Define the assembly, period and unit

The analyst states which body is studied, over which sessions, and what the unit of comparison is: a pair of states in a session, a state relative to a reference state, or a state relative to a group. The rules of the body should be recorded at this point because they determine what a vote means. In the General Assembly each member has one vote, decisions on important questions need a two-thirds majority of members present and voting, and a member in arrears may lose its vote under Article 19 of the Charter.

3.2. Step 2: Collect vote records and texts

Two things are gathered for each recorded vote: the choice of every member, and the full text of the resolution with its symbol, title, date and agenda item. For the General Assembly both are published in the United Nations Digital Library, and the roll-call dataset maintained by Voeten and colleagues can serve as a check on the vote records (Bailey et al., 2017). The symbol is the key that joins votes to text. Votes on separate paragraphs or amendments should be kept and flagged, since they are often more divisive than the vote on the whole.

3.3. Step 3: Code each vote choice

Four outcomes are possible for a member on a recorded vote: yes, no, abstain and absent. The coding rule must say whether abstention is a middle category, as in Lijphart's index and the S score, or a soft no; and whether absence is treated as missing, which is the usual choice, or as informative where a state is known to stay away deliberately. A further rule is needed for the many resolutions adopted without a vote. Leaving them out overstates division, because recorded votes are requested on the contentious items; the general problem that roll calls are a selected sample of decisions was set out for legislatures by Carrubba et al. (2006). Including them as unanimous agreement pushes every similarity score towards its ceiling. The procedure asks the analyst to compute measures both ways and report the difference.

3.4. Step 4: Classify resolutions by topic

Topic classification is where the text tools enter. The resolution texts are loaded into a corpus, tokenised and reduced to a document-feature matrix. Preprocessing choices such as stemming and the removal of rare terms should be made deliberately and reported, since they can change the results of unsupervised models (Denny and Spirling, 2018). Two routes are then open. Where the categories are known in advance, for example the Middle East, disarmament, human rights and development, a dictionary or a supervised classifier trained on a hand-coded sample assigns each resolution. Where they are not, a topic model proposes groupings that the analyst labels after reading the most representative documents (Roberts et al., 2019).

Either way, a random sample is coded by hand by at least two readers and compared with the machine output, with agreement reported using a chance-corrected coefficient (Krippendorff, 2018). Resolutions repeated from year to year with little change should be identified by text similarity at this stage, as they are needed for any dynamic model.

3.5. Step 5: Compute alignment measures

No single measure is correct for every purpose, and the procedure asks for at least two. Table 1 compares the main options. A reasonable default is a chance-corrected dyadic coefficient for questions about pairs of states and an ideal point estimate for questions about change over time, each computed for all votes and separately within the topics from Step 4.

3.6. Step 6: Validate

Validation follows the advice of Grimmer and Stewart (2013) that every automated measure be checked against something outside itself. Three checks are proposed. Where a published benchmark exists, as it does for the General Assembly, the new measure is compared with it over the overlapping period and large discrepancies are traced to their source. The vote-based measure is compared with positions scaled from speeches in the same sessions; the two should be related without being identical, and persistent gaps between what a state says and how it votes are worth examining in their own right. Finally, the highest and lowest scoring pairs in each topic are inspected by a reader who knows the cases. If any check fails, the analyst returns to the coding rules.

3.7. Step 7: Document and release

The final step is to publish the vote file, the topic codes, the scripts and a codebook stating every rule adopted in Steps 1 to 5 and the software versions used. Since the inputs are public documents, there is rarely a legal obstacle to release. A measure that others can rebuild and alter is worth more to the field than a marginally better one that cannot be inspected.

4. Adapting the Procedure to Other Assemblies

The General Assembly is the easy case, because its records are complete and a benchmark exists. Elsewhere the analyst must adapt. In bodies with elected and rotating membership, such as the Human Rights Council, whose voting was studied by Hug and Lukács (2014), the set of voters changes each year, so dyadic measures are available only for the sessions in which both states sit, and ideal point models need observations that bridge one membership to the next. Where most business is settled by consensus, recorded votes may be too few to support any measure, and text-based positions from statements become the main source instead of a check. Documents in several languages should be analysed in one official language throughout to avoid treating translation differences as differences of substance.

5. Limitations and What Remains to Be Shown

This article proposes a procedure and does not demonstrate it. Before it could be recommended without reservation, three things would have to be shown on real data. The first is that the topic classification in Step 4 reaches acceptable agreement with human coders across sessions, since the vocabulary of resolutions drifts over decades. The second is that conclusions are stable under the alternative coding rules of Step 3; if a finding holds only when consensus decisions are excluded, it is a finding about contested votes and should be described as one. The third is that the topic-specific measures add information beyond the overall measure, which is the main reason for the extra work.

There are limits no validation will remove. A vote records a public position taken under instructions, group pressure and occasionally inducement, and it is evidence of alignment in the assembly and not of a state's private preferences (Voeten, 2013). Abstention and absence are strategic and their meaning varies by state. Automated classification of formulaic diplomatic prose can be confidently wrong. These are reasons to report choices openly and to read the documents, and the procedure is built to make both easier.

6. Conclusion

Voting alignment can be coded from public records with open software by any researcher prepared to make a small number of decisions explicit. The seven steps set out above pair the established measures of vote similarity with text-based classification of what is being voted on, and they place validation and release inside the workflow instead of after it. The procedure awaits application. Its promise is a family of alignment measures that are specific to topic, portable across assemblies and open to correction by those who use them.

Conflict of interest

To be declared by the author(s) before publication.

Funding

To be declared by the author(s) before publication.