Designing and Evaluating a Reader Membership Trial in a Regional Newsroom After Advertising Revenue Collapse: A Methodological Framework

1. Introduction

The business of regional news was built on local advertising: classified notices, property, motoring, recruitment and the display space bought by nearby retailers. Most of that income has moved to search engines, social platforms and specialist listing sites, and the consequences for local reporting are well documented. Titles have closed or merged, newsrooms have shrunk, and areas without any regular local coverage have multiplied (Abernathy, 2018; Nielsen, 2015). Pickard (2020) treats the collapse as a structural market failure and not a passing adjustment.

Reader revenue is the response most often recommended to publishers who remain. Paywalls were the first form, and their record for general news at regional scale is modest (Myllylahti, 2014; Chyi and Ng, 2020). Membership, in which readers are asked to support journalism that stays wholly or largely open and are offered a closer relationship in return, has been promoted as better suited to publications with a strong local identity and a small potential market (Hansen and Goligoski, 2018). What a newsroom seldom has is a way of launching such a scheme so that it can later say, with evidence, what happened and why.

This article proposes one. It describes a six-phase framework for designing, running and evaluating a membership trial in a regional newsroom. The framework is assembled from published guidance and from standard thinking on experiments in organisations. It is offered as a proposed design. It has not been applied in a newsroom for the purposes of this article, and no results are reported.

2. Background

Three bodies of work inform the design: the distinction between membership and other forms of reader payment, research on who pays for news and why, and the methods available for learning from a single organisation's experience.

2.1. Membership, subscription and donation

The Tow Center guide by Hansen and Goligoski (2018) separates three models. Subscription is a transaction: payment buys access to a product. Donation is a gift to a cause, with nothing expected in return. Membership sits between them. Members give money, and often time, knowledge or contacts, because they believe in the organisation's purpose, and the organisation in turn opens itself to them. The Membership Puzzle Project developed this idea into practical guidance and stressed that the relationship must be designed before the payment page (Membership Puzzle Project, 2020). The underlying notion is close to what Lewis et al. (2014) call reciprocal journalism, in which journalists and audiences exchange benefits over time and trust accumulates through the exchange.

2.2. What is known about paying for local news

Comparative survey work finds that only a minority of people in most countries pay for online news, that payment is associated with age, interest in news and an existing habit of paying for print, and that those who do not pay often say free alternatives are sufficient (Fletcher and Nielsen, 2017). A systematic review of the drivers of willingness to pay reaches similar conclusions and notes how much of the evidence rests on stated intention and not on observed behaviour (O'Brien et al., 2020). Studies of regional newspapers add two cautions. Introducing a paywall can reduce traffic and with it the title's civic reach (Olsen et al., 2020), and stated intent to pay for a digital edition is weak when the same content has long been free (Chyi, 2012). A newsroom planning membership should therefore expect a small core of supporters, should measure what people do and not what they say, and should watch for effects on readership as well as revenue.

2.3. Learning from one organisation

A regional title has one market and one launch. It cannot randomise towns. Two traditions help. Within a website, visitors can be allocated at random to alternative versions of a message or page, and the conditions under which such online controlled experiments give trustworthy answers are well described (Kohavi et al., 2020). Where randomisation is impossible, quasi-experimental designs such as interrupted time series and staggered introduction offer weaker but still disciplined inference, provided the threats to validity are named in advance (Shadish et al., 2002). Qualitative evidence from interviews and documents is best organised as a case study with a stated protocol (Yin, 2018).

3. The Proposed Framework

The framework has six phases, shown in sequence in Figure 1. Table 1 lists what each phase is for and the record it should leave. The phases are described below in the order a newsroom would meet them. The durations a newsroom assigns to each will depend on its size; the order should not change.

Figure 1: The six phases of the proposed membership trial, from baseline audit to formal review

3.1. Phase 1: Baseline audit

Before anything is offered, the newsroom records where it stands. The audit covers revenue by source for at least the preceding two years, so that seasonal patterns are visible; audience measures, separating occasional visitors from the regular local readers who are the realistic market for membership; existing direct relationships such as newsletter lists and print subscribers; and the cost of the journalism the scheme is meant to sustain. Picard (2011) gives the standard account of how media revenues and cost structures should be analysed. The audit fixes the comparison against which later change is judged, and it often shows that the newsroom does not know how many loyal readers it has.

3.2. Phase 2: Listening

Published guidance is consistent that membership fails when it is designed from the publisher's needs alone (Membership Puzzle Project, 2020). The listening phase combines a short survey of newsletter recipients and regular visitors, semi-structured interviews with readers chosen to differ in age, location and attachment to the title, and one or more open meetings. The questions concern what readers value in the coverage, what they miss, how they would want to be involved and what they would regard as a fair request. Stated willingness to pay is recorded and treated with reserve, for the reasons given above. The output is a written summary agreed by editorial and commercial staff.

3.3. Phase 3: Offer, hypotheses and stopping rules

The offer is now written down: the appeal, the price points, what members receive and what remains open to all. Alongside it the newsroom writes its hypotheses in plain terms, for example that an appeal based on the title's civic role will convert more regular readers than one based on member benefits, or that members who attend an event will renew at a higher rate than those who do not. It also writes the rules by which it will judge the trial: the level of recurring income by a stated date that would justify continuing, the level below which the scheme would be redesigned or closed, and the editorial time it is prepared to spend. Fixing these before launch is the single most important discipline in the framework, because it prevents the result from being reinterpreted to suit the hopes of those who ran it.

3.4. Phase 4: Staged launch with planned comparisons

The launch is staged so that comparisons are possible. On the website, regular visitors can be allocated at random to alternative appeal messages, and the allocation and outcome logged. Guidance on online experiments warns against stopping a test as soon as a difference appears, against running many variants on little traffic, and against judging on clicks when the outcome of interest is payment (Kohavi et al., 2020). A regional site has limited traffic, so the newsroom should test few alternatives, each expressing a real difference in approach, and accept that small effects will not be detectable. Channels that cannot be randomised, such as a print appeal or an editor's letter, are introduced at different times so that their effects can be distinguished in the daily series of joins.

3.5. Phase 5: Retention by cohort

A launch produces a burst of goodwill. What sustains a newsroom is renewal. Members are therefore grouped by the month in which they joined and each group is followed separately, since an overall retention figure hides the difference between early enthusiasts and later joiners. Fader and Hardie (2007) show why retention observed in the first periods of a contractual relationship should not be projected forward as a constant rate. Reasons for leaving are collected at cancellation, and participation is recorded alongside payment: attendance at events, responses to call-outs, contributions of knowledge to reporting.

3.6. Phase 6: Review

At the date fixed in Phase 3 the newsroom compares outcomes with its rules and records a decision to continue, change or stop. The review draws on the quantitative record and on a second round of interviews with members, lapsed members and staff, analysed against the case study protocol (Yin, 2018). It should state what cannot be concluded as plainly as what can.

4. Measures and Their Interpretation

Four families of measure are proposed. Conversion is the proportion of the loyal audience identified in the audit who join, and not the proportion of all visitors, which flatters or condemns a scheme according to how much passing traffic the site happens to receive. Revenue is recurring income net of payment charges and the direct costs of serving members. Retention is measured by cohort as described. Participation and cost capture what makes membership different from subscription: how many members do something other than pay, and how many staff hours the relationship takes. A scheme that covers its cash cost while absorbing a reporter's time each week has not yet been shown to sustain journalism. Readership of open content should be tracked throughout, given the evidence that payment barriers can reduce local reach (Olsen et al., 2020).

5. Safeguards

Two safeguards belong in the trial plan. Reader data gathered for the trial should be limited to what the measures require, held under the data protection law that applies, and used for testing only in ways readers have been told about. And the terms of membership should state that payment buys no influence over coverage. Closer relations with supporters are the purpose of the model and also its hazard, particularly in small communities where members may be the subjects of reporting (Hess and Waller, 2017).

6. Limitations and What Remains to Be Shown

The framework is untested. Its claim is that a newsroom following it will end with better grounds for its decision than one that does not, and that claim is itself open to examination, for instance by comparing the quality of the review documents produced by titles that adopt the framework with those of titles that launch conventionally. Several weaknesses are already apparent. A single title in a single market yields results that may not transfer, and the general lesson of the willingness-to-pay literature is that context matters a great deal (O'Brien et al., 2020). Traffic on regional sites limits what randomised tests can detect. The staggered introduction of channels gives only quasi-experimental evidence and is exposed to coincident events, such as a major local story, which the review must weigh (Shadish et al., 2002). Finally, the framework asks for record-keeping that a depleted newsroom may find hard to maintain, and the burden should be judged honestly in Phase 1.

7. Conclusion

Membership is asked to carry a great deal of hope in regional journalism. The framework proposed here does not make it more likely to succeed. It makes success or failure legible, by fixing a baseline, listening before designing, committing to rules before launch, staging the launch so that alternatives can be compared, following members beyond the first payment, and reviewing against what was written down. Whether newsrooms can apply it under the pressures they face is a question for those who try, and their documented experience would be a useful addition to a literature that still has more advocacy than evidence.

Conflict of interest

To be declared by the author(s) before publication.

Funding

To be declared by the author(s) before publication.