Abstractive summarisation changed character when sequence-to-sequence networks with attention were shown to produce fluent summaries of news articles (See et al., 2017), and it changed again when pretrained Transformers arrived. Models such as BART and PEGASUS are pretrained on large text collections with denoising or gap-sentence objectives and then fine-tuned on document and summary pairs (Lewis et al., 2020; Zhang et al., 2020). On short inputs they are hard to improve on. Their weakness is the length of what they can read.

The Transformer encoder relates every token to every other token (Vaswani et al., 2017). That design is the source of its strength, and it also fixes a practical ceiling: BERT accepts 512 word-piece tokens and BART 1,024 (Devlin et al., 2019; Lewis et al., 2020). A research article or a court decision runs to many thousands. The usual workaround is to truncate, which was tolerable for news, where the lead paragraph carries most of the content, and is much less so for documents whose conclusions sit at the end.

Sparse attention is the family of techniques that lets each token attend to a chosen subset of positions, so that cost grows roughly linearly with length. Most of the published work reports experiments on data-centre accelerators. Many of the people who would benefit from summarising long documents, in universities, small firms, newsrooms and public offices, have a single consumer graphics card or a laptop. This review reads the literature with that reader in mind and asks which ideas survive the move to ordinary equipment.

This is a narrative review of the published literature on efficient attention as it bears on summarising long documents. It draws on the original papers that introduced the main mechanisms, on the summarisation models built from them, and on surveys and benchmark studies that compare them. It is not a systematic review and makes no claim to completeness; the selection favours work that is widely cited and whose methods are publicly implemented, since those are the methods a practitioner can use. No new experiments are reported. Where the article speaks of commodity hardware it means a machine with one consumer graphics card, or none, and the memory that such machines normally carry.

In a self-attention layer each of the n input tokens produces a query, a key and a value, and the layer computes a score for every query and key pair. The score matrix therefore has n × n entries per head, and it must be held in memory during training so that gradients can pass back through it. Doubling the input quadruples the matrix. Parameters do not grow with length, so the model itself fits comfortably; it is the activations that exhaust the card.

Two consequences follow for a user with limited memory. Training is harder than inference, because training stores activations for every layer while inference can discard them as it goes. And the encoder is the problem, not the decoder: a summary is short, so decoder self-attention is cheap, and the cross-attention from summary to document grows only linearly with document length. Most sparse designs accordingly leave the decoder alone and change only how the encoder reads.

The survey by Tay et al. (2022) sorts efficient Transformers by the device each uses to avoid the full score matrix. For the purpose of summarisation four families are enough, and Figure 1 lays them side by side. The first two are sparse in the strict sense, since most pairs of positions are never scored. The third approximates the dense matrix instead of thinning it, and the fourth restructures the input so that no single attention call sees the whole document.

Three encoder-decoder models carried sparse attention into summarisation proper. The Longformer Encoder-Decoder replaces the encoder attention of BART with the windowed and global pattern and is initialised from BART's weights, with position embeddings copied to cover the longer input (Beltagy et al., 2020). BigBird was paired with a PEGASUS decoder in the same spirit (Zaheer et al., 2020). LongT5 adds to the T5 architecture a transient global mechanism, in which blocks of tokens are summed into temporary global tokens at each layer, and adopts the gap-sentence pretraining objective of PEGASUS (Guo et al., 2022). Table 1 summarises how these and the other approaches discussed here behave from the standpoint of a memory-limited user.

The most direct comparison for summarisation is that of Huang et al. (2021), who introduced a corpus of long government reports and examined several efficient encoder attentions under a fixed memory budget. Their argument is worth stating plainly, because it frames the whole subject: an efficient attention is valuable chiefly because it lets the model read more of the document in the same memory, and reading more of the document is what improves the summary. They also point out that encoder-decoder attention becomes a cost of its own once inputs are very long, and propose a strided variant to contain it.

A second message from the comparative literature is one of modesty. The Long Range Arena benchmark evaluated many efficient Transformers on a common suite of long-sequence tasks and found no single mechanism that led on all of them, with differences in quality often smaller than differences in speed and memory (Tay et al., 2021). For a practitioner this shifts attention from the elegance of the pattern to the availability of a pretrained checkpoint and a stable implementation.

Asymptotic complexity describes how cost grows, not where it starts. A linear method with a large constant can be slower than exact attention at the lengths a small card can hold in any case. The clearest demonstration is FlashAttention, which computes exact attention but organises the work in tiles so that the score matrix is never written out in full, bringing memory use down to linear in sequence length and reducing traffic between slow and fast memory on the accelerator (Dao et al., 2022). The lesson is that part of the problem sparse attention was invented to solve is an artefact of how attention was implemented.

Other levers are independent of the attention pattern and combine with it. Half-precision arithmetic and gradient checkpointing reduce training memory at some cost in time. Accumulating gradients over several small batches stands in for a large one. For inference, weights can be quantised to integers (Jacob et al., 2018), and a smaller student can be trained to imitate a large summariser by knowledge distillation (Hinton et al., 2015). None of these is specific to long documents, but on a small machine they often decide whether a long-input model can be fine-tuned at all.

There is also a case for restraint that goes beyond convenience. Strubell et al. (2019) drew attention to the energy and financial cost of training large language models and to the way those costs restrict who can take part in research. Methods that bring long-document summarisation within reach of ordinary equipment widen participation, and reports that state memory use and running time alongside quality scores make it possible for others to judge whether a method is within their means.

Nearly every paper cited here reports ROUGE, which counts the overlap of word sequences between a system summary and a reference (Lin, 2004). ROUGE is cheap and reproducible, and it rewards content selection reasonably well. It does not check whether a statement in the summary is supported by the source. Maynez et al. (2020) showed that abstractive systems frequently produce content that cannot be inferred from the input, and that overlap metrics correlate poorly with human judgements of faithfulness.

Long inputs sharpen the difficulty. A human judge must read the whole document to verify a summary, which is slow and expensive, so human evaluation of long-document systems is rare and usually small. The reference summaries themselves are often abstracts written by the authors for a different purpose. A reader comparing attention mechanisms should therefore treat small differences in ROUGE with caution, especially when the systems compared also differ in pretraining, input length or decoding settings.

Several questions are left open by the literature as it stands. Comparisons at equal memory, of the kind Huang et al. (2021) carried out, remain uncommon, and comparisons at equal memory on consumer cards are rarer still. It is not well established how much of the benefit of a long-input model comes from the attention pattern and how much from simply seeing more text, nor how sparse models compare with a careful extract-then-abstract pipeline when both are given the same budget. Faithfulness metrics that work on inputs of many thousands of tokens are needed before claims about quality can be settled.

Languages other than English are thinly served: the long-input checkpoints that a practitioner can download were mostly pretrained on English text, and the summarisation corpora of long documents are concentrated in scientific, legal and governmental English. Work on efficient attention for morphologically rich or low-resource languages, and on documents with tables and figures, would extend the reach of the methods reviewed here. Finally, as exact attention becomes cheaper through better implementations, the boundary at which sparsity is worth its approximation should be measured again.

The literature on sparse attention offers a user of modest hardware more than one workable route to summarising long documents. Windowed attention with a few global tokens stands out for practical reasons: it is simple, its memory use is predictable, and it can inherit the weights of an existing summariser. More elaborate mechanisms have not shown a consistent advantage that would justify their complexity at the lengths small machines can handle. Implementation choices, precision, and the decision to chunk or extract first weigh as heavily as the attention pattern. The weakest part of the evidence concerns evaluation, and progress there would benefit every method discussed.

To be declared by the author(s) before publication.

To be declared by the author(s) before publication.