Public argument about what recommendation systems show people has outrun the means of finding out. Platform operators hold the logs. Outside researchers can ask users to donate their data, can observe their own feeds, or can construct accounts that behave in a controlled way and record what those accounts are served. The last approach, the sock-puppet audit, is one of the designs set out in the foundational typology of Sandvig et al. (2014), and it has become a standard tool for studying personalisation (Bandy, 2021; Metaxa et al., 2021).

Most of the method's conventions were developed on search engines and on long-form video, where a user expresses interest by typing a query, clicking a result or choosing a video. A short-video feed works differently. Content plays automatically in a single stream, and the system infers interest chiefly from how long each item is watched before the user swipes on. An audit therefore has to manipulate dwell time with care, and many more items pass in a session, so the volume of material to be classified is large. Published audits of these feeds exist, but each has made its own choices about accounts, scripts and controls, and the choices are not always reported in enough detail to repeat.

This article proposes a design that can be repeated. It is a protocol, with reasons, for researchers in journalism and communication who want to ask what a short-video feed serves to accounts with different interests in news and public affairs. The protocol has not been executed for this article. It is assembled from published practice and stated as a proposal to be tested.

The design draws on the general logic of algorithm audits, on the experience of earlier sock-puppet studies, and on what is publicly known about how short-video recommendation behaves.

The protocol has eight steps. Figure 1 shows their order and Table 1 states, for each, the main decision to be taken and what must be reported for another team to repeat the work.

Step 1 fixes the question and the outcome. A question such as whether accounts that linger on election content are subsequently served a larger share of content from unverified sources than matched controls names a treatment, a comparison and a measurable property of feed items. Vague aims, for example to see what the algorithm does, cannot be audited. Step 2 specifies the personas. Each treatment persona is defined by the content categories it dwells on, and each is matched with at least two control accounts that share its location, language, device profile and creation time: one idle control that watches every item for the same short duration, and one duplicate of the treatment persona. The duplicate supplies the noise floor in the manner of Hannak et al. (2013).

Step 3 builds the environment. Short-video platforms are used chiefly through mobile applications, and a browser session may not receive the same feed. The protocol therefore prefers physical handsets or emulated devices running the application, each with a fresh installation, no contact list and a network location that matches the persona's declared country. The application version, operating system, location method and every step of account creation are recorded. Step 4 writes the behaviour scripts. A script needs a rule for deciding, as each item arrives, whether it belongs to the persona's interest category; the rule can use captions, hashtags and sounds, and it should be simple enough to publish. Items that match are watched to the end, others are skipped after a fixed short interval, and small random variation is added to both. Likes, follows and searches are either excluded or introduced as separate conditions, so that their effects are not confounded with dwell time.

Step 5 pilots, registers and runs. A pilot with a few accounts exposes practical failures: login challenges, interstitial screens, scripts that misclassify. After the pilot the design is registered with its hypotheses and analysis plan, in keeping with the transparency standards proposed for empirical research generally (Nosek et al., 2015). All accounts in a replicate then run at the same times of day over the same days, since feeds respond to current events. Step 6 captures the feed: for every item, its identifier, position in the session, timestamp, creator, caption, hashtags, sound, and visible engagement counts, with the video file or a screenshot archived where storage and law allow, because items are frequently removed.

Step 7 annotates. Whatever property the outcome depends on, such as topic, source type or accuracy rating, is coded from a written codebook by at least two trained coders working independently on a common subset, and agreement is reported with a chance-corrected coefficient such as Krippendorff's alpha (Krippendorff, 2018). Automated classifiers may be used to extend the coding once they have been checked against the human-coded subset. Step 8 analyses and releases. The quantity of interest is the difference between treatment and idle control in the share of feed items with the coded property, and it is interpreted only in relation to the difference observed between duplicate accounts. Scripts, configuration, codebook, item identifiers and analysis code are deposited so that the audit can be rerun.

Four threats deserve explicit treatment in any report. The first is ecological: a puppet that watches one category with perfect consistency is unlike any person, and results describe the system's response to an extreme input (Chen et al., 2023; Hosseinmardi et al., 2024). Personas can be made less artificial by mixing interests, at the cost of a weaker treatment. The second is detection. Platforms act against automated accounts, and an account that has been flagged may be served a degraded or atypical feed without notice; sudden changes in a puppet's feed relative to its duplicate are a warning sign. The third is temporal. Recommendation models, content supply and moderation rules change continually, so an audit is a dated observation, and the systematic review by Bandy (2021) notes how few audits are repeated. The fourth is measurement: an outcome that depends on unreliable coding cannot support a conclusion however clean the experimental design.

Creating accounts under invented identities and collecting data by automated means are commonly prohibited by platform terms of service, although such terms are often ambiguous and unevenly enforced (Fiesler et al., 2020). Researchers should take advice on the law of their jurisdiction before starting. In the European Union the Digital Services Act creates routes for vetted researchers to obtain data from very large platforms and addresses research use of publicly accessible data (European Union, 2022), which may in time reduce the need for puppets but does not replace the independent check that an external audit provides. Ethical review should cover the interests of the creators whose videos are collected, who have not consented to study. The guidelines of the Association of Internet Researchers recommend proportionate collection, secure storage and care in republishing identifiable material (franzke et al., 2020). Puppets should not post, comment or message, and should be few enough that they do not distort the engagement counts of small creators.

The protocol is proposed and unvalidated. A validation would need to establish at least three things. Independent teams following the protocol with the same registered design at the same time should obtain compatible estimates, which would show that the reporting requirements are sufficient. Differences between duplicate accounts should be small relative to differences between treatment and control, which would show that the noise floor is usable in this setting. And feeds served to emulated devices should be compared with feeds on physical handsets, since the equivalence is assumed here and not demonstrated. The protocol also leaves open how many accounts and sessions are enough. That depends on the variability of the feed, which only a pilot on the platform in question can reveal.

Sock-puppet audits remain one of the few ways for outsiders to test claims about what recommendation feeds serve. On short-video platforms the method must be adapted to a system that learns mainly from watching time, shapes a feed from the first moment of an account's life and changes quickly. The design proposed here responds with matched idle and duplicate controls, precisely scripted and published dwell rules, a documented device environment, registration after piloting, reliability-tested annotation and full release of materials. Its worth will be settled by teams who run it and report where it fails.

To be declared by the author(s) before publication.

To be declared by the author(s) before publication.