The cheapest operation is the one never performed, and the largest savings come from choosing a frugal architecture before any compression is applied. MobileNet replaced the standard convolution with a depthwise convolution followed by a pointwise one, cutting both operations and parameters, and exposed width and resolution multipliers that scale the network down smoothly (Howard et al., 2017). MobileNetV2 added inverted residual blocks with linear bottlenecks (Sandler et al., 2018). The depthwise separable networks that performed best in the keyword spotting study of Zhang et al. (2017) follow the same pattern, and most microcontroller vision and audio models today descend from these designs.

Pruning removes parameters that contribute little. The idea predates deep learning: optimal brain damage used second-derivative information to select weights for deletion (LeCun et al., 1990). Han et al. (2015) showed that simple magnitude pruning followed by retraining removes a large share of the weights of modern networks with little loss of accuracy, and the lottery ticket hypothesis of Frankle and Carbin (2019) gave the observation a theoretical framing. On a microcontroller the distinction between unstructured and structured pruning is decisive. Unstructured pruning leaves a sparse matrix whose storage needs indices and whose multiplication needs irregular memory access, so the saving in flash is partial and the saving in time is often nil on a processor without sparse support. Structured pruning deletes whole filters or channels (Li et al., 2017) and yields a smaller dense network that any runtime executes faster. A critical survey by Blalock et al. (2020) also cautions that pruning results are hard to compare across papers because baselines and metrics differ.

Quantisation stores weights and activations at reduced precision. Moving from 32-bit floating point to 8-bit integers cuts model size by a factor of four and, as important, allows inference with integer arithmetic alone, which suits cores that lack a floating-point unit. The affine scheme of Jacob et al. (2018), in which a real value is represented by an integer, a scale and a zero point, is the basis of the formats used by the mainstream tools. Quantisation can be applied after training using a small calibration set, or simulated during training so that the network adapts to it; Krishnamoorthi (2018) sets out the practical trade-offs, and Gholami et al. (2021) survey the wider literature. Below eight bits, accuracy becomes harder to hold. Binarised networks constrain weights and activations to two values (Courbariaux et al., 2016) and replace multiplication with bitwise operations, at a cost in accuracy that has limited their use to simple tasks.

Knowledge distillation trains a small student network to reproduce the softened output distribution of a large teacher, which often recovers accuracy that the student would not reach from labels alone (Hinton et al., 2015). It changes nothing at deployment and combines freely with the other methods. Deep compression chained pruning, weight sharing and entropy coding into a single pipeline (Han et al., 2016) and established the habit of stacking techniques that TinyML work has followed since.

Manual design gives way to automated search once the constraint is stated precisely. SpArSe searched jointly over architecture and pruning for networks that fit microcontroller memory (Fedorov et al., 2019). MCUNet coupled a search over a space tailored to the memory limit with an inference engine tuned for the resulting models (Lin et al., 2020), and MicroNets used differentiable search with operation count as a proxy for latency on Cortex-M devices (Banbury et al., 2021a). The shared lesson is that the search must optimise the quantities the hardware imposes, peak memory above all, and not parameter count.

TensorFlow Lite for Microcontrollers is the most widely used runtime. It is an interpreter that reads a serialised model, plans tensor memory in a single arena supplied by the application, and dispatches each operator to a kernel. Its design avoids dynamic allocation and operating system dependencies so that it builds for bare-metal targets (David et al., 2021). Interpretation adds a small fixed overhead; most of the execution time is spent in the kernels. On Arm cores those kernels are commonly provided by CMSIS-NN, a library of fixed-point convolution, pooling and activation routines that use the processor's DSP extensions where present (Lai et al., 2018).

The alternative to interpretation is compilation of the model into code specialised for it. TVM introduced a compiler stack with a tensor-level intermediate representation and automated tuning of operator implementations for a given target (Chen et al., 2018), and its microcontroller support emits C that links without a runtime interpreter. Silicon vendors provide generators of their own that take a quantised model and produce a C library for their parts. Code generation removes the interpreter and allows optimisations across layers, such as the in-place depthwise convolution used in MCUNet's engine. The price is a less transparent build and tighter coupling between model and firmware.

Comparing results was difficult while every paper used its own model, dataset and board. The MLPerf Tiny suite defines four tasks, keyword spotting, visual wake words, image classification and anomaly detection, with reference models and rules for reporting latency, accuracy and energy (Banbury et al., 2021b). The keyword spotting task uses the Speech Commands dataset (Warden, 2018). The suite has given toolchain developers a common yardstick, though its workloads are small and do not cover every sensing application.