ASR Deployment on Edge Devices

Working Group Task Plan – DRAFT

Draft for group discussion

1. Purpose

Our white paper already covers ASR at survey depth: architectures, datasets, benchmarks, and a general note on edge deployment strategies. What it does not answer is the question we are actually asked in practice — given a specific device, which ASR model can run on it, and what has to be done to the model first?

This task is intended to close that gap. The scope is deliberately narrower than the white paper section and deeper: we want to characterise ASR models and hardware platforms on the same set of axes, so that the two can be matched against each other systematically rather than case by case. The output should be usable as a decision aid, not only as a literature review.

2. Objective and Main Deliverable

The main deliverable is a model × platform deployment matrix. For each ASR model we survey and each hardware class we survey, the matrix states whether the model is:

  • deployable as-is;
  • deployable after a specific treatment (quantization, pruning, distillation, architectural modification);
  • or out of reach for that platform class,

together with the expected cost in accuracy, latency, memory footprint and power.

Supporting deliverables:

  • Model requirement database — structured spec sheet per ASR model (Task 1).
  • Hardware capability database — structured spec sheet per platform class (Task 2).
  • Methods survey — techniques for fitting a model onto a target, with their measured cost (Task 3).
  • Feasibility methodology — the documented rules used to fill the matrix (Task 4).
  • Common reporting template — so that all contributors report the same fields (Task 5).
  • Reference deployment — one model taken end-to-end onto one device, to validate the methodology (Task 6).

3. Task 0 — Scope Definition

I propose we settle these points in the first meeting. Everything downstream depends on them, and if we leave them open the databases will not be comparable across contributors.

  • Use-case profiles in scope. Command and control, dictation, long-form transcription, multilingual. Each has a different accuracy floor and latency budget, so I suggest we treat them as separate target profiles rather than as one generic “ASR”.
  • Pipeline boundaries. Does “ASR system” include VAD and endpointing, the feature front-end, an external language model, punctuation and inverse text normalisation? This matters more than it looks: on small targets the output projection and the LM often dominate memory more than the encoder does.
  • Streaming vs offline, and the definition of latency. We should fix one set of definitions — algorithmic lookahead, first-token latency, real-time factor, endpointing delay — because published numbers mix them freely and are not comparable as reported.
  • Accuracy acceptance criteria and canonical test sets. Which WER target per profile, measured on which sets (e.g. LibriSpeech test-clean / test-other, Common Voice, an internal noisy set).
  • Hardware classes in scope. How far down we go (MCU-class) and how far up (application-processor / edge SoC).

4. Task 1 — ASR Model Landscape

Goal: a structured spec sheet per model. This is not a literature review — the fields are chosen so that they can be joined mechanically against the hardware sheet produced in Task 2.

Fields to record per model

  • Architecture family and streaming mode: CTC, RNN-T / transducer, attention encoder–decoder, Conformer / Zipformer / FastConformer, Whisper-class, SSL encoders (wav2vec 2.0, HuBERT, WavLM), and small end-to-end models such as Moonshine.
  • Parameter count broken out by block — encoder, predictor, joiner, output projection — not only the total.
  • MACs per second of audio at the target frame rate. This, rather than MACs per inference, is the number that meets the hardware throughput figure.
  • Peak activation memory and persistent state (KV cache, convolution cache, RNN state) as a function of context length.
  • Operator inventory — relative-position attention, GLU, LayerNorm, depthwise convolution, dynamic shapes. In practice this is what blocks NPU deployment more often than raw size does.
  • Decode-side cost: beam width, vocabulary and tokenizer size, language-model type and size.
  • Reported baseline WER on the agreed test sets, licence, and availability of weights and training recipe.

Output

A single shared spreadsheet (the model database), with one row per model and one column per field above, plus a short written summary per model family for the white paper.

5. Task 2 — Hardware Platform Landscape

The same discipline applied to the target side, on mirrored axes. We should cover the tiers from MCU-only, through MCU with a micro-NPU and audio DSP, up to mid-range NPU, mobile SoC and edge SoC.

Fields to record per platform

  • Peak MACs/s and realistic sustained utilisation for this class of workload — peak alone will overstate what is achievable.
  • Supported numeric formats: INT4 / INT8 / INT16, FP16 / BF16, per-channel vs per-tensor scaling, accumulator width.
  • Memory hierarchy: on-chip SRAM tiers and sizes, presence of external DRAM, and memory bandwidth. ASR decoders have low arithmetic intensity and frequently end up bandwidth-bound, so a compute-only screen will give us false positives.
  • Operator coverage, graph partitioning behaviour, and the cost of falling back to the CPU for unsupported operators.
  • Sparsity support (structured 2:4, block sparsity) and weight streaming from flash.
  • Toolchain maturity: ONNX / TFLite ingest, quantizer, calibration flow, debug and profiling support.
  • Power envelope, energy per inference, sustained thermal behaviour, and whether ASR is expected to share the accelerator with other workloads.

Output

The platform database, using field names that map one-to-one onto the model database where the two must be compared (memory against memory, MACs/s against MACs/s, operator list against operator support).

6. Task 3 — Model Fitting and Optimization Methods

A survey of the techniques available for closing the gap between a model and a target. For each method we should record the same four things, so that they can be compared and combined:

  1. typical compression or speed-up achieved;
  2. typical accuracy cost (WER delta);
  3. hardware prerequisite — what the target must support for the method to pay off;
  4. tooling maturity and engineering effort.

Methods to cover

  • Quantization — post-training vs quantization-aware training, weight-only vs full, mixed precision, sensitivity analysis, and the known-fragile points (attention scores, LayerNorm, output projection).
  • Pruning and sparsity — structured (channels, heads, whole layers) vs unstructured; note that unstructured sparsity only pays off if the target can exploit it.
  • Knowledge distillation — large or foundation model to a small student, and where this outperforms training a small model from scratch.
  • Low-rank factorisation — of the output projection and feed-forward blocks; LoRA-style adapters for on-device personalisation.
  • Architecture-level changes — frame-rate reduction and subsampling, chunked or linear attention, U-shaped encoders, replacing attention with state-space blocks.
  • Decode-side reduction — beam width, blank-skipping in transducers, LM pruning and quantization, tokenizer and vocabulary size.
  • Runtime and compiler optimisation — operator fusion, memory planning, tiling, double buffering, DVFS.
  • Constrained NAS and efficiency-aware training — architecture search under latency, memory or energy budgets.

7. Task 4 — Mapping Methodology

This is the core of the task and the part that turns two databases into a usable result. We need to define an explicit feasibility function that takes a model requirement vector and a platform capability vector and returns: deployable as-is, deployable with treatment X, or infeasible.

Proposed screens

  • Weight memory against available flash / on-chip storage.
  • Peak activation plus persistent state against available SRAM.
  • Required MACs/s against sustained throughput, at the target real-time factor.
  • Operator coverage, including the cost of CPU fallback where coverage is partial.
  • Power and thermal budget.
  • A roofline / arithmetic-intensity check, to catch the memory-bound cases that pass the throughput screen but fail on device.

Output

The deployment matrix, with each cell annotated by the required treatment and the predicted accuracy, latency and memory outcome, plus a written methodology note describing the assumptions behind each screen.

8. Task 5 — Evaluation and Reporting Protocol

A single results template that every contributor fills in, covering task accuracy, latency and RTF, peak memory, energy per inference, and the measurement conditions. Without this we will reproduce exactly the fragmentation that the white paper criticises — accuracy reported without runtime, or runtime reported without the conditions under which it was measured.

9. Task 6 — Reference Deployment

Finally, we take one model onto one device and run the whole flow end to end. The purpose is not the demo itself: it is to validate the methodology of Task 4 against measured silicon, and to expose the toolchain gaps that only appear once you try.

Proposed steps

  1. Select the target pair — one model and one device from the matrix, chosen so that the case is non-trivial but plausible (i.e. a cell that Task 4 predicts as “deployable with treatment” rather than “deployable as-is”).
  2. Reproduce the baseline — floating-point model, WER on the agreed test sets, reference latency and footprint on host.
  3. Export and inspect the graph — ONNX or equivalent; check operator coverage against the target and identify what will fall back or need replacement.
  4. Apply the fitting methods — quantization first, then pruning, distillation or architectural changes as required, following the recipes documented in Task 3.
  5. Compile and deploy — through the vendor toolchain, recording graph partitioning and any manual intervention required.
  6. Measure on device — WER, latency and RTF, peak memory, power, under the Task 5 template.
  7. Compare predicted vs measured — and feed the discrepancies back into the Task 4 screens. A methodology we have never checked against hardware is only a spreadsheet.

Output

A reproducible deployment note (model, versions, toolchain, flags, calibration data, measured results) and a short revision list for the feasibility methodology.

10. Deliverables and Ownership

Owners and dates to be assigned in the kick-off meeting.

TaskDeliverableOwnerTarget
Task 0Scope definition note (profiles, boundaries, metrics, test sets)TBDTBD
Task 1ASR model requirement database + family summariesTBDTBD
Task 2Hardware platform capability databaseTBDTBD
Task 3Survey of fitting and optimization methodsTBDTBD
Task 4Feasibility methodology note + deployment matrixTBDTBD
Task 5Common evaluation and reporting templateTBDTBD
Task 6Reference deployment note + methodology revisionTBDTBD

11. Risks and Open Issues

  • Vendor throughput and power figures are published under undocumented conditions and are generally not comparable between vendors.
  • Real-time factor is reported inconsistently in the literature; we may have to re-measure rather than cite.
  • Several relevant models have no publicly available weights, which limits what we can verify independently.
  • On-device energy measurement requires instrumentation and lab time — this needs an owner early, not at the end.
  • The two databases will diverge unless a single person owns the schema.
  • Scope creep towards a full survey. The matrix is the deliverable; the survey text supports it.

12. Points for Discussion

  1. Do we agree that the deployment matrix, rather than a survey chapter, is the primary deliverable?
  2. Which use-case profiles and hardware tiers do we commit to covering, and which do we explicitly exclude?
  3. Is the reference deployment (Task 6) in scope for this round? It is the expensive part, but it is our only source of ground truth.
  4. Who owns the database schema, and where do the databases live?
  5. Does the output feed back into the white paper as an expanded §3.3, or stand as a separate document?