Audio At The Edge

WORKING GROUP WHITE PAPER

DRAFT EDITION

Applications, architectures, evaluation, and open challenges

A task-centred survey of efficient, responsible, and deployable audio intelligence

August 2026

Executive Summary

The rapid growth of artificial intelligence has enabled a wide range of audio applications, and increasingly, these technologies are being deployed on edge devices such as smartphones, wearables, and embedded sensors, where low latency, privacy preservation, and energy efficiency are critical.

This paper provides an overview of the evolving landscape of Audio AI on the edge. We review major application areas, including speech enhancement, keyword spotting, voice activity detection, speaker recognition, acoustic event detection, and emerging domains such as affective computing and audio anomaly detection, highlighting representative models and design strategies that strike a balance between accuracy and computational constraints.

A particular focus is placed on evaluation: we examine traditional metrics and hardware-aware profiling. We analyze the advantages and limitations of these evaluation methods and discuss how they complement one another when assessing both system quality and real-world usability.

Finally, we identify open challenges, such as robust non-intrusive metrics, standardization of edge performance reporting, and scalable personalization techniques. By synthesizing current practices and identifying future research needs, this paper aims to provide researchers and practitioners with a concise guide to the opportunities and challenges of deploying Audio AI at the edge.

PURPOSE: This white paper maps the task landscape, deployment constraints, evaluation practices, and responsible-development priorities shaping Audio AI at the edge.

What this paper contributes

  • A cross-task synthesis of edge-ready Audio AI techniques.
  • A unified perspective on model quality, hardware efficiency, and real-world evaluation.
  • A review of emerging research directions for responsible and sustainable deployment.
  • A practical agenda for standardization, representation, privacy, and deployment readiness.

Intended audience

Researchers, developers, product teams, hardware practitioners, and policy stakeholders working across the edge-AI ecosystem.

Contributors

Working Group MembersAffiliation
Abigail OppongResearcher
Tomer BadugIndependent
Jonathan RussofemtoAI
Venkat RayuduSiLabs
Yuval RamotCeva
John MarconiEmbedUR
Mallik MoturiSyntiant
Gabriele StampinoNeuronova

Contents

Introduction
1Background
2Task-by-Task Survey
3Cross-Cutting Issues
4Edge Hardware and Deployment
5Open Challenges and Future Directions
6Conclusion
7Recommendations
8References

1  Introduction

The rapid growth of artificial intelligence has enabled a wide range of audio applications, and increasingly, these technologies are being deployed on edge devices such as smartphones, wearables, and embedded sensors, where low latency, privacy preservation, and energy efficiency are critical.

This survey provides a glimpse into the evolving landscape of Audio AI on the edge. We review major application areas, including speech enhancement, keyword spotting, voice activity detection, speaker recognition, acoustic event detection, and emerging domains such as affective computing and audio anomaly detection.

We highlight representative models and design strategies that strike a balance between accuracy and computational constraints. A particular focus is placed on evaluation, where both traditional signal- and task-based metrics are complemented by hardware-aware profiling that captures real-world efficiency. Finally, we identify open challenges, including the need for robust, non-intrusive metrics, standardization of edge performance reporting, and scalable personalization techniques.

By synthesizing current practices and identifying future research needs, this paper aims to provide researchers and practitioners with a concise guide to the opportunities and challenges of deploying Audio AI at the edge.

Audio processing has become a cornerstone of modern AI systems, enabling tasks such as voice assistants, acoustic monitoring, and human–machine interaction. Traditionally, these applications relied on cloud-based computation, leveraging large-scale models trained and executed on powerful servers. However, as computing increasingly shifts toward the edge, these systems must operate under severe resource constraints: limited memory, computational capacity, and energy budgets (Wang et al., 2025). Edge deployment brings new opportunities and challenges.

It enables real-time, low-latency processing while ensuring privacy, as sensitive audio never leaves the device. Yet it also demands efficient inference pipelines capable of meeting user expectations despite hardware limitations. Balancing these competing requirements has catalyzed a surge of research into lightweight, energy-aware, and privacy-preserving audio models (Singh, 2023; Surianarayanan et al., 2023).

1.1  Scope and Contribution

This survey focuses on how audio AI systems are being redesigned for edge deployment. We adopt a task-centric view that spans foundational areas, such as keyword spotting, speech enhancement, automatic speech recognition, and speaker biometrics, as well as rapidly evolving domains, including affective computing and audio anomaly detection. For each, we examine the datasets, representative architectures, deployment techniques, and evaluation strategies. In a nutshell, we

  • Synthesizes edge-ready Audio AI techniques across major tasks.
  • Provides a unified perspective on performance and efficiency evaluation.
  • Identifies emerging research directions for responsible and sustainable deployment.

The analysis also considers the following cross-cutting issues:

  • The rise of TinyML and on-device neural inference.
  • Trade-offs between performance and computational footprint.
  • The lack of standardized benchmarks for evaluating Audio AI in edge environments.
  • Ethical and privacy concerns are emerging from the handling of voice and biometric data.

By consolidating this information, we aim to support both academic researchers and industry practitioners in designing responsible, efficient, and scalable audio systems for edge devices.

2. Background

Edge AI refers to performing inference locally on constrained hardware such as microcontrollers (MCUs), mobile processors, or neural processing units (NPUs). Advances in TinyML, quantization, pruning, and distillation have made this possible, reducing model size while maintaining reasonable accuracy.

Modern deployment toolchains, such as TensorFlow Lite, ONNX Runtime, PyTorch Mobile, and CoreML, enable developers to port models directly to consumer devices. At the same time, frameworks like MLPerf Tiny, SUPERB, and HEAR have established benchmarks for evaluating model accuracy and efficiency.

Privacy-preserving techniques, including federated learning and secure enclaves, are gaining importance as edge devices handle sensitive audio data such as voiceprints and medical recordings.

2.1 Defining Edge AI

Edge AI refers to performing model inference directly on local devices, ranging from microcontrollers (MCUs) and mobile processors to dedicated neural processing units (NPUs), without relying on constant cloud connectivity (Wang & Jia, 2025). Unlike traditional cloud inference, edge AI must achieve acceptable accuracy under strict latency and power constraints, necessitating the co-optimization of algorithms and hardware.

This trend is supported by progress in model compression, quantization, network pruning, and knowledge distillation (Kerpicci et al., 2022). Collectively, these techniques enable large-scale models to be transformed into compact yet capable networks suitable for real-time audio tasks on resource-constrained platforms.

2.2 Toolchains and Deployment Frameworks

Modern edge-AI development benefits from mature toolchains that bridge training and deployment:

  • TensorFlow Lite and TensorFlow Lite Micro for mobile and MCU inference.
  • PyTorch Mobile and Core ML for Android and iOS ecosystems.
  • ONNX Runtime for cross-platform portability.

These frameworks automate hardware-aware optimizations such as 8-bit quantization and operator fusion, enabling developers to achieve substantial runtime and energy savings. For instance, experiments on low-power keyword spotting demonstrate that quantized CNNs can retain over 95% accuracy while reducing model size by 80% (Wang et al., 2022). 

2.3 Benchmarks and Evaluation Initiatives

Benchmarking initiatives play a crucial role in evaluating the performance of edge audio models. MLPerf Tiny provides standardized efficiency comparisons for microcontroller workloads, while SUPERB (Yang et al., 2021) and HEAR (Turian et al., 2022) assess general-purpose audio representations.

However, most current benchmarks emphasize accuracy alone and rarely capture hardware-level metrics, such as latency, memory footprint, and power consumption, creating a gap that this survey seeks to address.

2.4 Privacy-Preserving Techniques

Because edge devices often handle sensitive audio streams (e.g., personal voiceprints, medical recordings), privacy preservation is central to system design. Federated learning enables decentralized model updates without raw data exchange, while secure enclaves and on-device encryption minimize exposure (Wang et al., 2025). These methods complement traditional compression and model-efficiency strategies, advancing the goal of trustworthy and privacy-respectful Audio AI.

3  Task-by-Task Survey

3.1 Wake Word and Keyword Spotting (KWS)

Problem Definition

Keyword Spotting (KWS) systems detect predefined phrases such as “Hey Siri” or “OK Google,” enabling always-on voice interfaces. Because they are continuously listening, KWS models must operate with ultra-low latency and energy consumption while maintaining robustness against background noise and speech variability (Bushur et al., 2023).

Datasets and Benchmarks

The Google Speech Commands dataset (Warden, 2018) remains the standard benchmark for small-vocabulary tasks. Variants such as Speech Commands v2, openWakeWord, and the MLPerf Tiny KWS benchmark extend evaluation to edge scenarios, emphasizing memory footprint and on-device latency.

Google Speech Commands (GSC) v2:

Released in 2018, Version 2 of the Google Speech Commands dataset remains the primary benchmark for comparing KWS architectures. 

Content: 

Total Samples: 105,829 utterances. 

Classes: 35 specific words including commands (“Stop”, “Go”), digits (“One”, “Two”), and auxiliary words (“Bird”, “Bed”).

Noise: The _background_noise_ folder contains longer audio clips (e.g., running tap, white noise) intended to be mixed with the clean speech during training to improve robustness.   

Significance:
GSC established the “12-class” benchmark standard (10 commands + Silence + Unknown), allowing for the first direct apples-to-apples comparison of model architectures like CNNs, RNNs, and Transformers. Its crowdsourced nature introduces essential variability in recording quality and accent.   

Limitations: 

Fixed Duration: All clips are exactly one second long. This artificial segmentation simplifies training but fails to represent the “streaming” reality where a model must ignore the silence before and after a word.

Limited Vocabulary: With only 35 words, it cannot train open-vocabulary models or effective phoneme-based recognizers.

Silence Handling: The “Silence” class must be synthetically constructed by the user by slicing the _background_noise_ files, leading to inconsistent implementation across different research papers.   

Hey Snips:

Released by Snips (now Sonos) in 2019, this dataset was designed specifically to address the “False Alarm” problem in always-on systems.

Content:

Positive: ~11,000 utterances of the specific phrase “Hey Snips”.

Negative: ~86,500 “hard negative” utterances. These are not just random noise but phonetically similar phrases (e.g., “Hey steps”, “Heyips”) and general sentences that often trigger false positives.  

Significance:
This is the gold standard for testing False Alarm Rates (FAR). While GSC tests classification accuracy (A vs. B), Hey Snips tests the binary ability to reject non-target speech, which is the primary battery-saving function of a KWS chip. It also explicitly licenses data for “academic research only,” reflecting the privacy-sensitive nature of wake-word data. 

Limitations:

Single Keyword: It only supports “Hey Snips”. It cannot be used to train a generic “Alexa” or “Hey Siri” detector.

Licensing: The strict non-commercial license prevents its use in pre-training commercial models.   

Multilingual Spoken Words Corpus (MSWC):

Released by MLCommons, MSWC addresses the lack of diversity in KWS data by using automated alignment rather than manual recording.

Content:

Scale: >23 million clips, >340,000 unique keywords, 50 languages.

Source: Extracted from the Mozilla Common Voice corpus using the Montreal Forced Aligner (MFA). It includes low-resource languages like Welsh and Tatar which previously had zero KWS resources.   

Significance: MSWC democratizes KWS. It enables Few-Shot Transfer Learning research, where a model trained on English MSWC data can be fine-tuned with just a few examples to work in Kinyarwanda. It is the backbone of the “Open Division” in KWS benchmarking.   

Limitations:

Alignment Noise: Because keywords are sliced automatically from sentences, alignment errors occur. A clip labeled “Yes” might contain “Yes, I…” or just “es”.

Context Artifacts: The audio often contains co-articulation effects (the sound of the previous word bleeding into the start of the keyword), which makes the data acoustically different from isolated commands spoken to a smart speaker.   

LibriWords / LibriPhrase:

Recent research has moved toward “User-Defined KWS” (where the user types a word to enroll it). This requires datasets that test phonetic matching rather than class memorization.

Content:

LibriWords: Isolates individual words from audiobooks to create a massive 1000-class classification task.

LibriPhrase: Constructed to test open-vocabulary systems. It specifically mines “Hard Negatives” based on Levenshtein distance (e.g., Anchor: “Friend”, Hard Negative: “Trend”).

Significance:
Essential for evaluating Query-by-Example (QbE) and Query-by-Text systems where the model must detect a keyword it has never seen during training.

Limitations:

Read Speech: The source audio (audiobooks) lacks the urgency, prosody, and background noise of real voice commands.

Derivation Complexity: Requires users to download the 100GB+ LibriSpeech corpus and run processing scripts to generate the actual training clips.

MobvoiHotwords:

While many datasets rely on crowdsourced “near-field” audio (direct into a phone microphone), real-world smart speakers operate in “far-field” conditions. MobvoiHotwords addresses this gap.

Content:

Size: A large-scale corpus of approximately 144 hours.

Keywords: Contains two specific Chinese wake words: “Hi Xiaowen” and “Nihao Wenwen”.

Samples: Approximately 36,000 positive utterances per keyword.

Negatives: Over 220 hours of non-keyword speech to serve as robust negative training data.

Acoustic Conditions: Uniquely, this data was collected at distances of 1 meter, 3 meters, and 5 meters from the microphone array, capturing realistic room reverberation and attenuation.

Significance: It is one of the few open-source datasets that accurately represents the Smart Speaker use case (far-field acoustics). Most other datasets are “near-field” (phone/laptop mic). It is also a critical resource for Chinese language KWS research, which is often underrepresented in English-dominated benchmarks.

Limitations:

Language Specific: The wake words are Chinese, limiting its utility for English-only model benchmarking.

License: Typically released under Apache 2.0 or specific competition licenses, which may vary.

Hey Snapdragon:

Released by Qualcomm, this dataset is smaller but specifically curated to challenge KWS models with phonetically confusing words.

Content:

Size: 4,270 utterances from 50 speakers.

Keywords: Four distinct triggers: “Hey Android”, “Hey Snapdragon”, “Hi Galaxy”, “Hi Lumina”.

Format: 16kHz, 16-bit mono WAV.

Significance:
This dataset is widely used to evaluate False Alarm Rates (FAR) and “Hard Negative” rejection. The keywords are designed to be distinct, but the dataset is often used in “Query-by-Example” studies to see if a model can distinguish “Hey Snapdragon” from a similar sounding phrase. It serves as a rigorous “stress test” for models that perform well on easier datasets like GSC.   

Limitations:

Small Scale: With only ~4k clips, it is too small for training large deep learning models from scratch; it is primarily an evaluation (test) set.

License: Strictly for research purposes.

Picovoice Wake Word Benchmark:

Unlike the academic datasets above, this is a framework designed to benchmark commercial-grade wake word engines against one another.

Content:

Repository: An open-source GitHub framework containing code and data.

Audio: Crowdsourced recordings of 6 keywords: “Alexa”, “Computer”, “Jarvis”, “Smart Mirror”, “Snowboy”, “View Glass”.

Volume: ~300 recordings per keyword from >50 speakers.

Noise Mixing: It explicitly mixes this clean speech with the DEMAND noise dataset (kitchen, office, transit environments) at various Signal-to-Noise Ratios (SNR).

Significance:
It standardizes the metric of False Alarms per Hour (FA/hr). Instead of reporting abstract “accuracy,” this benchmark reports “Miss Rate at 1 False Alarm per 10 hours,” which is the critical metric for product managers. It allows for fair comparison between engines like Picovoice Porcupine, Snowboy (deprecated), and PocketSphinx.

Limitations:
Small Core Dataset: The custom keyword recordings are relatively few (~300), making it a test-only resource, not a training resource.

openWakeWord: Synthetic Data Generation

openWakeWord represents a paradigm shift. It is not just a dataset, but an open-source library and model collection that relies on synthetic data generation to solve the data scarcity problem.

Content:

Synthetic Generation: It uses high-quality Text-to-Speech (TTS) models (specifically Piper) to generate thousands of “positive” samples for any text phrase.

Augmentation: These synthetic positives are mixed with real-world background noise and “negative” speech features from datasets like ACAV100M and FMA (Free Music Archive) to create robust training sets.

Pre-trained Models: It includes models trained on ~100,000 synthetic positives and >30,000 hours of negative data.

Significance:
It solves the “Cold Start” problem. Developers can create a custom wake word (e.g., “Hey Robot”) without needing to record thousands of people first. It proves that synthetic data, when properly augmented with real noise, can train production-grade KWS models that run on commodity hardware (TFLite/ONNX).

Limitations:

Domain Shift: Models trained purely on TTS speech may struggle with unique human accents or dysarthric speech unless the TTS engine itself is diverse.

Technical Barrier: While it offers pre-trained models, generating a new custom word requires running Python scripts and understanding the generation pipeline, unlike simple “upload and click” web UIs.

Model Families

DS-CNN: Depthwise Separable Convolutions

While standard 2D convolutions are powerful, they are computationally expensive for edge devices. The breakthrough for KWS came with the adoption of Depthwise Separable Convolutions (DSCNN), popularized by architectures like MobileNet.

A DSCNN decomposes a standard convolution into two distinct layers:

Depthwise Convolution: Applies a single spatial filter to each input channel independently. This captures spatial/temporal patterns but does not combine information across channels.

Pointwise Convolution (1×1): Projects the output of the depthwise layer into a new channel space, combining information across channels

TC-ResNet: Temporal Convolution ResNets

Instead of treating the spectrogram as a 2D image, TC-ResNet treats it as a time-series of frequency vectors. It applies 1D convolutions along the time axis.

It consists of a stack of residual blocks, each containing two 1D convolution layers with Batch Normalization and ReLU.

This architecture demonstrated that 1D convolutions could effectively capture phonetic dynamics while significantly reducing the parameter count compared to 2D CNNs. Variants like TC-ResNet8 and TC-ResNet14 became standard baselines in the field

BC-ResNet: Broadcasted Residual Learning

Broadcasted Residual Block (BC-ResBlock), splits the signal processing into two parallel pathways that interact via broadcasting:

The 2D Pathway (Frequency-Aware): The input tensor (Time x Frequency x Channels) is processed by a lightweight 3×3 or 5×5 depthwise convolution. This step captures local spectral correlations (e.g., formant structures).

SubSpectralNorm (SSN): A critical component here is SSN. Unlike standard Batch Norm which normalizes the entire channel, SSN splits the frequency dimension into sub-bands and normalizes them independently. This accounts for the fact that different frequency bands in speech have different statistics and noise profiles.2

The 1D Pathway (Temporal aggregation): The output of the 2D path is averaged across the frequency dimension, collapsing it into a 1D temporal vector. This vector is then processed by a depthwise separable temporal convolution.

Broadcasting: The processed 1D vector is “broadcast” (expanded) back to the original 2D shape and added to the residual connection. This effectively modulates the 2D features with global temporal context

KWT Architecture: Vision Transformer for Audio

KWT adapts the Vision Transformer (ViT) architecture almost directly to the audio spectrogram.

Patch Embedding: The input spectrogram (e.g., 98 x 40) is sliced into non-overlapping patches (e.g., 1 x 40, representing 10ms of audio across all frequencies). These patches are linearly projected into a hidden dimension d.

Positional Encoding: Since self-attention is permutation-invariant, learnable positional embeddings are added to each patch token to encode the temporal order.

Transformer Encoder Blocks: The model consists of a stack of standard Transformer blocks:

Multi-Head Self-Attention (MSA): Allows every patch to attend to every other patch. This provides a global receptive field at every single layer, a distinct advantage over CNNs which only build global context deep in the network.

MLP: A feed-forward network processes each token embedding.

PostNorm: KWT utilizes a “PostNorm” configuration (LayerNorm applied after the residual connection), which was found empirically to stabilize training for KWS tasks.

Class Token: A learnable token is prepended to the sequence. The final embedding of this token is used for classification.

Performance and Distillation

KWT demonstrated that Transformers could achieve state-of-the-art results on KWS without massive pre-training.

Accuracy: KWT-3 achieves 98.54% on GSC V2, effectively tying with the best CNNs.

Parameter/Compute Cost: The trade-off is efficiency. KWT-3 requires 5.36M parameters and 526M MACs, which is significantly heavier than BC-ResNet-8 (321k params, 89M MACs). Even the smaller KWT-1 (607k params) is computationally intensive compared to CNNs.

Distillation: To mitigate this, KWT is often trained via Knowledge Distillation, using a larger MHAtt-RNN or ConvMixer as a teacher. This helps the Transformer learn the inductive biases of speech more rapidly from the teacher’s logits.

3.2 Speech Enhancement (Denoising, Dereverberation)

3.2.1 Problem Definition

Speech enhancement (SE) aims to improve the intelligibility and perceived quality of speech signals by suppressing background noise, reverberation, and other distortions introduced by everyday acoustic environments. This is a core building block for applications such as hearing aids, voice communication and conferencing, far-field voice assistants, and telemedicine, where users frequently operate in noisy, reverberant, or highly dynamic conditions (e.g., cars, open offices, homes) (Loizou, 2013; Wang & Chen, 2018).

Formally, the observed signal y(t) is modeled as a mixture of clean speech s(t) and interference components, including additive noise n(t) and/or reverberation modeled as convolution with a room impulse response h(t):

The SE system estimates either the clean signal s^(t) directly in the time domain or a mask/representation in the time–frequency domain that can be used to reconstruct an enhanced waveform.

For edge deployment, the problem is further constrained by:

Tight compute and memory budgets: models must fit within on-device RAM/Flash and run on low-power CPUs, DSPs, or NPUs, sometimes under a few hundred kB for hearing aids or microcontrollers.

Real-time, low-latency operation: enhancement must process streaming audio with strict end-to-end latency budgets (e.g., < 20–50 ms one-way for interactive communication) so as not to degrade user experience (F. ITU-T G.114).

Robustness to real-world variability: devices must generalize across diverse speakers, rooms, and noise conditions, often with far-field microphones and non-stationary noises, where mismatch between training and deployment conditions is common (Reddy et al., 2020).

Figure 1. Edge speech-enhancement pipeline and deployment constraints.

3.2.2 Datasets and Benchmarks

High-quality datasets are essential for training and evaluating speech enhancement (SE) systems. Most SE datasets follow a paired paradigm, providing clean speech and corresponding noisy or reverberant mixtures, although recent work also explores unpaired and synthetic data. Below we review the most commonly used corpora, their composition, and their role in benchmarking.

VoiceBank + DEMAND (Valentini-Botinhao et al., 2016)

The VoiceBank-DEMAND dataset combines clean recordings from the CSTR VCTK speech corpus with noise scenes from the DEMAND environmental dataset. Clean utterances (28 speakers for training, 2 for testing) are artificially mixed with 10 noises and multiple SNRs to create paired examples.

Use cases: Supervised denoising; academic baselines.

Metrics: PESQ, STOI, SDR.

Why important: Still the most widely used standard benchmark for traditional SE papers.

DNS Challenge Dataset (Reddy et al., 2020–2023)

Created for the Deep Noise Suppression (DNS) Challenge series, this dataset contains large-scale real and synthetic noise scenes (e.g., domestic, office, transportation), speech from Librispeech, VCTK, and public audiobooks, and impulse responses capturing realistic reverberation.

Use cases: Real-world noise suppression, dereverberation, streaming SE.

Metrics: DNSMOS (non-intrusive MOS predictor), subjective ITU-T P.835 tests.

Why important: The de-facto industry-oriented benchmark, with annual challenge leaderboards and real-time constraints.

CHiME (Barker et al., 2017)

The CHiME-3/4 datasets contain real far-field recordings in challenging public environments (bus, street, café) captured by multiple microphones. Speech is read from WSJ0 prompts.

Use cases: Joint SE + ASR; beamforming; multi-channel enhancement.

Strength: Provides realistic acoustic conditions and microphone-array data.

REVERB Challenge (Kinoshita et al., 2013)

A dataset targeting dereverberation, offering both simulated and real room impulse responses with 1–2 microphone channels. Based on WSJCAM speech.

Use cases: Dereverberation, multi-channel SE.

Strength: Well-established evaluation framework with reverberation-focused metrics.

Aurora-4 (Hirsch & Pearce, 2000)

A classical noisy ASR dataset with real and synthetic noise additions. Though not originally for SE, it is widely repurposed for denoising and separation research. Strength: High-quality mixtures for separation + denoising research.

DatasetPublisher / YearClean Speech SourcesNoise / Reverb SourcesReal or Synthetic# Hours
/ Size
ChannelsCommon Use Cases
VoiceBank-DEMANDValentini-Botinhao et al., 2016VCTK (28 train / 2 test speakers)DEMAND noise scenesSynthetic~11 hrs total1 chClassical SE benchmark, academic baselines
DNS ChallengeMicrosoft, 2020–2023Librispeech, VCTK, audiobooks150+ noise types; 500k+ RIRsBoth> 500 hrs synthetic + real1–2 chIndustry-grade denoising, streaming SE
CHiME-3/4Barker et al., 2017WSJ0 speechReal environments (bus, street, café)Real (multi-mic)~10 hrs6 chFar-field SE + ASR, beamforming
REVERB ChallengeNTT, 2013WSJCAM0Real/Simulated RIRsBoth~8 hrs1–2 chDereverberation, multi-channel SE
WHAM! / WHAMR!Wichern et al., 2019WSJ0-2mixEnvironmental noise & RIRsSynthetic~30 hrs1 chSE + speech separation
Aurora-4Hirsch & Pearce, 2000WSJ0Additive noise + channel distortionBoth~15 hrs1 chASR robustness; SE front-end testing

3.2.3 Representative Architectures

Modern speech enhancement (SE) systems can be categorized broadly into time–frequency (TF) domain models, time-domain models, and hybrid or multi-path architectures. Each family brings different trade-offs in accuracy, latency, and computational efficiency, critical considerations for edge deployment.

A. Time–Frequency Domain Architectures

TF-domain methods operate on the short-time Fourier transform (STFT), typically estimating magnitude masks or complex spectral components. They remain widely used due to their interpretability and efficiency.

1. DNN / CNN / RNN Masking Models

Early deep learning SE models used fully connected DNNs (Xu et al., 2014) or RNNs/LSTMs (Weninger et al., 2015) to estimate spectral masks. CNN-based models such as U-Net variants (Pascual et al., 2017, SEGAN) introduced encoder–decoder structures with skip connections.

Strengths: Low computational cost, well-suited for streaming, strong performance in stationary noise.

Weaknesses: Limited modeling of long-range temporal patterns.

Edge Notes: Compact CNNs and GRUs are often used in low-power DSP pipelines.

B. Time-Domain Architectures

Time-domain SE avoids STFT and directly models waveform samples, enabling end-to-end optimization. These approaches have driven large improvements in separation and SE benchmarks.

1. Conv-TasNet (Luo & Mesgarani, 2019)

A convolutional encoder–decoder architecture with learned bases and temporal convolutional networks (TCNs). Originally proposed for speech separation, it was rapidly adapted to denoising.

Strengths: Excellent SI-SDR improvements, highly effective on WHAM!/DNS tasks.

Limitations: Larger memory footprint compared to TF-domain models; depthwise and dilated convolutions can be expensive on some edge hardware.

Edge Notes: Quantized and pruned Conv-TasNet variants exist, but require careful optimization.

2. Wave-U-Net / SEGAN Family

Architecture families extending U-Net principles to raw waveforms (e.g., SEGAN; Pascual et al., 2017).

Strengths: End-to-end learning of enhancement artifacts; GAN-based approaches improve perceptual quality.

Weaknesses: GAN training instability; often high compute load.

C. Dual-Path and Transformer-Based Architectures

1. Dual-Path RNN / Transformer (DPRNN, DPTNet)

Models such as DPRNN (Luo et al., 2020) and DPTNet (Chen et al., 2020) segment long sequences into chunks for intra-chunk and inter-chunk processing.

Strengths: Strong long-range modeling with manageable complexity; SOTA results in separation and enhancement.

Weaknesses: Often large parameter counts; memory-intensive due to dual-path processing.

Edge Notes: Chunking helps reduce RAM usage, making dual-path models more edge-plausible than full-transformers.

2. Conformer and Transformer SE

Borrowed from ASR advances (Gulati et al., 2020), Conformer blocks integrate convolution + self-attention. Recent works apply lightweight transformers for SE (e.g., Kim et al., 2023).

Strengths: Strong global and local modeling; excellent robustness.

Weaknesses: Multi-head attention incurs latency on mobile CPUs.

Edge Notes: Mobile-friendly transformer variants (e.g., linearized attention, conformer-lite) are emerging.

D. Classical + Hybrid Approaches

1. RNNoise (Jean-Marc Valin, 2018)

Strengths: Extremely low complexity; <5 ms frame latency; practical for MCUs and WebRTC.

Weaknesses: Lower enhancement quality vs modern deep models.

Edge Notes: Gold-standard lightweight baseline for embedded SE.

2. Beamforming + Neural Post-filtering

In multi-channel devices (smart speakers, conference bars), neural SE often complements MVDR/WPE beamformers.

Strengths: Robust far-field performance; strong industrial adoption.

Weaknesses: Requires multi-mic hardware.

Edge Deployment

Quantized SE models (INT8) run on embedded DSPs or NPUs with a 100 ms latency for 1-second windows. Adaptive frame-length scheduling can further lower power draw.

Evaluation Metrics

PESQ, STOI, SI-SDR, SDR, and DNSMOS (non-intrusive MOS prediction). Edge-specific profiling reports runtime latency, RTF (real-time factor), and energy consumption.frequently used to test SE systems as a front-end for ASR.

Use cases:

WHAM! / WHAMR! (Wichern et al., 2019)

Derived from WSJ0-2mix speech separation datasets, WHAM! adds environmental noise while WHAMR! further adds reverberation.

Use cases: SE-SEPARATION hybrid; time-domain models like TasNet.

3.3 Automatic Speech Recognition (ASR)

Problem Definition

Automatic Speech Recognition (ASR) aims to convert spoken language into a sequence of linguistic units (characters, word pieces, or words) so that machines can understand and respond to human speech. ASR is a core component in voice assistants, dictation systems, live captioning, accessibility applications, and telemedicine, where users speak naturally in diverse and unpredictable environments (Jurafsky & Martin, 2023; Li et al., 2020).

Basically, an automatic Speech Recognition (ASR) system aims to learn a mapping from an input acoustic sequence to an output symbol sequence:

where:

    is the input acoustic feature sequence of length ,

 is the output label sequence of length ,

   is the vocabulary (e.g., characters, subwords, or words).

Figure 2. Representative edge Audio AI system architecture.

Architecture of a traditional ASR (Papastratis, 2021; Ahlawat, Aggarwal, & Gupta, 2025)

Datasets and Benchmarks

DataPublisher/YearClean Speech SourcesNoise/SourcesReal/Synthetic#Hours/SizeLangugeChannelsCommon Use CasesAcessibility
LibriSpeech
https://www.openslr.org/12
Panayotov et al., 2015English)Librivox audiobooksMostly clean; mild channel/noise variationReal~1000hEnglish1 chCore ASR benchmarkFully open (CC BY 4.0)
Libri-light
https://github.com/facebookresearch/libri-light
Kahn et al., 2020 (Meta AI)LibriVox audiobooksMostly cleanReal60000hEnglish1 chSelf-supervised ASR pretrainingFully Open
Mozilla Common VoiceMozilla, 2017–presentCrowdsourcedReal-world background noiseReal>30,000 h, (140+ languages)Multilingual1 chMultilingual & Low-resource ASRFully Open
AISHELL-1Bu et al., 2017Read speechClean studio conditionsReal~178 hMandarin Chinese1 chMandarin ASROpen
AISHELL-2AISHELL Foundation, 2018Read & spontaneous speechRealistic acoustic environmentsReal~1,000 hMandarin ChineseMulti-chIndustrial & robust ASRRestricted (license required)
TED-LIUM 3Hernandez et al., 2018TED/TEDx talksLecture hall acousticsReal~452hEnglish1 chLong-form & lecture ASROpen
SPGISpeech / v1-v2Gupta et al., 2021 / 2025Financial & business speechReal-world noiseReal~5,000 h / ~50,000 hEnglish1 chDomain-specific ASRRestricted
SwitchboardGodfrey et al., 1992Telephone conversationsTelephone channel noiseReal300hEnglish2 chConversational ASRRestricted (LDC)
2000 HUB5LDC, 2000Telephone speechTelephone noiseReal~40 hEnglish2 chASR evaluation benchmarkRestricted (LDC)
CHiME-5Barker et al., 2018Dinner party conversationsReal domestic noiseReal~50 hEnglishMulti-chFar-field & noisy ASROpen
GigaSpeechChen et al., 2021Audiobooks, podcastsVariable background noiseReal~10,000 hEnglish1 chLarge-scale ASR trainingOpen
CSTR VCTKYamagishi et al., 2019Read speechMultiple microphone conditionsReal~44 hEnglish (varied accents)Multi-chSpeaker & accent modelingOpen
Libri-transKocabiyikoglu et al., 2018LibriSpeechSynthetic translation noiseSynthetic~236 hEn–Fr1 chSpeech translationOpen
Fisher-Cieri et al., 2004Telephone conversationsTelephone channel noiseReal~2,000 hEnglish2 chConversational ASR, LM trainingRestricted (LDC)

Model Architectures

Modern ASR systems are predominantly based on end-to-end neural architecturesConnectionist Temporal Classification (CTC), Recurrent Neural Network Transducers (RNN-T), and attention-based encoder–decoder models.

Connectionist Temporal Classification (CTC)

CTC enables sequence-to-sequence learning without requiring frame-level alignment between audio frames and output labels. It introduces a blank symbol and marginalises over all valid alignments during training. CTC-based models are computationally efficient and well-suited for streaming and low-latency ASR, making them attractive for on-device and edge deployments. However, CTC models often rely on an external language model to achieve competitive performance.

Recurrent Neural Network Transducer (RNN-T)

RNN-T extends CTC by incorporating a prediction network that conditions on previous output tokens, allowing it to jointly model acoustics and linguistic context. This makes RNN-T particularly effective for streaming ASR applications, such as voice assistants and real-time transcription. While RNN-T models generally outperform pure CTC systems, they are more complex to train and require careful optimization to achieve optimal performance.

Attention-Based Encoder–Decoder Models

Attention-based encoder–decoder architectures directly learn alignments between acoustic inputs and output tokens using a soft attention mechanism. These models typically achieve strong performance on offline ASR tasks due to their ability to capture long-range dependencies. However, their reliance on full input sequences limits their suitability for real-time and low-latency scenarios.

Conformer and Hybrid Architectures

The Conformer architecture combines convolutional neural networks with self-attention mechanisms, allowing for the effective modeling of both local and global temporal dependencies. Conformer-based models have become a dominant choice in recent ASR systems due to their strong performance across a range of datasets.

Hybrid architectures, such as Conformer-Transducer and Transformer-Transducer models, integrate the strengths of Conformers with the streaming capabilities of RNN-T. These hybrids deliver near state-of-the-art accuracy while maintaining low-latency inference, making them suitable for both cloud-based and on-device ASR systems.

Recent work has further explored model compression and quantization to improve deployment efficiency. In particular, INT8-quantized Conformer models have demonstrated a 3× inference speedup on Neural Processing Units (NPUs) with minimal degradation in word error rate (WER), highlighting their potential for resource-constrained environments (Guimarães et al., 2023).

ArchitectureStrengthsLimitationsTypical Use Cases
CTCSimple, efficient, low latencyWeaker language modelingOn-device, edge ASR
RNN-TStrong streaming performanceTraining complexityVoice assistants
Attention Enc-DecHigh accuracy, global contextNot real-timeOffline transcription
ConformerLocal + global modelingHigher compute costGeneral-purpose ASR
Conformer-RNN-TSOTA accuracy + streamingModel complexityProduction ASR

A table summarizing the various architectures of ASR

Speech-to-Text Translation (STT / S2TT)

Speech-to-Text Translation (STT) frequently categorized as End-to-End Speech Translation (E2E ST) surpasses the scope of traditional ASR by directly converting a source acoustic sequence x into a target text sequence z. Unlike modular systems, E2E ST models bypass the requirement for a source-language transcript, establishing a direct mapping between audio and translated text:

Traditionally, this task relied on cascaded pipelines where ASR output served as input for Machine Translation (MT). While leveraging established components, these architectures are prone to error propagation where acoustic misrecognitions derail the translation step and incur significant computational latency during real-time inference.

3.3.2.1 End-to-End STT Architectures

Contemporary research favors End-to-End (E2E) attention-based encoder–decoder frameworks and large-scale foundation-style multi-task Transformers (e.g., Whisper, SeamlessM4T). These models address several unique challenges:

Encoder Complexity: The STT encoder must distill raw acoustic frames into high-level semantic representations that bridge disparate grammatical and linguistic boundaries.

Structural Reordering: Unlike monotonic ASR, the attention mechanism in STT must handle non-linear alignment to account for differing word orders across languages (e.g., SVO vs. SOV).

3.3.2.2 Training Modalities & Loss Functions

Due to the scarcity of direct speech-to-translation data, E2E STT systems rely heavily on multi-task learning. Models are typically joint-trained with auxiliary objectives utilizing both speech and text corpora:

Source ASR Loss: Constrains the encoder to capture phonetic properties by predicting source transcripts.

Target MT Loss: Leverages the decoder as an MT agent, processing semantic embeddings to generate target-language text.

Self-Supervised ASR Models

Self-supervised learning (SSL) has become a dominant paradigm for ASR, enabling models to learn speech representations from unlabeled audio. wav2vec 2.0, HuBERT, and WavLM are among the most widely studied SSL models, and are consistently reviewed and compared in surveys of unsupervised and semi-supervised speech recognition (Baevski et al., 2020; Yang et al., 2021; Lakhotia et al., 2021). Large multilingual SSL models, such as XLS-R, extend wav2vec-style pretraining to over 100 languages, demonstrating strong cross-lingual transfer, particularly for low-resource languages (Conneau et al., 2021; Shi et al., 2023).

End-to-End and Foundation ASR Models

End-to-end ASR models trained on large-scale and weakly supervised data have gained prominence due to their robustness across domains. Whisper, an attention-based encoder–decoder model, is frequently discussed in surveys of end-to-end ASR systems as an example of a foundation-style ASR model trained on diverse multilingual data (Radford et al., 2023; Zhang et al., 2023).

Toolkits such as NVIDIA NeMo, ESPnet, and SpeechBrain provide pretrained implementations of Conformer-CTC and Conformer-RNN-T models, and are commonly referenced in surveys as practical instantiations of modern ASR architectures used in both research and production systems (Zhang et al., 2023).

Multilingual and Low-Resource ASR Models

Surveys of multilingual ASR emphasize that shared representations across languages can improve performance in low-resource settings, but may introduce imbalance when training data is dominated by high-resource languages (Besacier et al., 2014; Shi et al., 2023). Models fine-tuned on datasets such as Common Voice, FLEURS, and XLS-R variants are frequently evaluated in this context.

Deployment-Oriented and Efficient ASR Models

Recent surveys also highlight a growing emphasis on efficiency-aware ASR models, including streaming-capable, quantized, and compressed systems for real-time deployment. Conformer-RNN-T models with INT8 quantization have been demonstrated to achieve significant inference speedups on specialized hardware, with minimal degradation in word error rate, making them suitable for edge and low-power environments (Guimarães et al., 2023).

ModelArchitectureSupervision LevelLanguage CoverageTypical Use Cases
Whisper Radford et al. (2023)Encoder–Decoder TransformerWeakly supervisedMultilingualRobust transcription
wav2vec 2.0 (Baevski et al., 2020)Transformer encoder + CTCSelf-supervisedMostly EnglishLow-resource fine-tuning
HuBERT Hsu et al. (2021)Transformer encoderSelf-supervisedMostly EnglishRepresentation learning
WavLM Chen et al. (2022)Transformer encoderSelf-supervisedMultilingualRobust ASR
XLS-R
Babu et al. (2021)
Transformer encoderSelf-supervised100+ languagesMultilingual ASR
NeMo Conformer-CTC Kuchaiev et al. (2019)Conformer + CTCSupervised / SSLEnglish + othersOffline ASR
NeMo Conformer-RNN-T Gulati et al. (2020)Conformer-RNN-TSupervised / SSLEnglish + othersProduction ASR
ESPnet ASR Watanabe et al. (2018)Transformer / ConformerSupervised / SSLMultilingualResearch benchmarking
SpeechBrain ASR  Ravanelli et al. (2021)CTC / Encoder-DecoderSupervisedMultilingualResearch & education
Kaldi ASR Povey et al. (2011)HMM-DNN / HybridSupervisedMultilingualClassical ASR
Fairseq S2TTransformer / ConformerSupervised / SSLMultilingualASR & Speech Translation
Coqui STT
Hannun et al. (2014)
DeepSpeech-styleSupervisedLimitedOpen-source ASR
Vosk Pyliaev et al. (2020)Kaldi-basedSupervisedMultilingualOffline ASR
NVIDIA Riva ASR  (2021)Conformer-RNN-TSupervised / SSLMultilingualEnterprise ASR
Google Cloud ASR 2020Conformer-RNN-TSupervised / SSLMultilingualCloud ASR

Edge Deployment Strategies and Challenges

Edge deployment of Automatic Speech Recognition (ASR) systems has become increasingly important for enabling low-latency, privacy-preserving, and offline-capable speech interfaces. Unlike cloud-centric architectures, edge ASR performs inference directly on resource-constrained devices such as smartphones, embedded systems, and IoT hardware. This shift introduces a set of design strategies aimed at balancing accuracy, efficiency, and robustness under strict computational constraints.

Model Compression and Optimization

A foundational strategy for edge ASR deployment is model compression. Techniques such as pruning, quantization, and knowledge distillation are widely adopted to reduce model size and computational cost. Pruning eliminates redundant parameters, while quantization reduces numerical precision (e.g., FP32 to INT8), significantly improving inference speed and memory efficiency. Knowledge distillation enables compact student models to learn from larger teacher models, preserving performance while reducing complexity. These techniques are often combined to achieve optimal trade-offs between latency, memory usage, and transcription accuracy, particularly for streaming or real-time applications.

Efficient and Streaming-Capable Architectures

Another key strategy involves designing lightweight and streaming-friendly neural architectures. Models such as streaming RNN-T, Conformer variants, and compact transformer-based encoders are optimized for incremental decoding, allowing real-time transcription with minimal delay. Recent developments in large-scale pre-trained models such as OpenAI Whisper have further influenced architectural thinking. While Whisper demonstrates strong robustness across multilingual and noisy conditions due to large-scale weak supervision, its full-size variants are computationally expensive for edge deployment. This has led to growing interest in distilled or quantized Whisper variants, which aim to transfer its robustness into lightweight models suitable for on-device inference.

Hardware-Aware and Framework-Level Optimization

Edge ASR systems increasingly rely on hardware-aware optimization frameworks such as TensorFlow Lite, ONNX Runtime, and PyTorch Mobile. These frameworks enable model conversion and optimization tailored to specific hardware accelerators (e.g., GPUs, DSPs, NPUs).

Hardware–model co-design is also becoming more prominent, where neural architectures are explicitly designed to align with device-level constraints such as memory bandwidth, power consumption, and thermal limits. This ensures more predictable and efficient deployment across heterogeneous devices.

Federated and On-Device Learning

To improve personalization while preserving privacy, federated learning and on-device adaptation are increasingly used in edge ASR systems. Instead of transmitting raw audio data to centralized servers, models are fine-tuned locally using user-specific data. Only model updates are shared, reducing privacy risks while enabling adaptation to accents, environments, and domain-specific vocabulary.

Hybrid Edge–Cloud Inference

Given the limitations of edge devices, many real-world systems adopt hybrid inference strategies, where lightweight models operate on-device for real-time transcription, while more complex models run in the cloud for refinement or rescoring. This layered approach allows systems to balance latency, cost, and accuracy dynamically depending on network availability and task complexity.

3.4 Speech Biometrics (Speaker Verification & Diarization)

Problem Definition

Speaker biometrics covers speaker verification, speaker identification/recognition, and speaker diarization. Speaker verification authenticates a claimed identity from speech by comparing an enrolled voice template with a test utterance, while speaker identification selects the most likely speaker from a known set. A typical verifier extracts embeddings for enrollment and test utterances then accepts the claim when a cosine, PLDA (Probabilistic Linear Discriminant Analysis), or neural score exceeds a specified threshold. (Snyder et al., 2018; Desplanques et al., 2020).

Speaker diarization segments multi-speaker audio into time-stamped, speaker-homogenous regions, often summarized as the task of determining “who spoke when”. It usually assumes no prior enrollment and no fixed speaker inventory, although production systems may combine diarization with speaker recognition. Diarization can be an essential pre-processing step for speaker-aware ASR, meeting transcription, speaker indexing, and accessibility interfaces.

For edge deployment, the challenge is not only performance but also trustworthiness. Voice embeddings are biometric templates; if compromised they can expose identity. Edge systems therefore need local inference, template protection, threshold calibration, and hardware-aware reporting of latency, memory and energy use.

Datasets and Benchmarks

Speaker verification and identification are commonly benchmarked on Voxceleb1/2, NIST Speaker Recognition Evaluations and CN-Celeb, while diarization uses meeting, conversational and in-the-wild corpora such as AMI, DIHARD and VoxConverse (Nagrani et al., 2017; Chung et al., 2018; Fan et al., 2020; Carletta et al., 2005; Ryant et al., 2021; Chung et al., 2020). On-device studies often create enrollment/test splits from these corpora, or use small local enrollment subsets, to simulate personalization. Results should be separated by utterance duration, microphone distance, noise condition, speaker count and enrollment size.

Dataset /
benchmark
Primary taskScale / coverageRelevance to speaker biometrics
Voxceleb1/2Verification /
identification
Voxceleb1 contains more than 100k utterances from over 1000 speakers.
Voxceleb2 scales to more than 1M utterances from over 6000 speakers.
Large in-the-wild pretraining and standard verification evaluation under noisy, unconstrained conditions.
NIST SREVerification / person recognitionRecurring evaluation series coordinated by NIST since 1996, with task conditions varying by cycle.Strong protocols for calibration, operating-point analysis and telephone/microphone robustness
CN-Celeb1/2Verification /
identification
CN-Celeb1: more than 130k utterances from 1000 speakers.
CN-Celeb2: more than 520k utterances from 2000 speakers
11 real-world genres
Non-English, multi-genre stress test for domain and language mismatch
AMIDiarization / meeting speech100 hours of meeting recordings with close-talking and far-field channelsUseful proxy for meeting assistants, speaker-aware ASR and conferencing devices
DIHARD IIIDiarizationDiverse development/evaluation audio across 11 domainsStress test for hard diarization, speaker-count estimation and domain shift
VoxConverseDiarizationMore than 50 hours of multispeaker YouTube audioIn-the-wild diarization with overlapping speech, background variability and realistic turn-taking.

Model Families

x-vector/TDNN systems: x-vector networks map variable-length utterances to fixed.dimensional speaker embeddings through frame-level TDNN layers and statistics pooling. Embeddings are commonly scored using cosine similarity or PLDA and robustness is improved through noise and reverberation augmentation (Snyder et al., 2018).

ECAPA-TDNN and lightweight variants: ECAPA-TDNN strengthens the x-vector family with Res2Net-style temporal blocks, squeeze-and-excitation channel attention, multi-layer feature aggregation, attentive statistics pooling (Desplanques et al., 2020). Lightweight variants such as MobileECAPA reduce model footprint through depthwise convolutions and network pruning enabling efficient speaker verification on resource-constrained devices.

Self-supervised speech encoders: wav2vec 2.0, HuBERT, WavLM and related encoders learn general representations from unlabeled audio and can improve verification or diarization when finetuned. Full SSL models are usually too large for low-power devices, so edge deployments typically freeze selected layers, distill a smaller student or use SSL only during training (Baevski et al., 2020; Hsu et al., 2021; Chen et al., 2022).

Diarization pipelines: A practical pipeline is Voice Activity Detection (VAD) -> speaker-change detection or fixed segmentation -> speaker embedding extraction -> clustering -> optional resegmentation and overlap handling. End-to-end neural diarization and pyannote-style neural pipelines reduce hand-engineered stages and better address overlapped speech, but they require careful chunking and state management for streaming edge operation (Bredin et al., 2020).

Edge Deployment Considerations

Local enrollment and thresholding: Store compact embeddings or class prototypes, not raw enrollment audio. Calibrate thresholds per device class and application, because false accept/false reject trade-offs vary with microphone, environment, utterance duration, and speaker population.

Template protection: Voiceprints should be encrypted at rest, bound to a device or account, and processed inside a secure enclave or trusted execution environment where available. Cancellable or revocable embeddings are preferable to raw embeddings because they reduce the impact of template leakage.

Presentation-attack detection: Authentication systems should reject replayed, synthesized, or voice-converted speech using anti-spoofing models, challenge-response phrases, or multimodal confirmation for high-risk actions (Yamagishi et al., 2021).

Compute path: Always-on systems should be gated by low-power VAD or KWS before running the speaker encoder. For online diarization, clustering must maintain a small active-speaker cache and periodically merge clusters without exceeding RAM budgets.

Model compression: INT8 quantization, structured pruning, channel-width scaling, and distillation usually preserve most verification accuracy, but papers should report the actual model size, MACs/FLOPs, peak RAM, and energy. ECAPA-TDNNLite reports 11.6M FLOPs and 3.07% EER on VoxCeleb1-O, with 2.31% EER in an asymmetric enroll-verify setup.

Bias and robustness: Speaker biometrics are sensitive to demographic imbalance, accent and language mismatch, health-related voice changes, room acoustics, and background noise. Evaluation should include subgroup and domain-shift analysis before deployment.

Metrics

For speaker verification, report Equal Error Rate (EER), minimum Detection Cost Function (minDCF), false-acceptance rate (FAR), false-rejection rate (FRR), DET/ROC curves, and operating-point metrics such as true-acceptance rate at a specified FAR. For speaker identification, report top-1/top-k accuracy and open-set rejection when unknown speakers may appear.

For diarization, report Diarization Error Rate (DER), Jaccard Error Rate (JER), speaker-confusion time, missed-speech time, false-alarm time, speaker-count error, and overlap-aware DER when overlapped speech is important. 

For edge systems, pair task metrics with latency per decision or per frame, real-time factor (RTF), peak RAM, Flash/model size, energy per inference, sustained power, and CPU/DSP/NPU utilization. For security-sensitive products, also report spoof false-acceptance and recovery behavior after failed authentication attempts.

3.5 Spoken Language Understanding (Short-Utterance Classification)

Problem Definition

Spoken Language Understanding (SLU) refers to the class of models that maps acoustic input directly to a semantic intent label or action class. The acoustic input typically takes the form of raw waveforms or time-frequency spectral features, though explicitly learned front end features may be used in some cases. Unlike ASR, SLU does not intend to reconstruct text from acoustic features. Instead it performs semantic classification over an acoustic manifold, allowing semantically equivalent utterances such as [“turn on the lights”, “switch the lights on”, “lights please”] to converge to the same user intent. While SLU ultimately attempts to output a single action sequence, the problem may be posed to parse more than 1 semantic class (i.e. slot) within an utterance to achieve the final sequence such that “turn on the kitchen lights” correctly maps [location:kitchen, object:light, object state:on]. This formulation makes SLU fundamentally different from Keyword Spotting (KWS), where a decision function is optimized to detect the presence of a predefined lexical tokens or short phases with limited variation. KWS systems are closed-set detectors that recognize specific strings. SLU systems by contrast operate at a semantic level and generalize over paraphrasing, prosodic variation, accent, and user-specific phrasing, without each variation being hard-coded into the set of solutions.

SLU is also distinct from many Intent Detection and Slot Filling frameworks in the Natural Language Understanding (NLU) domain. NLU pipelines assume text input and model user intent as a classification task and slot filling as a structured prediction problem where semantic labels are assigned at the token level. Many of these models require robust ASR models with relatively large general vocabularies and assume a broad range of slot varieties  (names, dates, times, etc) with IDs recognized from a vectorized database or dictionary. SLU avoids transcriptions entirely and maps speech features to a relatively domain-specific set of intents without explicitly modeling linguistic or slot entity structures. While narrower in scope, SLU is more tightly coupled to phonetic variation and generally requires smaller models than ASR or NLU systems individually, making it an excellent choice for low power embedded deployments and applications with a limited scope but with high input variance. While this is generally true, more recent approaches to both problems highlight a trend towards defining the tasks as a continuum, rather than clearly distinct systems. SLU systems may explicitly identify aligned slot locations, classify defined intents, and fill slots from a defined dictionary for each slot type, making them similar to NLU systems. Conversely, modern end-to-end NLU systems may not necessarily require text output from an ASR model as input, making them similar to SLU front ends encode audio features into a unique embedding space.

Architectures

SLU systems consist of 2 distinct components: the acoustic encoder and the supervised learning strategy to learn a mapping between output embeddings from the encoder and the downstream intent classes. Architectures for SLU range in size, from O(10kB) to O(10MB)., with more complex end-to-end systems requiring more. SLU architectures are amenable to quantization, allowing for INT16 activations and INT8 weights with little to no loss in accuracy compared to floating point.

Small Architectures

Smaller architectures aimed at more constrained deployments on microcontrollers and small embedded NPUs often consist of CNN-RNN hybrid architectures like a CRNN or RCNN. In the CRNN, audio input is transformed into spectral features via STFT, a log-Mel transform is applied, and a 1-D or 2-D CNN captures high-level time frequency patterns. Outputs of the CNN are fed into an RNN, LSTM, or GRU encoder for temporal feature aggregation to model temporal behavior before being sent to the intent classification layers (usually fully connected layers). The CRNN remains competitive in embedded deployments due to its low parameter count, computational complexity, and latency. Audio input is normally sampled at 16kHz (as opposed to 8kHz in some KWS or WWD systems) with 16-bit integers, and window sizes range from 8ms to 50ms, with 16ms or 32ms being common choices.

Larger Architectures

More recent architectures replace the recurrent layers with transformer or hybrid convolutional-transformer components such as Conformer. These architectures require more MAC operations and parameters, though performance is generally improved compared to RCNN or CRNN architectures. Modifications to the convolutional components such as replacing 2D convolutions with depthwise separable convolutions can make Conformer more amenable for embedded deployment. Low-rank attention and adaptation (e.g. LorA) and restricted context window schemes (e.g. SWIN) can improve robustness to acoustic environments while keeping total footprint low enough for embedded accelerators and mid-sized application processors.

SLU architectures span the gamut from piecewise models constructed of several task-specific modules, to end-to-end audio-to-action models. Single model SLU architectures utilize separate modules for intent detection and slot filling where each module is dedicated to one task. These modules can either have separate encoders or share a phoneme or word-level encoder that produces shared embeddings. The intent detector module aims to take in an embedding and produce a.) space of intents or actions from which the most probable is chosen and b.) identification and classification of slots to be filled. The slot filling module takes in embeddings and selects slot values that are present in the utterance. Each model in this case has its own decoder and the outputs of both are used by a decision layer to determine the final action to be performed.

Pipeline vs. End-to-End

In more recent architectures, shared embedding spaces are used not only for the acoustic inputs, but for the intermediate representations of both the intent detector and the slot filler. In implicit joint models, a shared embedding space feeds to separate decoders. In explicit joint models, the embedding space may contain an interaction model that learns the correlations between intent classifications and slot values. The interaction model may be unidirectional, where the slot filler takes in information from the intent detector but not vice versa, or bidirectional, where information about either model output may be exchanged between the models. In many cases, slots and intents are highly correlated, leading to improved performance and a potential reduction in size of the joint embedding space when pruning is implemented. These modules may be trained independently, with only the interaction model being learned jointly during fine tuning, or, the models may be jointly trained during the entirety or even during certain parts of training via freezing and unfreezing layer weights. In end-to-end models, a single encoder and decoder is jointly learned to produce intent and slot outputs, without explicit blocks for each task. While end-to-end models have been shown to yield state of the art performance on more complex or wide breadth SLU tasks, they have the largest parameter count and often rely upon large attention layers to model complex interactions between tokens. This makes them larger and harder to train than their piecewise constructed counterparts. Thus, end-to-end models are rarely used for small embedded deployments or task-specific applications due to their large memory requirements, high computational complexity, in addition to taking longer to train and implement.

Training Approaches

Training procedures for SLU span across the supervised-unsupervised continuum. End-to-end models can learn the speech-to-intent mappings directly from the labeled audio with the help of additional loss terms like the connectionist temporal classification (CTC) loss commonly used in ASR models. Joint SLU models such as those used in  also utilize CTC loss, though the entire model is not trained end-to-end. Instead, an ASR model such as Wav2Vec can be pre-trained on word-level, sub-word-level, phoneme-level, or grapheme-level targets (or a combination thereof. If whole word level targets are not used, additional modules may be trained to take in intermediate representations from the lower level targets to construct word level embeddings and predicted words from the previous targets. Once the ASR model is pretrained, the large weight matrix used to predict the target words is discarded and the hidden states from the word-level ASR model are used to train a supervised SLU model that maps inputs to the predicted intents (e.g. slot classes and slot values). This methodology allows the use of pre-trained ASR models as a boiler plate architecture from which to train a relatively smaller, less complex, SLU model, significantly speeding up training and development time.

Large, jointly-trained intent detection and slot filling algorithms generally rely upon text input for both training and inference, not merely intermediate representations generated from a small subsection of layers from an ASR network. While it may be possible that inter-word semantic information can be encoded directly from audio features produced by small networks, we have yet to see an approach that proves such semantic information can be reliably extracted and used to perform larger, more broad NLU tasks, such as those seen in SLURP, with models that are lower in complexity and parameter count, such as those found in the Fluent Speech Commands paper.

Datasets and Benchmarks

SLU datasets range from smaller sets intended for highly-specialized, narrowly-scoped applications to relatively broad-scoped more general applications. We have included a collection of datasets along with information about their sizes and attributes in the table below..

SLU Dataset Size and Attributes

DatasetNumber of ExamplesReal SpeakersDuration (hrs)Licence
Fluent Speech Commands30,0439719non-commercial
Snips5,886695.5non-commercial
SLURP72,27717758Creative Commons
Timers and Such2,151952.5Creative Commons

SLU datasets are often structured to include tags for explicit actions (called “intents”), slot types (sometimes called “slot classes” or “slot labels”), and slot values (sometimes called “entities”). A unique phrasing may be constructed from 1 or more entities and actions along with any number of non-identifier filler words. In table XX, we include some information for common SLU datasets to understand their complexity, depth, and breadth.

SLU Dataset Structure

DatasetSlot TypesIntentsUnique PhrasingsTotal Vocabulary Size
Fluent Speech Commands331248O(1000)
Snips (smart light)361765400
Snips (smart speaker)8 (English)
9 (French)
93003 (English)
2555 (French)
65,000 (English)
70,000 (French)
SLURP554611,367O(100,000)
Timers and Such442,151~20,000

One desirable feature of SLU models compared to their KWS counterparts is semantic generalizability. As it is impractical to include every conceivable formulation for a given intent, SLU models should be able to adapt to words and formulations not previously encountered during training. This flexibility should ideally include inclusion of extraneous words (e.g. “turn on the lights, please”), new formulations of previously seen words/intents (e.g. “lights turn on”), truncation or omission of words (e.g. “bed lights on” instead of “bedroom”), and replacement with semantically equivalent words (e.g. “bathroom lights on” instead of “washroom”). It is important to note that some of these differentiations are more gracefully handled than others. In Fluent Speech Commands, authors note that while their pre-trained ASR-with-SLU-classifier model was able to correctly generalize to formulations seen in one slot value but not another (e.g. seen: “turn on the lights”,” turn off the lights”, “switch on the lights”; unseen: “switch off the lights”), it was not able to generalize to semantically equivalent words (seen: “turn on the washroom lights”, “turn on the bedroom lights”; unseen: “turn on the bathroom lights”). Unlike NLU systems which utilize word embeddings that encode distance in semantic space, SLU models trained piecewise have only learned to map phonetic embeddings to word-level embeddings without modeling how the word-level embeddings relate to one another. To achieve the flexibility of semantic equivalence, the pre-trained ASR module may be fine-tuned to output embeddings for word-like representations that encode some semantic structure so that the downstream intent classifier can learn to recognize semantically equivalent words within formulations not previously seen during training, even if the word embeddings are not being transcribed to text.

Several data augmentation techniques commonly used for audio/speech classification tasks also work for SLU. These include introducing competing background noise, time warping/stretching, pitch/frequency shifting, spectral augmentation, and room impulse response simulation. These techniques can be used to expand a dataset to improve accuracy and robustness to varying environments. In addition to conventional speech augmentation techniques, synthetic data techniques like text-to-speech, voice cloning, accent transfer, and voice changing can be used to expand a dataset. Adding additional synthetic examples to existing phrasings of real data can make the model more robust to new speakers and accents and improve overall accuracy. These techniques can also be used in moderation to add new phrasings within a given intent class that are not covered by the real dataset. While synthetic data techniques can be used to introduce entirely new classes into a dataset, even the highest performing commercial text-to-speech systems yield results that underperform real recorded examples or real examples augmented with synthetic data. A synthetic-only approach may reach acceptable accuracy for proof-of-concept or demo-level systems, which lower the time and cost associated with traditional data collection significantly.

Performance Metrics

If the SLU model has an explicit intent detection block, true positive rate (TPR) is often used to evaluate performance. This calculates the proportion of correctly identified intent sequences given an audio input. If the SLU model also has an explicit slot filling module that fills in predicted slots with the appropriate label, F1 score (the harmonic mean of the precision and recall) is most commonly used as a performance metric as it accounts for not only the true positive rate but the false rejection rate as well. Finally, overall accuracy is used to evaluate the proportion of cases in which both the intents and the slots were correctly identified and filled. For the ASR pretraining step, phoneme, grapheme, or word-level transcription accuracy (e.g. word error rate) is commonly used as a training objective and an evaluation metric. In the case of small, joint models with low complexity and without explicit blocks, such as those one might find in a smart coffee maker, the F1 score may be used to assess the overall performance of the model.

Edge Constraints and Considerations

In many small embedded applications, low complexity and narrow scope SLU models are an excellent choice for voice control systems. Models utilizing the approach found in Fluent Speech Commands can yield acceptable performance (>=90% TPR) with <1M parameters at INT8 precision. These models are able to perform 6 actions across 3 slots as found in FSC or enable a smart lighting assistant. When weight sparsity is exploited, models can be compressed up to 70-80% without loss of accuracy, making them deployable on small MCUs or accelerators. Running on cutting edge AI NPUs, these compressed models can run for 300-400 µW, including the front-end spectral processing, making them suitable for adding flexible voice command capabilities to small low-cost appliances or battery operated devices. Larger SLU models (~10MB) without explicit standalone ASR front ends can scale to much broader scenarios and are able to perform more complicated SLU tasks or on a wider range of actions/domains. This may be applicable to subsets of more complicated SLU datasets such as Snips or, custom datasets collected for a wide range of home automation and control, such as those found in a smart speaker or home hub. 

3.6 Voice Cloning / Text-to-Speech (TTS)

Problem Definition

Text-to-speech (TTS) synthesis converts written text into intelligible, natural-sounding speech waveforms, while voice cloning (also called speaker adaptation or voice conversion) aims to reproduce the vocal identity of a target speaker from a small number of reference utterances, ranging from several minutes down to a few seconds in “zero-shot” settings. Modern TTS pipelines typically decompose the task into (1) a text/linguistic front end that normalizes text and predicts phonetic and prosodic representations, (2) an acoustic model that maps these representations to an intermediate spectral representation (commonly a mel-spectrogram) or, in more recent systems, to discrete audio tokens, and (3) a vocoder or codec decoder that renders the intermediate representation into a raw waveform. Voice cloning introduces an additional speaker-conditioning pathway, typically a speaker embedding derived from a reference encoder, that is injected into the acoustic model or vocoder to steer timbre, prosody, and speaking style toward the target identity. On the edge, TTS and voice cloning power offline assistive-reading and accessibility tools, personalized voice assistants, in-vehicle navigation and infotainment systems, and, increasingly, on-device personalization of assistant voices all of which must operate without relying on continuous cloud connectivity and while respecting the sensitivity of biometric voice data.

Datasets and Benchmarks

Single- and multi-speaker corpora underpin the field. LJSpeech provides roughly 24 hours of single-speaker read speech commonly used to benchmark baseline model quality; LibriTTS extends LibriSpeech with speaker-segmented, higher-fidelity recordings suitable for multi-speaker training; VCTK contains about 110 English speakers reading the same prompts and is the de facto standard for evaluating speaker-conditioned and voice-cloning systems; HiFi-TTS offers higher sampling-rate (44.1 kHz) recordings for evaluating vocoder fidelity. Zero-shot cloning research increasingly relies on much larger, noisier web-scale corpora to expose models to thousands of speaker identities. Benchmarking generally combines objective signal metrics with listening tests (see Metrics, below); the field still lacks a widely adopted, hardware-aware benchmark analogous to MLPerf Tiny or SUPERB, which remains a persistent gap for edge TTS specifically.

Representative Architectures

Compact, Regression-Based Architectures

The dominant paradigm for constrained, on-device TTS remains a non-autoregressive acoustic model paired with a lightweight neural vocoder. Tacotron 2 established the attention-based sequence-to-sequence mel-spectrogram prediction paradigm, but its autoregressive decoding is slow and ill-suited to embedded latency budgets. FastSpeech 2 replaced autoregression with an explicit duration predictor and length regulator, enabling fully parallel mel-spectrogram generation and multiple-times-faster-than-real-time inference on mobile CPUs; Glow-TTS and VITS integrate normalizing flows (and, in VITS’s case, a variational auto-encoder and adversarial vocoder trained end to end) to close the naturalness gap with autoregressive systems while remaining parallelizable. On the vocoder side, HiFi-GAN and lightweight derivatives such as Multi-Band MelGAN and LPCNet generate waveforms an order of magnitude faster than autoregressive vocoders such as WaveNet or WaveRNN, and are the default choice for embedded deployment. Purpose-built compact systems such as Nix-TTS (a fully non-autoregressive, distilled, end-to-end model under 5M parameters) and LightSpeech (which applies neural-architecture search to the FastSpeech backbone) demonstrate that natural-sounding single- or few-speaker TTS is achievable within single-digit-megabyte memory budgets appropriate for mobile NPUs.

Large-Scale and Zero-Shot Cloning Architectures

A second family targets zero-shot voice cloning from a few seconds of reference audio, generally at the cost of significantly larger models better suited to cloud or high-end mobile SoCs, and increasingly distilled toward the edge. VALL-E and related codec-language-model approaches reframe TTS as an autoregressive language-modeling problem over discrete neural-audio-codec tokens, conditioning generation on codec tokens from a short reference clip to clone timbre without explicit speaker-embedding training; Bark and XTTS follow a similar codec-LM philosophy while emphasizing multilingual, cross-lingual cloning. StyleTTS 2 combines diffusion-based style modeling with adversarial training to achieve strong naturalness and speaker similarity from short references without a fully autoregressive backbone. For edge deployment, the practical pattern is to decouple the (heavier, sometimes cloud-assisted) speaker-embedding extraction step from a lightweight, on-device synthesis step: a compact reference encoder produces a fixed-length speaker embedding once, which is cached and reused by an on-device FastSpeech- or VITS-class model, avoiding the need to run a full codec-LM at inference time.

Training Approaches

Acoustic models are trained with a combination of reconstruction losses on the intermediate representation (L1/L2 mel-spectrogram loss), duration-prediction losses supervised by forced alignment, and, in flow- or GAN-based systems, adversarial and feature-matching losses against the vocoder’s discriminator. Multi-speaker and zero-shot cloning models add a speaker-conditioning objective, most often a contrastive or classification loss on a pretrained speaker-verification embedding (e.g., derived from an ECAPA-TDNN or d-vector encoder), jointly fine-tuned or frozen and injected into the acoustic model via conditioning layers such as adaptive instance normalization or cross-attention. Knowledge distillation is widely used to compress large, high-fidelity teacher models into compact student networks suitable for MCUs and mobile NPUs, typically combined with INT8 post-training quantization or quantization-aware fine-tuning of the acoustic model and vocoder separately, since vocoders are markedly more sensitive to quantization noise (audible artifacts) than acoustic models.

Edge Deployment

On-device deployments now report favorable quality-efficiency trade-offs: INT8-quantized FastSpeech 2 / VITS-class models achieve Mean Opinion Scores (MOS) above 4.0 on flagship smartphone NPUs, and distilled multi-speaker models can share a single vocoder across many voices, keeping the combined acoustic-model-plus-vocoder footprint under 10–20 MB. Streaming or chunk-based vocoding generating and playing audio in small windows rather than waiting for the full utterance further reduces perceived latency to sub-200 ms for short utterances, which is critical for assistant “read-back” interactions. On microcontroller-class targets, footprint constraints typically restrict deployment to single-speaker, low-sampling-rate (16 kHz) systems using heavily pruned LPCNet- or WaveRNN-style vocoders, and full zero-shot cloning generally remains out of reach without a cloud or companion-device assist for the embedding-extraction step.

Metrics

Mean Opinion Score (MOS) and its comparative variant (CMOS) remain the gold standard for perceived naturalness, gathered via crowdsourced listening tests, though they are costly and slow to run and are increasingly complemented by learned, non-intrusive predictors for rapid iteration. Speaker similarity is most commonly reported via cosine similarity between speaker-verification embeddings or, in security-oriented contexts, Equal Error Rate (EER) against a speaker-verification system used as an adversarial judge, which also serves as a de facto measure of resistance to unauthorized cloning. Intelligibility is measured via word error rate (WER) obtained by transcribing synthesized speech with a reference ASR system. On the edge, the same efficiency metrics used throughout this survey apply: inference latency (ms per utterance or per audio second, real-time factor), model memory footprint, and energy per synthesized second.

Edge Constraints and Considerations

Beyond raw efficiency, TTS and voice cloning raise distinct responsible-deployment questions. Because voice cloning can be misused for impersonation, fraud, and disinformation, several vocoders and codec-LM systems now embed imperceptible watermarks in synthesized audio, and edge deployments are increasingly expected to run a lightweight, on-device synthetic-speech detector alongside the synthesis pipeline itself, together with consent-gated enrollment flows for personalized cloning. These considerations, together with the strict latency and memory budgets already discussed, make TTS/voice-cloning one of the clearer examples in this survey of where evaluation must extend beyond acoustic quality to include provenance, consent, and misuse resistance.

3.7 Embedded Music Assistance

Problem Definition

AI-driven music assistance on the edge spans symbolic (MIDI/score-level) generation composition, continuation, accompaniment, and harmonization and audio-domain generation and processing style transfer, audio continuation, real-time accompaniment, and intelligent practice tools. Edge use cases include interactive digital instruments and practice aids that generate backing tracks or harmonize a live performance in real time, mobile creativity apps for hobbyist composition, and embedded “music assistants” in smart speakers or in-car systems that can generate short jingles, adapt existing tracks, or provide adaptive background music, all under strict latency constraints musically, generation must keep pace with tempo, often within tens of milliseconds per event for real-time accompaniment and, frequently, no-cloud-dependency requirements for offline instruments and IP-sensitive creative tools.

Datasets and Benchmarks

MAESTRO provides roughly 200 hours of aligned MIDI and audio piano performances and is the standard benchmark for symbolic and audio piano transcription/generation research; Lakh MIDI (LMD) offers on the order of 176,000 unique MIDI files spanning many genres and is widely used for large-scale symbolic pretraining; MusicNet supplies labeled multi-instrument classical recordings with aligned note annotations for transcription-adjacent tasks; the Free Music Archive (FMA) contributes a large, freely licensed corpus of full-length audio tracks across genres for audio-domain generative and representation-learning work; NSynth and Slakh2100 support instrument-timbre and source-separation-adjacent research relevant to embedded synthesis and accompaniment. As in TTS, benchmarks that jointly report generation quality and on-device efficiency are essentially absent, and most published edge-music work reports efficiency metrics ad hoc.

Architectures

Symbolic generation is dominated by transformer-based sequence models operating over tokenized MIDI events: Music Transformer introduced relative positional attention to capture the long-range, repetitive structure characteristic of music; MuseNet and follow-on GPT-style symbolic models scaled this approach across genres and instrumentations; MIDI-BERT adapts masked-language-model pretraining to symbolic music for representation learning and downstream tasks such as melody extraction and genre classification. Audio-domain generation increasingly relies on token-based language models over neural audio codecs  in the spirit of the codec-LM approach discussed in (3.6) as in MusicGen, or on diffusion models adapted to mel-spectrogram or latent audio representations, as in Riffusion and MusicLM-style systems; differentiable digital signal processing (DDSP) and RAVE offer a lighter-weight, more edge-amenable alternative by learning compact latent representations of timbre that can be decoded through fast, interpretable signal-processing vocoders rather than large neural vocoders, making them attractive for real-time, on-device timbre transfer and interactive instruments. Across both symbolic and audio domains, knowledge distillation and aggressive quantization compress multi-gigabyte research-scale models to tens of megabytes for mobile deployment, though full audio-domain generation remains the most resource-intensive task profiled in this survey and is generally restricted to short-form, cloud-assisted, or high-end mobile use cases today.

Training Approaches

Symbolic models are trained with next-token cross-entropy prediction over discretized MIDI event vocabularies (pitch, velocity, duration, time-shift tokens), analogous to text language modeling, and benefit from the same pretrain-then-fine-tune recipes used in NLP large-scale pretraining on Lakh MIDI followed by task- or style-specific fine-tuning. Audio-domain codec-LMs are trained autoregressively over discrete audio-codec tokens produced by a pretrained neural codec such as EnCodec, while diffusion-based systems are trained with a denoising objective over spectrogram or latent representations, typically conditioned on text or symbolic prompts via cross-attention. DDSP/RAVE-style models are trained with a combination of multi-scale spectral reconstruction loss and, in RAVE’s case, an adversarial and variational objective, and are notable for training efficiently on comparatively small, single- or few-instrument datasets, since they encode strong signal-processing priors rather than learning waveform synthesis from scratch.

Edge Deployment

Symbolic generation and DDSP/RAVE-style timbre transfer are the two sub-tasks most amenable to embedded deployment today: distilled symbolic transformers with tens of millions of parameters can run comfortably on mobile CPUs for interactive continuation and accompaniment, and RAVE-class models have been demonstrated running in real time on embedded platforms including Raspberry Pi-class hardware and dedicated embedded synthesizer modules at low-single-digit-millisecond latency per audio buffer. Full audio-domain generation via codec-LMs or diffusion remains largely cloud-hosted or reserved for higher-end mobile NPUs and flagship on-device AI accelerators, given both the model scale and the multi-second generation latency typical of current systems; a common edge pattern is therefore to generate short musical material once in the cloud or on a companion device and cache or loop it locally, rather than performing full generation on the constrained device itself.

Metrics

Fréchet Audio Distance (FAD) compares the distribution of generated and real audio in an embedding space and is the most widely reported objective metric for audio-domain music generation; CLAP-score measures alignment between generated audio and a text/audio joint embedding, useful for text-conditioned generation; symbolic generation is often assessed with information-theoretic measures, such as pitch-class and note-duration distribution similarity to a reference corpus, alongside task-specific measures of harmonic and rhythmic consistency. Because musicality and creative quality are inherently subjective, human evaluation typically pairwise preference or Likert-scale ratings of musicality, coherence, and creativity remains indispensable and is reported alongside objective metrics in essentially all major studies. On the edge, generation latency relative to musical tempo, i.e., whether the system can generate the next musical event before it is needed, is as important as absolute inference time, since a technically fast model that cannot keep pace with a live performer is unusable for interactive accompaniment.

Edge Constraints and Considerations

Embedded music assistance surfaces some of the same provenance and rights questions as voice cloning: generative systems trained on copyrighted recordings and scores raise ongoing questions about training-data licensing and attribution, which several commercial systems address through provenance metadata or watermarking of generated output. Real-time interactive use cases add a hard synchronization constraint largely absent elsewhere in this survey generated audio or symbolic events must be produced within a musically coherent window relative to a live performer or a fixed tempo grid making worst-case latency, rather than average latency, the operative constraint for on-device deployment.

3.8 Affective Computing (Speech Emotion Recognition)

Problem Definition

Affective computing for speech, most often studied as Speech Emotion Recognition (SER), aims to infer a speaker’s emotional or affective state categorical labels such as anger, happiness, sadness, and neutral, or continuous dimensions such as arousal and valence directly from acoustic and prosodic cues, independent of linguistic content. On the edge, SER supports adaptive user-experience features (e.g., a voice assistant that adjusts its response style to detected user frustration), in-cabin driver-monitoring systems that flag stress or drowsiness-adjacent affect, call-center and wellness applications that run entirely on-device for privacy, and accessibility tools that surface affective context to users with limited access to facial or tonal cues.

Datasets and Benchmarks

IEMOCAP remains the most widely used benchmark, offering roughly 12 hours of scripted and improvised dyadic interactions with categorical and dimensional emotion annotations; RAVDESS and CREMA-D provide acted, controlled emotional speech (and, in RAVDESS’s case, paired song and video) useful for clean baseline comparisons; MSP-IMPROV and the larger, in-the-wild MSP-Podcast corpus extend coverage to more naturalistic, less scripted affect. As with other emerging tasks in this survey, cross-corpus generalization is poor models trained on acted emotion corpora transfer weakly to spontaneous, in-the-wild speech and no edge-specific SER benchmark yet reports hardware efficiency alongside accuracy.

Representative Architectures

Classical pipelines extract hand-crafted prosodic and spectral features (pitch, energy, formants, MFCCs, and functionals thereof, e.g., via openSMILE) and feed them to lightweight classifiers such as SVMs or shallow feedforward networks; these remain attractive for microcontroller-class deployment given their small footprint. Deep-learning approaches typically use CNN or CNN-LSTM encoders over log-Mel spectrograms to jointly learn spectral and temporal affect cues, while more recent systems fine-tune self-supervised speech encoders such as wav2vec 2.0, HuBERT, or WavLM for SER, which improves accuracy substantially but at a much larger parameter and compute cost; distillation of these encoders into compact student models is an active area for bringing SER within edge budgets. Multimodal fusion with facial or physiological signals further improves accuracy in cloud or tethered-device settings but is generally out of scope for audio-only edge deployment.

Metrics

Unweighted average recall (UAR) is the standard metric for categorical SER, chosen specifically because emotion classes are typically imbalanced in both acted and naturalistic corpora; weighted F1 and concordance correlation coefficient (CCC) are commonly reported for dimensional (arousal/valence) prediction. As with other tasks in this survey, latency, model size, and energy per inference are the operative edge-efficiency metrics, though SER-specific hardware profiling is rarely reported in the literature to date.

Edge Constraints and Considerations

Affective computing is among the most ethically sensitive tasks discussed in this survey: emotional state is a highly personal, protected category of inference in several emerging biometric-privacy regulations, and SER models trained predominantly on Western, acted-emotion corpora carry a well-documented risk of cultural and demographic bias in how affect is expressed and recognized. On-device inference mitigates transmission risk but does not resolve consent or bias concerns, and deployments should pair SER with clear user disclosure and opt-in consent, alongside cross-cultural validation before production use.

3.9 Audio Anomaly Detection

Problem Definition

Audio anomaly detection identifies acoustic events or patterns that deviate from a learned model of “normal” sound, typically without labeled examples of the anomaly itself, since anomalies are by definition rare and diverse. Edge use cases include predictive maintenance (detecting bearing wear, motor faults, or HVAC malfunctions from machine sound), smart-building and safety monitoring (glass breaking, alarms, distress sounds), and environmental or industrial monitoring where continuous cloud streaming of raw audio is impractical or privacy-invasive. Because the task is framed as unsupervised or self-supervised outlier detection rather than closed-set classification, it differs architecturally from most other tasks in this survey.

Datasets and Benchmarks

The DCASE (Detection and Classification of Acoustic Scenes and Events) challenge series, and in particular its Task 2 sub-challenges built around the MIMII and ToyADMOS machine-sound datasets, is the de facto benchmark for industrial audio anomaly detection, providing normal-only training audio with anomalous test recordings across multiple machine types. MLPerf Tiny separately incorporates an anomaly-detection workload for standardized efficiency comparison on microcontroller targets. Beyond machine sound, urban and environmental anomaly detection draws on general acoustic-scene corpora (e.g., AudioSet subsets) adapted to an anomaly-detection framing, though standardized benchmarks here are less mature than for industrial applications.

Representative Architectures

The dominant paradigm is self-supervised reconstruction: an autoencoder (convolutional or recurrent) is trained only on normal-condition audio, and anomalies are flagged via elevated reconstruction error at inference time; variational autoencoders and normalizing-flow-based density models extend this to explicit likelihood-based scoring. A second family uses self-supervised contrastive or classification pretext tasks (e.g., predicting machine ID or operating condition from normal audio) and treats low confidence or embedding-space distance as an anomaly signal, which has shown strong results in recent DCASE challenge submissions. Both families are attractive for edge deployment because the underlying encoders are typically small CNNs operating on log-Mel spectrogram or MFCC input, comparable in size to keyword-spotting models, and inference requires only a forward pass and a threshold comparison rather than a full generative decode.

Metrics

Area under the ROC curve (AUC) and partial-AUC restricted to a low false-positive-rate region are the standard DCASE metrics, since operators typically require very low false-alarm rates in production monitoring; precision, recall, and F1 at a chosen operating threshold are also commonly reported. On the edge, the same latency, memory, and energy metrics used throughout this survey apply, with the added consideration that many anomaly-detection deployments run continuously (always-on), making sustained energy draw, rather than per-inference energy alone, the more relevant figure of merit.

Edge Constraints and Considerations

Because normal-condition audio can itself be sensitive (e.g., recordings from inside a home or workplace), on-device processing is often a stronger privacy requirement here than a convenience; deployments should minimize any retained audio to short buffered windows needed for detection rather than persistent recordings. Model drift is a distinct edge challenge for this task: what counts as “normal” can shift over a device’s lifetime (e.g., as machinery wears or a room’s ambient noise floor changes), motivating lightweight on-device recalibration or federated updates rather than a purely static, factory-trained model.

Cross-Sectional Observation

Across all tasks, two consistent findings emerge:

  • Quantization-aware training and knowledge distillation remain the most impactful techniques for preserving accuracy while reducing resource footprint.
  • Evaluation gaps persist; few studies jointly report accuracy and hardware-level efficiency, underscoring the need for standardized edge benchmarks.

Cross-Task Efficiency and Accuracy

The table below consolidates the representative model sizes, latency/efficiency figures, and primary accuracy metrics reported across (3.1)–(3.9), to make the accuracy-versus-footprint trade-off directly comparable across tasks at a glance. Figures reflect representative edge-optimized systems discussed in this survey rather than absolute state-of-the-art ceilings, which are typically larger and cloud-hosted.

Task (§)Typical Model SizeTypical LatencyTypical PowerPrimary Metric(s)
Wake Word / KWS10–250 KB<10 ms<1 mW (always-on)TPR/FRR @ FAR
Speech Enhancement0.5–10 MB5–20 ms/frame1–50 mWPESQ, STOI, SI-SDR
ASR (on-device)10–80 MB50–300 ms/utt.50–500 mWWord Error Rate (WER)
Speaker Biometrics1–20 MB20–100 ms5–100 mWEER
Spoken Lang. Understanding<1–10 MB10–50 ms0.3–5 mWTPR / F1
Voice Cloning / TTS5–20 MB~200 ms/utt.20–200 mWMOS, Speaker Sim. EER
Embedded Music Assistance10–100+ MBms–seconds50–500+ mWFAD, CLAP-score, MOS
Affective Computing (SER)1–10 MB10–50 ms1–20 mWUAR, CCC
Audio Anomaly Detection10–250 KB<10 ms<1 mW (always-on)AUC, pAUC
OPPORTUNITY LENSThe following section moves from individual audio tasks to the cross-cutting conditions that determine whether systems can be evaluated, trusted, and deployed responsibly.

Cross-Cutting Issues

4.1 Evaluation Landscape

Evaluating Audio AI models for edge deployment requires more than traditional task accuracy or signal-quality metrics. While PESQ, WER, and EER remain the dominant measures across subfields, they fail to capture the hardware performance and user experience implications that determine real-world usability. Recent initiatives, such as MLPerf Tiny (Banbury et al., 2021) and HEAR (Turian et al., 2022), highlight a growing need for multidimensional benchmarking that combines algorithmic performance with system-level metrics, including latency, memory footprint, throughput, and energy per inference.

Nevertheless, a persistent fragmentation remains. For example, speech-enhancement studies frequently report only PESQ and STOI, omitting runtime metrics, whereas keyword-spotting papers emphasize microcontroller energy efficiency but not robustness under noise or accent variation (Wang & Li, 2022). This inconsistency hampers comparability and reproducibility across research groups.
A promising direction involves unified evaluation frameworks that jointly report (i) task performance, (ii) hardware utilization, and (iii) environmental robustness, allowing fairer assessment of trade-offs between accuracy and efficiency.

4.2 Dataset Gaps and Representation Bias

Despite substantial progress, the data foundation for edge-audio AI remains biased toward high-resource languages and clean laboratory recordings. Datasets like LibriSpeech or VoxCeleb are English-centric and studio-recorded, leaving limited coverage for low-resource languages, dialectal variation, or field recordings common in real-world contexts (Guimarães et al., 2023). Such imbalances lead to models that generalize poorly in noisy, multilingual, or culturally diverse settings, conditions typical of deployments in the Global majority. Moreover, small-footprint edge models are often distilled from large teacher networks trained on biased data, which can potentially amplify these biases after compression.
Future datasets should therefore emphasize diverse acoustic scenes, accent inclusivity, and community-based annotation pipelines, particularly for underrepresented languages. This aligns with broader efforts toward responsible AI and data justice in low-resource contexts.

4.3 Privacy and Security

Audio signals are inherently personal: they may encode identity, emotion, and even health information. Edge deployment offers a partial solution by processing data locally, but risks persist. Speaker embeddings and voiceprints can be exploited if stored insecurely, and on-device caching of speech segments may expose private content. To mitigate this, research increasingly integrates federated learning, secure enclaves, and homomorphic encryption to protect user data (Wang & Jia, 2025). Yet these methods raise computational overhead that conflicts with edge efficiency. Balancing privacy and performance remains a core design dilemma.

4.4 Ethical Concerns and Responsible Audio AI

Beyond privacy, ethical challenges include:

  • Deepfake misuse in voice cloning and synthetic media (Zhao et al., 2023).
  • Accent and demographic bias in ASR and speaker recognition systems.
  • Environmental sustainability, given the carbon footprint of training teacher models for distillation.

Responsible Audio AI requires transparent disclosure of datasets, open reporting of energy and bias audits, and adherence to ethical principles for the creation of synthetic voices. Community-driven initiatives and open benchmarks must adopt ethical licenses and data-attribution standards to prevent misuse while encouraging innovation.

5  Edge Hardware and Deployment

5.1 Hardware Tiers

Edge AI spans a continuum of hardware from microcontrollers (MCUs) with kilobytes of RAM to smartphones with NPUs capable of several TOPS (tera-operations per second).

  • MCU-class devices (e.g., Arm Cortex-M4, ESP32) handle ultra-light KWS or VAD models with <250 kB flash.
  • Mobile-class SoCs (e.g., Qualcomm Hexagon DSPs, Apple Neural Engine) support quantized ASR and TTS.
  • Dedicated accelerators/NPUs enable multi-task pipelines or real-time audio synthesis at <1 W power.

5.2 Benchmarking and Profiling

Benchmarks for edge-audio tasks remain nascent. MLPerf Tiny defines standardized workloads for keyword spotting, visual wake words, and anomaly detection (Banbury et al., 2021). However, audio-centric multi-task evaluation is still missing.
Profiling must therefore extend beyond accuracy to include:

  • Inference latency (ms / frame).
  • Energy per inference (µJ).
  • Model size (MB) and memory footprint (RAM).
  • Thermal load and throughput under concurrent tasks.

Researchers increasingly report energy–accuracy Pareto curves to quantify trade-offs (Wang et al., 2025). A unified reporting protocol could standardize comparisons across device classes.

5.3 Software–Hardware Co-Design

Joint optimization of model architecture and hardware parameters is emerging as a key theme. Techniques include:

  • Neural architecture search (NAS) constrained by latency and energy budgets.
  • Dynamic voltage and frequency scaling (DVFS) for adaptive inference.
  • Mixed-precision quantization guided by sensitivity analysis.

Such co-design principles yield end-to-end efficiency improvements exceeding 40% without significant degradation in accuracy (Tambe et al., 2022). As TinyML matures, automated pipelines linking NAS, quantization, and compiler optimization will become central to deploying Audio AI universally from hearing aids to IoT sensors.

5.4 Deployment Challenges

Despite technological advances, several bottlenecks persist:

  • Heterogeneous hardware complicates portability and reproducibility.
  • OTA updates of models raise firmware security concerns.
  • Thermal and battery constraints limit sustained real-time inference on wearables.
  • Debugging and profiling tools for audio latency are still immature compared to those for vision tasks.

Open Challenges and Future Directions

Despite remarkable progress in deploying audio models on resource-constrained devices, several scientific and ethical challenges remain unresolved. These issues define the frontier for future research on responsible, robust, and inclusive Audio AI at the edge.

6.1 Joint and Multi-Task Modeling

Most edge-AI solutions remain narrowly specialized (e.g., separate models for keyword spotting, speaker verification, or speech enhancement). Integrating these into joint, multi-task architectures could reduce redundancy and improve efficiency. However, co-training multiple objectives under tight resource budgets increases the risk of task interference and instability. Future work should explore modular or mixture-of-experts approaches that enable selective task activation depending on user context or energy budget.

6.2 Model Compression without Quality Loss

Quantization and pruning techniques have reached maturity; yet, achieving ultra-compact models (<1 MB) without sacrificing perceptual quality, particularly for TTS or ASR, remains challenging. Hybrid compression strategies that combine distillation, low-rank decomposition, and adaptive quantization hold promise. Another research direction involves energy-aware training, where loss functions directly penalize high power consumption, ensuring models are efficient by design rather than by post-hoc compression.

6.3 Cross-Lingual and Low-Resource Robustness

Current benchmarks focus heavily on English and Mandarin, neglecting thousands of languages that dominate speech in the Global South regions. Cross-lingual transfer and self-supervised representation learning could help bridge this divide. Establishing community-curated audio datasets for underrepresented languages and dialects would allow evaluation of fairness, accent robustness, and inclusion, particularly relevant for applications in healthcare and education.

6.4 Responsible and Ethical Edge Audio

Edge computing mitigates some privacy risks but does not eliminate ethical concerns. The proliferation of voice cloning and deepfake tools necessitates the development of new detection and provenance mechanisms. Future research must address:

  • Verifiable audio authenticity, e.g., watermarking or cryptographic signatures.
  • Transparent model documentation, specifying data sources, bias audits, and energy costs.
  • On-device explainability, enabling users to understand model decisions.

These principles align with global efforts toward trustworthy and sustainable AI, ensuring that Audio AI systems remain beneficial across diverse cultural contexts.

6.5 Standardization of Edge Performance Reporting

The absence of uniform reporting protocols hinders reproducibility. Establishing standard metadata descriptors that cover accuracy, latency, energy, and hardware context would enable meaningful comparisons across studies. Similar to MLPerf’s reproducibility framework, a dedicated Audio Edge Benchmark could unify metrics for speech enhancement, recognition, and synthesis, encouraging transparency and cross-disciplinary collaboration.

Conclusion

Audio AI at the edge represents a convergence of signal processing, embedded systems, and ethical AI research. Through this survey, we have synthesized recent advances across seven core tasks ranging from wake-word detection to music generation, highlighting the shared technical foundations and design trade-offs that govern performance on constrained hardware.

Our analysis reveals that while quantization-aware training, pruning, and knowledge distillation have enabled substantial progress, evaluation fragmentation, linguistic inequity, and ethical vulnerabilities remain major obstacles. Addressing these requires not only algorithmic innovation but also interdisciplinary collaboration between engineers, ethicists, linguists, and policy experts.

By mapping open research gaps and proposing unified evaluation principles, this work aims to guide future developments toward responsible, efficient, and inclusive Audio AI ecosystems capable of serving diverse global communities.

Recommendations

PRIORITY ACTIONEstablish a working-group position.
Outline:This is what we observeMarket trendsOver supplied vs. blue ocean vs. nicheResearch trendsWhat is working, what is notWhat needs more attentionIntersection between market and research?This is what we would like to seeWhat does the future look like?What is underserved and why is it important?This is what we recommend working on/researchingThese are the research opportunitiesThese are the business opportunities
Different perspectives:Hardware Developers/ArchitectsSoftware DevelopersApplication DevelopersBusiness Professionals/Founders

References

Banbury, C., Reddi, V. J., Lam, M., Fu, W., Holleman, J., Huang, J., & others. (2021). MLPerf Tiny Benchmark. In Proceedings of the MLSys Conference.

Bushur, J., et al. (2023). Neural network exploration for keyword spotting on edge devices. Future Internet, 15(6), 219. https://doi.org/10.3390/fi15060219

CEA-Leti. (2024). Accurate speech recognition at power consumption below one microwatt [Press release, ISSCC 2024].

Guimarães, H., Pimentel, A., Ávila, A., Rezagholizadeh, M., & Falk, T. H. (2023). An exploration into the performance of unsupervised cross-task speech representations for “in-the-wild” edge applications. arXiv preprint arXiv:2305.05443.

Kerpicci, M., Nguyen, V., Zhang, S., & Visser, E. (2022). Application of knowledge distillation to multi-task speech representation learning. In Proceedings of Interspeech 2022.

Lin, J., & Lyu, R.-Y. (2023). Compact CNNs for end-to-end keyword spotting on resource-constrained edge AI devices. In Proceedings of ROCLING 2023 (pp. 231–239). Association for Computational Linguistics.

Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audiobooks. In Proceedings of ICASSP 2015 (pp. 5206–5210). IEEE.

Reddy, C. K., Dubey, H., Gopal, V., Cutler, R., Matusevych, S., Aichner, R., … & Gomez, J. (2020). The Interspeech 2020 Deep Noise Suppression Challenge. arXiv preprint arXiv:2001.08662.

Singh, R. (2023). Edge AI: A survey. AI Open, 4, 180–205. https://doi.org/10.1016/j.aiopen.2023.07.004

Surianarayanan, C., Rajan, A., & Chandrasekaran, K. (2023). A survey on optimization techniques for edge artificial intelligence. Electronics, 12(3), 691. https://doi.org/10.3390/electronics12030691

Tambe, T., Kulkarni, P., Luo, Y., Huang, Q., & Wang, Y. (2022). A 16-nm SoC for noise-robust speech and NLP edge AI. In Proceedings of ISSCC 2022 (pp. 154–156). IEEE.

Turian, J., McDermott, J., & Bellet, A. (2022). HEAR: Holistic evaluation of audio representations. In NeurIPS Datasets and Benchmarks Track (2022).

Wang, J., & Li, S. (2022). Keyword spotting system and evaluation of pruning and quantization methods on low-power edge microcontrollers. arXiv preprint arXiv:2208.02765.

Wang, X., & Jia, W. (2025). Optimizing Edge AI: A comprehensive survey on data, model, and system strategies. arXiv preprint arXiv:2501.03265.

Warden, P. (2018). Speech commands: A dataset for limited-vocabulary speech recognition. In Proceedings of Interspeech 2018 (pp. 801–805). ISCA.

Yang, S. W., Chi, P. H., Chuang, Y. S., Lin, C. I., Liu, A. T., Huang, J., … & Lee, H. Y. (2021). SUPERB: Speech processing universal Performance benchmark. In Proceedings of Interspeech 2021 (pp. 1194–1198). ISCA.

Zhao, Y., Wang, Y., Li, J., & Xu, X. (2023). Audio deepfakes: A review on threats, countermeasures, and future directions. IEEE Access, 11, 124309–124325. https://doi.org/10.1109/ACCESS.2023.3314982

Li, J., Wu, Y., Chen, G., & Subramanian, A. (2020). On-device neural ASR. IEEE Signal Processing Magazine.

Jurafsky, D., & Martin, J. H. (2023). Speech and Language Processing (3rd ed.).

Papastratis, I. (2021). Speech Recognition: a review of the different deep learning approaches. Accessed on, 2, 2021.

Ahlawat, H., Aggarwal, N., & Gupta, D. (2025). Automatic Speech Recognition: A survey of deep learning techniques and approaches. International Journal of Cognitive Computing in Engineering.

Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems (NeurIPS).

Besacier, L., Barnard, E., Karpov, A., & Schultz, T. (2014). Automatic speech recognition for under-resourced languages: A survey. Speech Communication, 56, 85–100.

Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M. (2021). Unsupervised cross-lingual representation learning for speech recognition. Proceedings of Interspeech 2021.

Guimarães, V., et al. (2023). Efficient deployment of Conformer-based ASR models through INT8 quantization. IEEE Signal Processing Letters.

Lakhotia, K., et al. (2021). A comparative study of self-supervised speech representation learning. IEEE Signal Processing Magazine, 38(6), 82–95.

Radford, A., et al. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of ICML 2023.

Shi, J., et al. (2023). Multilingual speech recognition: A survey. IEEE/ACM Transactions on Audio, Speech, and Language Processing.

Yang, S., Wang, Y., & Xie, L. (2021).
Unsupervised speech recognition: A review. arXiv preprint arXiv:2106.04897.

Zhang, S., Wu, J., & Yu, D. (2023).
End-to-end speech recognition: A survey. arXiv preprint arXiv:2303.03329.

Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). LibriSpeech: An ASR corpus based on public domain audio books. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5206–5210. https://doi.org/10.1109/ICASSP.2015.7178964

Kahn, J., Lee, A., Hannun, A., & Auli, M. (2020). Libri-Light: A benchmark for ASR with limited or no supervision. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 7669–7673. https://doi.org/10.1109/ICASSP40776.2020.9052974

Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., … Agarwal, S. (2020). Common Voice: A massively-multilingual speech corpus. Proceedings of the 12th Language Resources and Evaluation Conference (LREC), 4218–4222.

Bu, H., Du, J., Na, X., Wu, B., & Zheng, H. (2017). AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline. Proceedings of the 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), 1–5. https://doi.org/10.1109/OCOCOSDA.2017.8359860

Du, J., Na, X., Liu, X., & Bu, H. (2018). AISHELL-2: Transforming Mandarin ASR research into industrial scale. Proceedings of the 24th International Conference on Pattern Recognition (ICPR), 365–370. https://doi.org/10.1109/ICPR.2018.8545669

Hernandez, F., Nguyen, V., Ghannay, S., Tomashenko, N., & Estève, Y. (2018). TED-LIUM 3: Twice as much data and corpus repartition for experiments on speaker adaptation. Proceedings of the 20th International Conference on Speech and Computer (SPECOM), 198–208.

Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992). SWITCHBOARD: Telephone speech corpus for research and development. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 517–520. https://doi.org/10.1109/ICASSP.1992.225858

Cieri, C., Miller, D., & Walker, K. (2004). The Fisher Corpus: A resource for the next generations of speech-to-text. Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), 69–71.

Linguistic Data Consortium. (2000). 2000 HUB5 English evaluation speech corpus. University of Pennsylvania.

Barker, J., Watanabe, S., Vincent, E., & Trmal, J. (2018). The fifth CHiME speech separation and recognition challenge: Dataset, task and baselines. Proceedings of Interspeech 2018, 1561–1565. https://doi.org/10.21437/Interspeech.2018-5

Chen, G., Chou, C.-L., Yeh, C.-K., Lee, H.-Y., & Hsu, W. (2021). GigaSpeech: An evolving, multi-domain ASR corpus with 10,000 hours of transcribed audio. Proceedings of Interspeech 2021, 3670–3674. https://doi.org/10.21437/Interspeech.2021-1175

Yamagishi, J., Veaux, C., MacDonald, K., & King, S. (2019). CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit. University of Edinburgh.

Gupta, A., et al. (2021). SPGISpeech: A large-scale corpus for financial domain speech recognition. Proceedings of Interspeech 2021, 1947–1951.

Kocabiyikoglu, A. C., Besacier, L., & Kraif, O. (2018). Augmenting LibriSpeech with French translations: A multimodal corpus for direct speech translation evaluation. Proceedings of the 11th Language Resources and Evaluation Conference (LREC).

Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020).
wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 12449–12460.

Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021).
HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451–3460. https://doi.org/10.1109/TASLP.2021.3122291

Chen, S., Wu, Y., Wang, C., Chen, Z., Liu, S., & Yao, Y. (2022).
WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6), 1505–1518. https://doi.org/10.1109/JSTSP.2022.3188118

Babu, A., et al. (2021).
XLS-R: Self-supervised cross-lingual speech representation learning at scale. Proceedings of Interspeech 2021, 181–185. https://doi.org/10.21437/Interspeech.2021-1385

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023).
Robust speech recognition via large-scale weak supervision. Proceedings of the 40th International Conference on Machine Learning (ICML), 28492–28518.

Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., & Pang, R. (2020).
Conformer: Convolution-augmented Transformer for speech recognition. Proceedings of Interspeech 2020, 5036–5040. https://doi.org/10.21437/Interspeech.2020-3015

Kuchaiev, O., et al. (2019).
Models for streaming speech recognition with RNN-Transducer. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 6261–6265. https://doi.org/10.1109/ICASSP.2019.8682525

Watanabe, S., Hori, T., Kim, S., Hershey, J. R., & Hayashi, T. (2018).
ESPnet: End-to-end speech processing toolkit. Proceedings of Interspeech 2018, 2207–2211. https://doi.org/10.21437/Interspeech.2018-1456

Ravanelli, M., et al. (2021).
SpeechBrain: A general-purpose speech toolkit. Proceedings of Interspeech 2021, 423–427. https://doi.org/10.21437/Interspeech.2021-933

Povey, D., et al. (2011).
The Kaldi speech recognition toolkit. Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU).

Ott, M., et al. (2019).
fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACL-HLT 2019: Demonstrations, 48–53. https://doi.org/10.18653/v1/N19-4009

Hannun, A., et al. (2014).
Deep Speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567.

NVIDIA. (2021). NVIDIA Riva: Speech AI SDK. NVIDIA Developer Documentation.

Google. (2020). Cloud Speech-to-Text documentation. Google Cloud.

Bredin, H., Yin, R., Coria, J. M., Gelly, G., Korshunov, P., Lavechin, M., Fustes, D., Titeux, H., Bouaziz, W., & Gill, M.-P. (2020). pyannote.audio: Neural building blocks for speaker diarization. Proceedings of ICASSP 2020, 7124-7128.

Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Karaiskos, V., Kraaij, W., Kronenthal, M., Lathoud, G., Lincoln, M., Lisowska, A., McCowan, I., Post, W., Reidsma, D., & Wellner, P. (2005). The AMI Meeting Corpus: A pre-announcement. Proceedings of MLMI 2005, 28-39.

Chung, J. S., Huh, J., Nagrani, A., Afouras, T., & Zisserman, A. (2020). Spot the conversation: Speaker diarisation in the wild. Proceedings of Interspeech 2020.

Chung, J. S., Nagrani, A., & Zisserman, A. (2018). VoxCeleb2: Deep speaker recognition. Proceedings of Interspeech 2018.

Desplanques, B., Thienpondt, J., & Demuynck, K. (2020). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN based speaker verification. Proceedings of Interspeech 2020, 3830-3834.

Fan, Y., Kang, J. W., Li, L. T., Li, K. C., Chen, H. L., Cheng, S. T., Zhang, P. Y., Zhou, Z. Y., Cai, Y. Q., & Wang, D. (2020). CN-Celeb: A challenging Chinese speaker recognition dataset. Proceedings of ICASSP 2020, 7604-7608.

Lin, Q., Yang, L., Wang, X., Qin, X., Wang, J., & Li, M. (2022). Towards lightweight applications: Asymmetric enroll-verify structure for speaker verification. Proceedings of ICASSP 2022.

Nagrani, A., Chung, J. S., Xie, W., & Zisserman, A. (2019). VoxCeleb: Large-scale speaker verification in the wild. Computer Speech & Language, 60, 101027.

National Institute of Standards and Technology. (n.d.). Speaker Recognition Evaluation (SRE). NIST.

Ryant, N., Singh, P., Krishnamohan, V., Varma, R., Church, K., Cieri, C., Du, J., Ganapathy, S., & Liberman, M. (2021). The Third DIHARD Diarization Challenge. Proceedings of Interspeech 2021.

Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. (2018). X-vectors: Robust DNN embeddings for speaker recognition. Proceedings of ICASSP 2018, 5329-5333.

Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K. A., Kinnunen, T., Evans, N., & Delgado, H. (2021). ASVspoof 2021: Accelerating progress in spoofed and deepfake speech detection. arXiv preprint arXiv:2109.00537.

Nagrani, A., Chung, J. S., & Zisserman, A. (2017). VoxCeleb: A large-scale speaker identification dataset. Proceedings of Interspeech 2017.

Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerry-Ryan, R. J., Saurous, R. A., Agiomyrgiannakis, Y., & Wu, Y. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. Proceedings of ICASSP 2018, 4779–4783.

Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T.-Y. (2021). FastSpeech 2: Fast and high-quality end-to-end text to speech. Proceedings of ICLR 2021.

Kim, J., Kong, J., & Son, J. (2021). VITS: Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. Proceedings of ICML 2021, 5530–5540.

Kim, J., Kim, S., Kong, J., & Yoon, S. (2020). Glow-TTS: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33, 8067–8077.

Kong, J., Kim, J., & Bae, J. (2020). HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33, 17022–17033.

van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.

Kalchbrenner, N., Elsen, E., Simonyan, K., Noury, S., Casagrande, N., Lockhart, E., Stimberg, F., van den Oord, A., Dieleman, S., & Kavukcuoglu, K. (2018). Efficient neural audio synthesis. Proceedings of ICML 2018, 2410–2419.

Valin, J.-M., & Skoglund, J. (2019). LPCNet: Improving neural speech synthesis through linear prediction. Proceedings of ICASSP 2019, 5891–5895.

Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brébisson, A., Bengio, Y., & Courville, A. (2019). MelGAN: Generative adversarial networks for conditional waveform synthesis. Advances in Neural Information Processing Systems, 32.

Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., & Wei, F. (2023). Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111.

Li, Y. A., Han, C., Jiang, X., & Mesgarani, N. (2023). StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. arXiv preprint arXiv:2306.07691.

Casanova, E., Davis, K., Gölge, E., Göknar, G., Gulea, I., Hart, L., Aljafari, A., Meyer, J., Morais, R., Olayemi, S., & Weber, J. (2024). XTTS: A massively multilingual zero-shot text-to-speech model. Proceedings of Interspeech 2024.

Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., & Wu, Y. (2019). LibriTTS: A corpus derived from LibriSpeech for text-to-speech. Proceedings of Interspeech 2019, 1526–1530.

Ito, K., & Johnson, L. (2017). The LJ Speech Dataset. https://keithito.com/LJ-Speech-Dataset/

Bakhturina, E., Lavrukhin, V., Ginsburg, B., & Zhang, Y. (2021). Hi-Fi multi-speaker English TTS dataset. Proceedings of Interspeech 2021, 2776–2780.

Chevi, R., & Purwarianti, A. (2022). Nix-TTS: Lightweight and end-to-end text-to-speech via module-wise distillation. Proceedings of IEEE Spoken Language Technology Workshop (SLT) 2022.

Luo, R., Tan, X., Wang, R., Qin, T., Li, J., Zhao, S., Chen, E., & Liu, T.-Y. (2021). LightSpeech: Lightweight and fast text to speech with neural architecture search. Proceedings of ICASSP 2021, 5699–5703.

Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2018). Music Transformer: Generating music with long-term structure. arXiv preprint arXiv:1809.04281.

Payne, C. (2019). MuseNet. OpenAI.

Chou, Y.-H., Chen, I., Chang, C.-J., Ching, J., & Yang, Y.-H. (2021). MidiBERT-Piano: Large-scale pre-training for symbolic music understanding. arXiv preprint arXiv:2107.05223.

Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., & Défossez, A. (2023). Simple and controllable music generation. Advances in Neural Information Processing Systems, 36.

Forsgren, S., & Martiros, H. (2022). Riffusion: Stable diffusion for real-time music generation.

Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., & Frank, C. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.

Engel, J., Hantrakul, L., Gu, C., & Roberts, A. (2020). DDSP: Differentiable digital signal processing. Proceedings of ICLR 2020.

Caillon, A., & Esling, P. (2021). RAVE: A variational autoencoder for fast and high-quality neural audio synthesis. arXiv preprint arXiv:2111.05011.

Défossez, A., Copet, J., Synnaeve, G., & Adi, Y. (2022). High fidelity neural audio compression. arXiv preprint arXiv:2210.13438.

Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., & Eck, D. (2019). Enabling factorized piano music modeling and generation with the MAESTRO dataset. Proceedings of ICLR 2019.

Raffel, C. (2016). Learning-based methods for comparing sequences, with applications to audio-to-MIDI alignment and matching. PhD thesis, Columbia University.

Thickstun, J., Harchaoui, Z., & Kakade, S. (2017). Learning features of music from scratch. Proceedings of ICLR 2017.

Defferrard, M., Benzi, K., Vandergheynst, P., & Bresson, X. (2017). FMA: A dataset for music analysis. Proceedings of ISMIR 2017.

Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., & Simonyan, K. (2017). Neural audio synthesis of musical notes with WaveNet autoencoders. Proceedings of ICML 2017, 1068–1077.

Manilow, E., Wichern, G., Seetharaman, P., & Le Roux, J. (2019). Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity. Proceedings of WASPAA 2019.

Kilgour, K., Zuluaga, M., Roblek, D., & Sharifi, M. (2019). Fréchet Audio Distance: A metric for evaluating music enhancement algorithms. Proceedings of Interspeech 2019, 2350–2354.

Elizalde, B., Deshmukh, S., Al Ismail, M., & Wang, H. (2023). CLAP: Learning audio concepts from natural language supervision. Proceedings of ICASSP 2023.

Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008). IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4), 335–359.

Livingstone, S. R., & Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS). PLOS ONE, 13(5).

Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014). CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4), 377–390.

Lotfian, R., & Busso, C. (2019). Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings (MSP-Podcast). IEEE Transactions on Affective Computing, 10(4), 471–483.

Eyben, F., Wöllmer, M., & Schuller, B. (2010). openSMILE: The Munich versatile and fast open-source audio feature extractor. Proceedings of ACM Multimedia 2010, 1459–1462.

Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33.

Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021). HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 3451–3460.

Koizumi, Y., Kawaguchi, Y., Imoto, K., Nakamura, T., Nikaido, Y., Tanabe, R., Purohit, H., Suefusa, K., Endo, T., Yasuda, M., & Harada, N. (2020). Description and discussion on DCASE 2020 challenge task 2: Unsupervised anomalous sound detection for machine condition monitoring. Proceedings of DCASE 2020 Workshop.

Purohit, H., Tanabe, R., Ichige, K., Endo, T., Nikaido, Y., Suefusa, K., & Kawaguchi, Y. (2019). MIMII Dataset: Sound dataset for malfunctioning industrial machine investigation and inspection. Proceedings of DCASE 2019 Workshop.

Koizumi, Y., Saito, S., Uematsu, H., Harada, N., & Imoto, K. (2019). ToyADMOS: A dataset of miniature-machine operating sounds for anomalous sound detection. Proceedings of IEEE WASPAA 2019.

Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017). AudioSet: An ontology and human-labeled dataset for audio events. Proceedings of ICASSP 2017.