Table of Contents

A practitioner's guide to digital signal processing, speech and audio understanding, generative sound, and real-time voice systems.

Hands-On AI Science Series · In Production

10 parts · 48 chapters · 336 sections · 47 hands-on labs, plus front matter, 7 appendices, and a capstone. This is the planning table of contents; titles link to the corresponding heading in the full book plan.

Front Matter · Why This Book Exists

5 entries
  1. F1
    Why This Book ExistsAudio is everywhere, yet many practitioners treat it as a spectrogram-then-CNN black box; this book makes audio AI a first-class discipline.
  2. F2
    What This Book CoversThe complete arc from sound physics and DSP to foundation models, speech and music generation, voice agents, and production engineering.
  3. F3
    Who Should Read This BookStudents and engineers who want to build audio AI systems, not only call speech APIs.
  4. F4
    How to Use This BookReading paths for audio ML engineers, speech and generative-audio specialists, production builders, and researchers.
  5. F5
    Running Tools and DatasetsThe core Python stack and the representative datasets used throughout the labs.

Part I · Sound, Signals & Audio Data

5 chapters · 35 sections

Make readers fluent in the physical, mathematical, and computational nature of audio before models enter the story.

  1. 1
    What Is Audio Intelligence? Audio AI turns continuous pressure waves into discrete representations, decisions, and generated sound; the whole book in miniature.
    1. 1.1 Sound as pressure, vibration, and information
    2. 1.2 Human hearing and machine hearing
    3. 1.3 Audio tasks: classification, transcription, separation, synthesis, retrieval, dialogue
    4. 1.4 Speech, music, environmental sound, bioacoustics, industrial acoustics
    5. 1.5 The audio AI pipeline: capture → representation → model → output
    6. 1.6 Latency, noise, privacy, and real-world constraints
    7. 1.7 Book map: hear, understand, speak, deploy
    8. Lab Load microphone recordings, plot waveforms, compute duration, loudness, clipping, silence ratio, and basic metadata.
  2. 2
    Digital Audio Foundations How a sound becomes a number: sampling, quantization, formats, and the hygiene every audio dataset depends on.
    1. 2.1 Sampling theorem and aliasing
    2. 2.2 Quantization, bit depth, dynamic range
    3. 2.3 Mono, stereo, multichannel, sample rates
    4. 2.4 File formats: WAV, FLAC, MP3, AAC, Opus
    5. 2.5 Resampling and sample-rate conversion
    6. 2.6 Normalization, clipping, gain, loudness units
    7. 2.7 Dataset hygiene and audio metadata
    8. Lab Implement a small audio loader/resampler and demonstrate aliasing by downsampling a chirp with and without anti-alias filtering.
  3. 3
    Time-Domain Signal Processing Before any transform, audio is a stream of samples; framing, filtering, and energy are the tools you reach for first.
    1. 3.1 Signals as arrays and streams
    2. 3.2 Convolution and filtering in time
    3. 3.3 FIR and IIR filters
    4. 3.4 Envelopes, zero crossings, RMS energy
    5. 3.5 Autocorrelation and periodicity
    6. 3.6 Windowing and frame-based processing
    7. 3.7 Streaming buffers and overlap-add
    8. Lab Build a frame-based energy detector and compare it with zero-crossing based voiced/unvoiced detection.
  4. 4
    Frequency-Domain Audio Every sound is a sum of sines; the STFT and mel spectrogram are the representations the rest of the book runs on.
    1. 4.1 Sine waves and Fourier intuition
    2. 4.2 DFT and FFT from first principles
    3. 4.3 STFT, spectrograms, and time-frequency tradeoff
    4. 4.4 Mel scale, filterbanks, MFCCs
    5. 4.5 Constant-Q transform and music representations
    6. 4.6 Phase, magnitude, and reconstruction
    7. 4.7 Spectrogram design choices for ML
    8. Lab Implement STFT and inverse STFT, then compare linear spectrograms, mel spectrograms, and MFCCs on speech, music, and environmental sounds.
  5. 5
    Audio Data Engineering Curation, segmentation, augmentation, and leakage-safe splits: the unglamorous work that decides whether a model is real.
    1. 5.1 Dataset curation and label noise
    2. 5.2 Segmentation and event boundaries
    3. 5.3 Silence trimming and voice activity detection
    4. 5.4 Augmentation: noise, reverb, speed, pitch, mixup
    5. 5.5 Train/test splits for speakers, rooms, and recording devices
    6. 5.6 Leakage in audio datasets
    7. 5.7 Reproducible audio pipelines
    8. Lab Create a leakage-safe split for an audio event dataset and show how random clip splitting overestimates performance.

Part II · Classical Audio Machine Learning

4 chapters · 28 sections

Establish durable baselines and interpretable methods before deep networks.

  1. 6
    Classical Features for Audio Recognition Spectral, tonal, and rhythmic features plus a shallow classifier still win when data is scarce and latency is tight.
    1. 6.1 Frame-level features
    2. 6.2 Spectral centroid, bandwidth, rolloff, flux
    3. 6.3 Chroma and tonal features
    4. 6.4 Rhythm and tempo features
    5. 6.5 Feature aggregation over clips
    6. 6.6 Scaling and feature selection
    7. 6.7 When classical features still win
    8. Lab Train a scikit-learn classifier on handcrafted audio features and compare feature importance across sound classes.
  2. 7
    Classical Speech Recognition Foundations HMMs, Gaussian mixtures, lexicons, and Viterbi: the ideas deep learning replaced but never erased.
    1. 7.1 Phones, phonemes, words, and language models
    2. 7.2 Hidden Markov models
    3. 7.3 Gaussian mixtures and acoustic modeling
    4. 7.4 Pronunciation lexicons
    5. 7.5 N-gram language models
    6. 7.6 Viterbi decoding
    7. 7.7 Why deep learning replaced but did not erase these ideas
    8. Lab Implement a toy HMM recognizer for a small command vocabulary.
  3. 8
    Audio Classification Baselines Framing, metrics, calibration, and error analysis by listening: baselines that survive review.
    1. 8.1 Problem framing: clip-level, frame-level, event-level
    2. 8.2 Logistic regression, SVMs, random forests
    3. 8.3 Imbalanced labels and multilabel audio
    4. 8.4 Metrics: accuracy, F1, mAP, event F-score
    5. 8.5 Calibration and thresholds
    6. 8.6 Error analysis with listening
    7. 8.7 Baseline reports that survive review
    8. Lab Build an ESC-50 baseline and produce a confusion matrix with representative audio examples for each error type.
  4. 9
    Source Separation and Beamforming Before Deep Learning The cocktail-party problem, solved classically with masking, NMF, ICA, and arrays, as an inductive bias for neural models.
    1. 9.1 The cocktail party problem
    2. 9.2 Spectral masking
    3. 9.3 Non-negative matrix factorization
    4. 9.4 Independent component analysis
    5. 9.5 Microphone arrays and beamforming
    6. 9.6 Wiener filtering
    7. 9.7 Classical separation as an inductive bias for neural models
    8. Lab Separate simple music mixtures using NMF and evaluate with SDR-like metrics.

Part III · Deep Learning for Audio Understanding

6 chapters · 42 sections

Build neural audio models from small CNNs to transformers and self-supervised encoders.

  1. 10
    Neural Audio Representations Waveform or spectrogram? Learnable filterbanks, receptive fields, and pooling decide what a network can hear.
    1. 10.1 Waveform input versus spectrogram input
    2. 10.2 1D convolution over waveforms
    3. 10.3 2D convolution over spectrograms
    4. 10.4 Learnable filterbanks
    5. 10.5 Receptive fields in audio
    6. 10.6 Pooling over time and frequency
    7. 10.7 Choosing representation by task and latency
    8. Lab Compare waveform CNN, mel-CNN, and MFCC-MLP on a keyword classification task.
  2. 11
    CNNs for Audio Classification Spectrograms as images, and where the analogy breaks; VGG- and ResNet-style tagging, transfer learning, and its failure modes.
    1. 11.1 Spectrograms as images, and where the analogy breaks
    2. 11.2 VGG-like audio CNNs
    3. 11.3 ResNet-style audio models
    4. 11.4 Efficient models for edge audio
    5. 11.5 Multilabel audio tagging
    6. 11.6 Transfer learning from pretrained audio CNNs
    7. 11.7 Failure modes: shortcuts, background bias, microphone bias
    8. Lab Fine-tune a pretrained audio tagging model on a custom environmental sound dataset.
  3. 12
    Sequence Models for Audio Audio is temporal; RNNs, TCNs, and Conformers, streaming versus non-streaming, and inference with cached state.
    1. 12.1 Why audio is temporal
    2. 12.2 RNNs, GRUs, LSTMs
    3. 12.3 Temporal convolutional networks
    4. 12.4 Conformer blocks
    5. 12.5 Streaming versus non-streaming models
    6. 12.6 Handling long recordings
    7. 12.7 Online inference and state caching
    8. Lab Build a streaming keyword spotter with chunked inference and latency measurement.
  4. 13
    Audio Transformers Attention over time-frequency patches: the Audio Spectrogram Transformer, long-context attention, and compute trade-offs.
    1. 13.1 Attention over frames and patches
    2. 13.2 Audio Spectrogram Transformer
    3. 13.3 Positional encoding for time-frequency patches
    4. 13.4 Long-context audio attention
    5. 13.5 Masking strategies for audio
    6. 13.6 Fine-tuning pretrained audio transformers
    7. 13.7 Compute and memory tradeoffs
    8. Lab Fine-tune an Audio Spectrogram Transformer and compare it to a CNN baseline.
  5. 14
    Self-Supervised Audio Learning Labels are scarce; contrastive prediction, wav2vec 2.0, HuBERT, and masked spectrogram modeling learn from raw sound.
    1. 14.1 Why labels are scarce in audio
    2. 14.2 Contrastive predictive coding
    3. 14.3 wav2vec 2.0 style masked prediction
    4. 14.4 HuBERT and pseudo-labeling
    5. 14.5 BYOL-A and non-contrastive learning
    6. 14.6 AudioMAE and masked spectrogram modeling
    7. 14.7 Transfer, probing, and representation evaluation
    8. Lab Use a pretrained self-supervised encoder as a frozen feature extractor for downstream classification.
  6. 15
    Robustness and Generalization in Audio Models Noise, reverb, devices, and domains: room impulse responses, adaptation, open-set recognition, and stress testing.
    1. 15.1 Noise, reverberation, devices, and domains
    2. 15.2 Room impulse responses
    3. 15.3 Far-field and near-field audio
    4. 15.4 Domain adaptation
    5. 15.5 Adversarial and accidental perturbations
    6. 15.6 Open-set recognition
    7. 15.7 Stress testing audio systems
    8. Lab Create a robustness benchmark by adding controlled noise, reverb, compression, and device simulation.

Part IV · Speech Recognition, Speaker AI & Spoken Language

6 chapters · 42 sections

The core speech technologies behind dictation, transcription, diarization, and spoken interfaces.

  1. 16
    Automatic Speech Recognition: Problem and Metrics ASR as sequence transduction; targets, word error rate, alignment, and the streaming constraints that shape every design.
    1. 16.1 ASR as sequence transduction
    2. 16.2 Character, subword, and word targets
    3. 16.3 Word error rate and character error rate
    4. 16.4 Alignment, substitutions, insertions, deletions
    5. 16.5 Domain vocabulary and proper nouns
    6. 16.6 Streaming ASR constraints
    7. 16.7 Human transcription as a noisy reference
    8. Lab Evaluate an ASR model on clean, noisy, accented, and domain-specific audio.
  2. 17
    CTC Speech Recognition Alignment without labels: blanks, the forward-backward algorithm, greedy and beam decoding, and language-model fusion.
    1. 17.1 The alignment problem
    2. 17.2 CTC blanks and collapse rules
    3. 17.3 Forward-backward dynamic programming
    4. 17.4 Greedy decoding
    5. 17.5 Beam search and language model fusion
    6. 17.6 Training stability
    7. 17.7 CTC in modern ASR systems
    8. Lab Train a small CTC recognizer on a toy speech command corpus.
  3. 18
    Encoder-Decoder and Transducer ASR Attention, RNN-T, and Conformer encoders; timestamping, multilingual ASR, and Whisper-style weak supervision.
    1. 18.1 Attention-based encoder-decoder ASR
    2. 18.2 RNN-T and streaming transduction
    3. 18.3 Conformer encoders
    4. 18.4 Decoding with external language models
    5. 18.5 Timestamping and word alignment
    6. 18.6 Multilingual and code-switching ASR
    7. 18.7 Whisper-style weakly supervised ASR
    8. Lab Fine-tune a modern ASR model and build a transcript post-processing pipeline.
  4. 19
    Speaker Recognition and Verification Identity, not content: speaker embeddings, x-vectors and ECAPA, EER and DET curves, spoofing, and biometric risk.
    1. 19.1 Speaker identity versus speech content
    2. 19.2 Speaker embeddings
    3. 19.3 x-vectors and ECAPA-TDNN
    4. 19.4 Verification metrics: EER, DET curves
    5. 19.5 Spoofing and presentation attacks
    6. 19.6 Privacy and biometric risk
    7. 19.7 Speaker-aware applications
    8. Lab Build a speaker verification demo with enrollment, scoring, and threshold calibration.
  5. 20
    Diarization: Who Spoke When? Segment, embed, cluster; overlapped speech, neural diarization, and aligning speakers with the transcript.
    1. 20.1 Segmentation, embedding, clustering
    2. 20.2 Overlapped speech
    3. 20.3 Neural diarization
    4. 20.4 Diarization error rate
    5. 20.5 Meeting transcription pipelines
    6. 20.6 Speaker naming and identity linking
    7. 20.7 Diarization in production
    8. Lab Run diarization on a meeting recording and align speakers with ASR transcript segments.
  6. 21
    Spoken Language Understanding From speech to meaning: intent, slots, end-to-end SLU, prosody, emotion, and spoken dialogue state.
    1. 21.1 Intent detection from speech
    2. 21.2 Slot filling and semantic parsing
    3. 21.3 End-to-end spoken language understanding
    4. 21.4 Prosody and paralinguistic cues
    5. 21.5 Emotion and affect recognition
    6. 21.6 Spoken dialogue state tracking
    7. 21.7 Multimodal spoken interaction
    8. Lab Build a spoken command system that combines ASR, intent classification, and slot extraction.

Part V · Audio Events, Scenes & Monitoring

4 chapters · 28 sections

Non-speech sound understanding across home, city, industry, bioacoustics, healthcare, and security.

  1. 22
    Environmental Sound Classification Weak labels and clip-level supervision over AudioSet-style taxonomies, multi-instance learning, and rare events.
    1. 22.1 Sound classes and acoustic scenes
    2. 22.2 Weak labels and clip-level supervision
    3. 22.3 AudioSet-style taxonomies
    4. 22.4 Multi-instance learning
    5. 22.5 Event localization in time
    6. 22.6 Class imbalance and rare events
    7. 22.7 Human-in-the-loop labeling
    8. Lab Train a multilabel environmental audio classifier and tune thresholds for different alert costs.
  2. 23
    Acoustic Scene Analysis Scene versus event: scene classification, sound event detection, temporal smoothing, and evaluation with collars.
    1. 23.1 Scene versus event
    2. 23.2 Background texture
    3. 23.3 Acoustic scene classification
    4. 23.4 Sound event detection
    5. 23.5 Temporal smoothing and post-processing
    6. 23.6 Evaluation with event collars
    7. 23.7 Deployment in noisy spaces
    8. Lab Build a sound event detector that outputs a timeline of events for a 10-minute recording.
  3. 24
    Industrial and Mechanical Audio AI Machines as sound sources: unsupervised anomaly detection, domain shift, and edge monitoring for motors, pumps, and bearings.
    1. 24.1 Machines as sound sources
    2. 24.2 Vibration-acoustic relation
    3. 24.3 Anomaly detection without labels
    4. 24.4 Predictive maintenance audio datasets
    5. 24.5 Domain shift across machines
    6. 24.6 Edge monitoring constraints
    7. 24.7 Case studies: motors, pumps, bearings, leaks
    8. Lab Build an unsupervised anomaly detector for machine sound using embeddings and reconstruction error.
  4. 25
    Bioacoustics and Healthcare Audio Birdsong, cough, and auscultation: weak labels, clinical validation, consent, and discovery from sound archives.
    1. 25.1 Animal vocalization analysis
    2. 25.2 Birdsong and species recognition
    3. 25.3 Cough, breathing, and auscultation signals
    4. 25.4 Medical label quality and clinical validation
    5. 25.5 Privacy and consent in health audio
    6. 25.6 Low-resource acoustic domains
    7. 25.7 Scientific discovery from sound archives
    8. Lab Construct a bioacoustic event detector and evaluate it under weak labels.

Part VI · Speech Synthesis & Generative Voice

5 chapters · 35 sections

Build the synthesis stack from vocoders through neural TTS and expressive, controllable speech.

  1. 26
    Speech Synthesis Foundations Text normalization, grapheme-to-phoneme, prosody, and the acoustic-model-plus-vocoder split every TTS system inherits.
    1. 26.1 Text normalization
    2. 26.2 Grapheme-to-phoneme conversion
    3. 26.3 Prosody, pitch, duration, energy
    4. 26.4 Concatenative and parametric synthesis
    5. 26.5 Neural acoustic models
    6. 26.6 Vocoders
    7. 26.7 Evaluating synthetic speech
    8. Lab Build a small text normalization and phoneme conversion pipeline.
  2. 27
    Neural Vocoders From spectrogram to waveform: Griffin-Lim, WaveNet, WaveRNN, GAN and diffusion vocoders, judged by listening.
    1. 27.1 From spectrogram to waveform
    2. 27.2 Griffin-Lim as a classical baseline
    3. 27.3 WaveNet and autoregressive vocoding
    4. 27.4 WaveRNN and efficiency
    5. 27.5 GAN vocoders: MelGAN, HiFi-GAN
    6. 27.6 Diffusion vocoders
    7. 27.7 Artifacts and listening tests
    8. Lab Compare Griffin-Lim and a pretrained neural vocoder on the same mel spectrograms.
  3. 28
    Text-to-Speech Models Tacotron, FastSpeech, and VITS; multispeaker and expressive TTS, voice cloning, and misuse prevention.
    1. 28.1 Tacotron-style sequence-to-sequence TTS
    2. 28.2 FastSpeech and non-autoregressive TTS
    3. 28.3 VITS and end-to-end TTS
    4. 28.4 Multispeaker TTS
    5. 28.5 Voice cloning and adaptation
    6. 28.6 Expressive TTS and style control
    7. 28.7 Responsible TTS and misuse prevention
    8. Lab Fine-tune a small TTS model on a speaker dataset and evaluate pronunciation, prosody, and artifacts.
  4. 29
    Voice Conversion and Speech Editing Disentangling content, speaker, and style; enhancement, inpainting, accent conversion, watermarking, and provenance.
    1. 29.1 Content, speaker, and style disentanglement
    2. 29.2 Any-to-one and any-to-any voice conversion
    3. 29.3 Speech enhancement and denoising
    4. 29.4 Speech inpainting and editing
    5. 29.5 Accent conversion and pronunciation correction
    6. 29.6 Watermarking and provenance
    7. 29.7 Evaluation beyond MOS
    8. Lab Build a voice conversion demo using pretrained content and speaker embeddings.
  5. 30
    Real-Time Conversational Speech Output Streaming and chunked synthesis under a latency budget, with barge-in, dialogue prosody, and safety filters.
    1. 30.1 Streaming TTS
    2. 30.2 Chunked synthesis
    3. 30.3 Latency budgets
    4. 30.4 Interruptibility and barge-in
    5. 30.5 Prosody for dialogue
    6. 30.6 Safety filters for generated speech
    7. 30.7 TTS serving architecture
    8. Lab Build a streaming TTS service and measure first-audio latency, total latency, and quality tradeoffs.

Part VII · Music, Sound Design & Audio Generation

5 chapters · 35 sections

Generative audio beyond speech: symbolic music, waveform generation, diffusion, audio-text models, and sound design.

  1. 31
    Music AI Foundations Notes, pitch, rhythm, timbre, and harmony; symbolic versus audio representations and the music-information-retrieval toolkit.
    1. 31.1 Notes, pitch, rhythm, timbre, harmony
    2. 31.2 Symbolic versus audio music representations
    3. 31.3 MIDI and piano rolls
    4. 31.4 Chroma and key estimation
    5. 31.5 Beat tracking and onset detection
    6. 31.6 Music information retrieval
    7. 31.7 Evaluation of music models
    8. Lab Build MIDI parsing, piano-roll visualization, and simple melody statistics.
  2. 32
    Symbolic Music Generation From Markov melodies to Transformer tokenizations: structure, conditioning, human editing, and copyright.
    1. 32.1 Markov models for melody
    2. 32.2 RNN music models
    3. 32.3 Transformer tokenizations for music
    4. 32.4 Structure, repetition, and long-range coherence
    5. 32.5 Conditioning on chord, style, and form
    6. 32.6 Human editing loops
    7. 32.7 Copyright and dataset issues
    8. Lab Train a small Transformer to continue symbolic melodies.
  3. 33
    Neural Audio Generation Waveform, spectrogram, and latent generation; autoregressive, GAN, diffusion, and flow matching, with guidance and control.
    1. 33.1 Waveform generation
    2. 33.2 Spectrogram generation
    3. 33.3 Autoregressive audio models
    4. 33.4 GAN-based audio synthesis
    5. 33.5 Diffusion and flow matching for audio
    6. 33.6 Latent audio generation
    7. 33.7 Sampling, guidance, and controllability
    8. Lab Use a pretrained text-to-audio model and analyze how prompt changes affect generated spectrograms and waveforms.
  4. 34
    Text-to-Audio and Audio-to-Audio Models CLAP-style embeddings, audio captioning, text-guided generation, style transfer, and semantic-alignment evaluation.
    1. 34.1 Audio-language pairs
    2. 34.2 CLAP-style contrastive embeddings
    3. 34.3 Audio captioning
    4. 34.4 Text-guided audio generation
    5. 34.5 Audio style transfer
    6. 34.6 Sound effect generation
    7. 34.7 Evaluation of semantic alignment
    8. Lab Build an audio-text retrieval system and compare retrieval using text queries, audio queries, and captions.
  5. 35
    Controllable and Interactive Sound Design Latent sliders, timbre transfer, prompt-to-sound, and inpainting for game, film, and human-in-the-loop creation.
    1. 35.1 Control signals and latent sliders
    2. 35.2 Timbre transfer
    3. 35.3 Prompt-to-sound workflows
    4. 35.4 Sound inpainting
    5. 35.5 Game and film sound generation
    6. 35.6 Human-in-the-loop creative systems
    7. 35.7 Dataset and rights management
    8. Lab Create a prompt-conditioned sound design notebook for short sound effects with versioned generations.

Part VIII · Audio-Language Models & Voice Agents

4 chapters · 28 sections

Connect audio AI with LLMs, multimodal models, and production conversational agents.

  1. 36
    Audio Embeddings and Cross-Modal Retrieval Joint audio-text spaces for semantic search, indexing long recordings, and multimodal RAG over meetings and media.
    1. 36.1 Joint audio-text embedding spaces
    2. 36.2 Contrastive learning for audio-language
    3. 36.3 Audio search and semantic retrieval
    4. 36.4 Indexing long recordings
    5. 36.5 Multimodal RAG over meetings and media
    6. 36.6 Retrieval evaluation
    7. 36.7 Cost and latency of audio retrieval
    8. Lab Build semantic search over a podcast or lecture archive using audio embeddings and transcript chunks.
  2. 37
    Audio Captioning and Question Answering From classification to captioning and QA, grounding answers in time, and handling hallucination and uncertainty.
    1. 37.1 From sound classification to captioning
    2. 37.2 Encoder-decoder captioning
    3. 37.3 Audio question answering datasets
    4. 37.4 Grounding answers in time
    5. 37.5 Hallucination and uncertainty
    6. 37.6 Long-audio reasoning
    7. 37.7 Evaluation with humans and LLM judges
    8. Lab Build an audio QA prototype that answers questions about a short recording with cited time intervals.
  3. 38
    Realtime Voice Agents Full-duplex interaction: VAD and endpointing, the ASR-LLM-TTS cascade versus native speech-to-speech, turn-taking, and safety.
    1. 38.1 Full-duplex voice interaction
    2. 38.2 Voice activity detection and endpointing
    3. 38.3 ASR → LLM → TTS cascade
    4. 38.4 Native speech-to-speech models
    5. 38.5 Turn-taking, interruption, and barge-in
    6. 38.6 Memory and context in voice systems
    7. 38.7 Safety, consent, and identity
    8. Lab Build a real-time voice assistant with VAD, streaming ASR, LLM response generation, streaming TTS, and interruption handling.
  4. 39
    Meeting, Lecture, and Media Intelligence Ingest, transcribe, diarize, summarize, and search organizational audio, with privacy built into the architecture.
    1. 39.1 Recording ingestion
    2. 39.2 Transcription and diarization
    3. 39.3 Summarization and chaptering
    4. 39.4 Action items and decisions
    5. 39.5 Audio-grounded search
    6. 39.6 Privacy in organizational audio
    7. 39.7 Product architecture
    8. Lab Build a meeting intelligence app that produces speaker-attributed notes, action items, and searchable audio segments.

Part IX · Spatial, Multichannel & Edge Audio

3 chapters · 21 sections

Handle audio in physical spaces and on constrained devices.

  1. 40
    Spatial Audio and Microphone Arrays Sound in rooms: interaural cues, direction of arrival, beamforming revisited, ambisonics, and neural spatial audio.
    1. 40.1 Sound propagation and rooms
    2. 40.2 Interaural time and level differences
    3. 40.3 Direction of arrival
    4. 40.4 Beamforming revisited
    5. 40.5 Binaural audio and ambisonics
    6. 40.6 Neural spatial audio
    7. 40.7 Applications in AR, robotics, meetings, and surveillance
    8. Lab Simulate a microphone array and estimate direction of arrival under noise.
  2. 41
    Audio on Edge Devices TinyML for audio: keyword spotting on microcontrollers, quantization and pruning, wake words, and battery and privacy trade-offs.
    1. 41.1 TinyML for audio
    2. 41.2 Keyword spotting on microcontrollers
    3. 41.3 Quantization and pruning
    4. 41.4 Streaming memory budgets
    5. 41.5 ONNX, TFLite, and embedded deployment
    6. 41.6 Wake words and always-on listening
    7. 41.7 Battery, thermal, and privacy tradeoffs
    8. Lab Quantize a keyword spotting model and measure size, speed, and accuracy.
  3. 42
    Real-Time Audio Systems Engineering Buffers, callbacks, jitter, and underruns; WebRTC, streaming protocols, GPU batching, observability, and graceful degradation.
    1. 42.1 Audio I/O stacks
    2. 42.2 Buffers, callbacks, jitter, and underruns
    3. 42.3 WebRTC and browser audio
    4. 42.4 Server-side streaming protocols
    5. 42.5 GPU batching versus latency
    6. 42.6 Observability for audio streams
    7. 42.7 Failure handling and graceful degradation
    8. Lab Build a browser-to-server streaming audio pipeline with latency instrumentation.

Part X · Evaluation, Trust, Deployment & Future Directions

6 chapters · 42 sections

Turn models into measurable, reliable, responsible systems.

  1. 43
    Evaluation of Audio AI Systems Task metrics, listening tests, subgroup evaluation, calibration, and regression tests that gate a production launch.
    1. 43.1 Task-specific metrics
    2. 43.2 Listening tests and MOS
    3. 43.3 Dataset bias and subgroup evaluation
    4. 43.4 Robustness benchmarks
    5. 43.5 Calibration and abstention
    6. 43.6 Regression tests for audio systems
    7. 43.7 Evaluation reports for production launch
    8. Lab Create a unified evaluation harness for ASR, classification, and TTS components.
  2. 44
    Privacy, Security, and Responsible Audio AI Audio as sensitive data: consent, biometrics, deepfake speech, watermarking, secure pipelines, and governance.
    1. 44.1 Audio as sensitive data
    2. 44.2 Consent and recording policy
    3. 44.3 Biometric identifiers
    4. 44.4 Deepfake speech and misuse
    5. 44.5 Watermarking and provenance
    6. 44.6 Secure audio pipelines
    7. 44.7 Governance and documentation
    8. Lab Write a model card and data card for a voice AI system, including privacy and misuse sections.
  3. 45
    Deploying Audio AI Batch, streaming, and real-time serving; scaling ASR and TTS, caching, drift monitoring, and cost-latency-reliability trade-offs.
    1. 45.1 Batch, streaming, and real-time deployments
    2. 45.2 Model serving choices
    3. 45.3 Scaling ASR and TTS
    4. 45.4 Caching and reuse
    5. 45.5 Monitoring quality drift
    6. 45.6 Human review workflows
    7. 45.7 Cost, latency, and reliability tradeoffs
    8. Lab Deploy a small audio inference service with monitoring and a regression test suite.
  4. 46
    Applications and Case Studies Call centers, accessibility, in-cabin sensing, healthcare, industry, creative tools, and education, end to end.
    1. 46.1 Call centers and customer support
    2. 46.2 Accessibility and assistive technology
    3. 46.3 Automotive and in-cabin sensing
    4. 46.4 Healthcare and remote monitoring
    5. 46.5 Industrial monitoring
    6. 46.6 Creative tools and games
    7. 46.7 Education, lectures, and language learning
    8. Lab Design an end-to-end case study blueprint for one chosen application, including data, model, evaluation, deployment, and risk controls.
  5. 47
    Frontier Research in Audio AI Speech foundation models, universal encoders, audio-language-action models, and real-time speech-to-speech reasoning.
    1. 47.1 Speech foundation models
    2. 47.2 Universal audio encoders
    3. 47.3 Audio-language-action models
    4. 47.4 Generative audio at long duration
    5. 47.5 Real-time speech-to-speech reasoning
    6. 47.6 Multimodal agents with hearing
    7. 47.7 Open research problems
    8. Lab Replicate a recent audio foundation model result at small scale using frozen encoders and lightweight adapters.
  6. 48
    Capstone: An End-to-End Audio AI System Choose a problem, plan the data, build a baseline and a modern model, evaluate honestly, deploy, and present.
    1. 48.1 Problem selection
    2. 48.2 Data plan and recording protocol
    3. 48.3 Baseline and modern model
    4. 48.4 Evaluation harness
    5. 48.5 Real-time or batch deployment
    6. 48.6 Responsible use review
    7. 48.7 Final report and demo

Appendices · Reference & Pedagogy

7 appendices
  1. A
    Mathematical Foundations for Audio AIComplex numbers, Fourier transforms, convolution, probability, entropy, cross-correlation, dynamic programming, and optimization.
  2. B
    PyTorch and Torchaudio PrimerTensors, datasets, dataloaders, transforms, GPU training, mixed precision, streaming inference, and checkpointing.
  3. C
    Audio Datasets and BenchmarksDataset cards, licenses, task mapping, common pitfalls, leakage warnings, and benchmark protocols.
  4. D
    Evaluation Metrics ReferenceWER, CER, PER, DER, EER, SDR, SI-SDR, PESQ, STOI, FAD, CLAPScore, mAP, event F-score, and MOS protocols.
  5. E
    Microphones, Audio Interfaces, and Edge HardwareMicrophone types, polar patterns, arrays, preamps, ADCs, noise floors, embedded boards, and inference trade-offs.
  6. F
    Course SyllabiUndergraduate and graduate audio and speech tracks, a professional voice-AI bootcamp, and a generative-audio seminar.
  7. G
    Solutions to Selected ExercisesWorked solutions for mathematical derivations, coding exercises, and system-design questions.