Audio Video And Multimodal
Music & Audio Generation
Speech has one structural requirement: be intelligible. Music has several, operating at very different timescales at once.
Jr Codex Generative AI Notes
Level: Intermediate Prerequisites: Chapter 1: Speech — TTS and Transcription Time to complete: ~15 minutes
Table of Contents
- What Makes Music Harder Than Speech
- Audio Tokens — the Dominant Approach
- Diffusion on Spectrograms
- Prompting for Music
- Where It Fits in Real Work
- The Licensing Question
- Summary & Next Steps
1. What Makes Music Harder Than Speech
Speech has one structural requirement: be intelligible. Music has several, operating at very different timescales at once.
Music's Nested Timescales
─────────────────────────────────────────
milliseconds timbre, attack, transients
~0.5 seconds individual notes
~2 seconds a rhythmic pattern
~8 seconds a phrase
~30 seconds a section (verse, chorus)
~3 minutes overall song structure
All of them must be coherent SIMULTANEOUSLY, and a
listener notices a failure at ANY level.
─────────────────────────────────────────
Why This Is a Hard Modelling Problem
─────────────────────────────────────────
A model that gets timbre right and structure wrong
produces something that sounds good for four seconds
and aimless for three minutes.
This is the long-range coherence problem from
Module 1, Chapter 4 — in its most demanding form,
because the listener is tracking repetition and
return explicitly.
─────────────────────────────────────────
2. Audio Tokens — the Dominant Approach
The winning strategy converts audio into a discrete sequence, then reuses the entire autoregressive Transformer machinery from the NLP & LLM Notes.
Residual Vector Quantisation, in Outline
─────────────────────────────────────────
A neural codec encodes ~50 audio frames per second.
Each frame becomes several STACKED tokens:
level 1 ──► coarse structure, pitch, rhythm
level 2 ──► refinement of level 1's error
level 3 ──► finer timbral detail
level 4 ──► the remaining fine texture
Each level encodes what the previous one MISSED —
hence "residual".
─────────────────────────────────────────
Why the Stacking Matters
─────────────────────────────────────────
Generation can be staged: model the COARSE tokens
first over a long window (capturing structure), then
fill in finer levels conditioned on them.
This is the same divide-and-conquer as generating an
image at low resolution and upscaling it (Module 3,
Chapter 3) — solve structure cheaply at low fidelity,
then add detail.
─────────────────────────────────────────
Once audio is a token sequence, everything transfers: a decoder-only Transformer, text conditioning through cross-attention, temperature and top-p sampling (Module 2, Chapter 2). This is why audio generation improved so quickly — it inherited the LLM stack wholesale.
3. Diffusion on Spectrograms
The alternative treats a mel spectrogram as an image and runs Module 3's latent diffusion on it unchanged.
The Two Approaches Compared
─────────────────────────────────────────
Autoregressive Spectrogram
audio tokens diffusion
─────────────────────────────────────────
Long structure stronger weaker
Generation serial, slower parallel, faster
Editing hard easy — inpaint a
region of the
spectrogram
Fine texture excellent good; vocoder
artefacts possible
─────────────────────────────────────────
The trade is exactly the one from Module 1, Chapter 2: autoregressive models win at long-range coherence, diffusion wins at parallel speed and editability. Nothing new — the same trade-off in a new modality.
4. Prompting for Music
Music prompts sit between image prompts and text prompts: descriptive like a caption, but the useful vocabulary is technical.
The Slots That Work
─────────────────────────────────────────
GENRE lo-fi hip hop, baroque, post-rock
INSTRUMENTATION upright bass, brushed drums, Rhodes
TEMPO / FEEL 85 BPM, laid back, swung
MOOD melancholy, warm, unhurried
PRODUCTION tape saturation, wide stereo, dry
STRUCTURE builds after 20 seconds, no vocals
─────────────────────────────────────────
Example
─────────────────────────────────────────
"lo-fi hip hop, 85 BPM, brushed drums, upright bass,
warm Rhodes chords, vinyl crackle, melancholy but
unhurried, no vocals, loopable"
─────────────────────────────────────────
import torch, soundfile as sf
from transformers import AutoProcessor, MusicgenForConditionalGeneration
proc = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small").to("cuda")
prompts = [
"lo-fi hip hop, 85 BPM, brushed drums, upright bass, warm Rhodes chords, "
"vinyl crackle, melancholy but unhurried, no vocals, loopable",
]
inputs = proc(text=prompts, padding=True, return_tensors="pt").to("cuda")
audio = model.generate(
**inputs,
do_sample=True, # SAMPLING, not greedy — Module 2, Chapter 2 applies here too
guidance_scale=3.0, # the same CFG idea as images (Module 3, Chapter 2)
max_new_tokens=512, # ~10s: audio TOKENS, not seconds
)
rate = model.config.audio_encoder.sampling_rate
sf.write("bed.wav", audio[0, 0].cpu().numpy(), rate)Note how little is new. do_sample, guidance_scale, max_new_tokens — the audio model exposes the same three controls as a text model and an image model, because underneath it is an autoregressive Transformer with classifier-free guidance.
Two Rules
─────────────────────────────────────────
1. NAME INSTRUMENTS, not adjectives. "Upright bass
and brushed drums" constrains far more than
"jazzy".
2. SPECIFY WHAT SHOULD NOT HAPPEN structurally —
"no vocals", "no key change", "loopable" — since
these are the properties most likely to drift and
ruin a usable take.
─────────────────────────────────────────
5. Where It Fits in Real Work
Genuinely Useful Today
─────────────────────────────────────────
Background beds video, podcast, presentation
underscores. Short, loopable,
deliberately unobtrusive.
Sound design one-shot effects, textures,
ambiences, foley variants.
Sketching conveying an arrangement idea
before booking a session.
Stem separation isolating vocals or drums from a
mix — a mature, reliable tool.
Not There Yet
─────────────────────────────────────────
- Full songs with coherent lyrics and structure
- Precise editing ("keep everything, change the
bassline") — regeneration is not editing
- Exact-length delivery to a video cue point
- Consistent instrumentation across a set of tracks
─────────────────────────────────────────
The practical pattern is the one from Module 1, Chapter 4: generate many candidates cheaply, let a human select. Music generation is a shortlisting tool, not a delivery tool.
6. The Licensing Question
Music sits in a sharper legal position than text or images, and it is worth knowing why before you ship anything.
Three Separable Rights
─────────────────────────────────────────
COMPOSITION the written work — melody, chords,
lyrics
RECORDING the specific performance captured
VOICE an artist's vocal likeness, protected
in some jurisdictions independently of
either copyright above
A generated track can implicate all three at once,
which is why "it's AI-generated" settles nothing.
─────────────────────────────────────────
Practical Guidance
─────────────────────────────────────────
- Prefer models trained on licensed catalogues, and
keep the provider's indemnity terms on file
- Never prompt with a living artist's name for
commercial output
- Treat "in the style of [artist]" as a legal risk,
not a prompt technique
- Check whether your provider grants commercial
rights — several do not, by default
─────────────────────────────────────────
Module 6, Chapter 1 covers the underlying copyright question across all modalities.
7. Summary & Next Steps
Key Takeaways
- Music must be coherent at six nested timescales simultaneously, making long-range structure the central difficulty rather than sound quality.
- Residual vector quantisation turns audio into stacked discrete tokens, letting audio generation inherit the entire autoregressive Transformer stack.
- Spectrogram diffusion trades some long-range coherence for parallel speed and easy region editing — the same trade-off seen in Module 1.
- Generation is currently a shortlisting tool for beds, textures and sketches; and a generated track can implicate composition, recording and voice rights at once.
Concept Check
- Why does staged generation — coarse tokens first, finer levels after — mirror the generate-then-upscale pattern from image work?
- Which approach would you pick to edit eight seconds in the middle of an existing loop, and why?
- Why does "it was generated by AI" fail to resolve the licensing question for a music track?
Next Chapter
Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index