Speech: TTS and Transcription

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Module 3, Chapter 5 Time to complete: ~20 minutes

Table of Contents

1. Why Audio Is a Different Problem
2. Speech-to-Text with Whisper
3. Making Transcription Reliable
4. Text-to-Speech
5. Voice Cloning
6. The Consent Line
7. Summary & Next Steps

1. Why Audio Is a Different Problem

The Data Rate Problem
─────────────────────────────────────────
  A 512x512 image           =    786,432 values
  ONE SECOND of audio       =     16,000 values
  ONE MINUTE of audio       =    960,000 values
  Ten minutes of audio      =  9,600,000 values

  Audio is not large per instant. It is large because
  it never stops, and because meaning is spread across
  seconds while the samples arrive thousands per second.
─────────────────────────────────────────

The universal answer is the one from Module 1, Chapter 3: do not work on raw samples. Compress first.

The Two Standard Compressions
─────────────────────────────────────────
  SPECTROGRAM   Convert to a time-frequency image
                (a mel spectrogram) and treat audio as
                a picture. Everything from Module 3
                then applies directly.

  AUDIO TOKENS  Use a learned codec (a VQ-VAE-style
                encoder) to turn audio into a sequence
                of discrete tokens, then model it
                autoregressively like text.
─────────────────────────────────────────

Nearly every modern audio system uses one of these two, which is why so little new theory is required here.

2. Speech-to-Text with Whisper

Whisper is an encoder-decoder Transformer — the architecture from DL Notes, Module 6 — trained on 680,000 hours of weakly-labelled multilingual audio.

The Pipeline
─────────────────────────────────────────
  audio ──► 30-second window ──► mel spectrogram
        ──► ENCODER (a Transformer over the spectrogram)
        ──► DECODER (autoregressive, emits text tokens)

  Notably: the decoder also emits SPECIAL TOKENS for
  language, task (transcribe vs translate), and
  timestamps — so one model does several jobs.
─────────────────────────────────────────

from faster_whisper import WhisperModel
 
model = WhisperModel("large-v3", device="cuda", compute_type="float16")
 
segments, info = model.transcribe(
    "interview.mp3",
    language="en",              # SPECIFY it — auto-detect fails on short or noisy clips
    vad_filter=True,            # voice activity detection: skip silence, avoids hallucination
    beam_size=5,
)
 
print(f"detected: {info.language} (p={info.language_probability:.2f})")
for seg in segments:
    print(f"[{seg.start:6.2f} → {seg.end:6.2f}] {seg.text}")

3. Making Transcription Reliable

Transcription is close to solved for clean English speech and distinctly unsolved elsewhere. Three failure modes account for most problems.

Failure 1 — HALLUCINATION ON SILENCE
─────────────────────────────────────────
  Given silence or noise, Whisper often emits a fluent,
  entirely invented sentence — commonly a phrase from
  its training data ("Thank you for watching!").

  Cause: it is an AUTOREGRESSIVE model (Module 1, Ch.2)
  and must emit something.

  Fix: vad_filter=True. This is not optional for
  real-world audio.

Failure 2 — DOMAIN VOCABULARY
─────────────────────────────────────────
  Proper nouns, product names, medical and technical
  terms are transcribed phonetically and wrongly.

  Fix: pass an initial_prompt containing the expected
  vocabulary. It biases the decoder without any
  retraining:

    initial_prompt="Discussion of Kubernetes, etcd,
                    and the CoreDNS resolver."

Failure 3 — SPEAKER CONFUSION
─────────────────────────────────────────
  Whisper does NOT do diarisation. A two-person
  interview comes back as one undifferentiated
  transcript.

  Fix: a separate diarisation model (pyannote) run
  alongside it, then merge by timestamp.
─────────────────────────────────────────

4. Text-to-Speech

Modern TTS is typically two stages, and knowing the split explains where quality and latency come from.

The Two-Stage Pipeline
─────────────────────────────────────────
  text ──► [1] ACOUSTIC MODEL ──► mel spectrogram
                                       │
        ──► [2] VOCODER ──► audio waveform

  [1] decides WHAT is said and HOW — prosody, rhythm,
      emphasis, pauses. This is where naturalness
      lives.

  [2] turns the spectrogram into actual sound. Modern
      vocoders are fast and near-transparent; this
      stage is rarely the quality bottleneck.
─────────────────────────────────────────

from openai import OpenAI
client = OpenAI()
 
with client.audio.speech.with_streaming_response.create(
    model="tts-1-hd", voice="alloy", input=script, response_format="mp3",
) as response:
    response.stream_to_file("narration.mp3")

Controlling Prosody Without a Prosody Control
─────────────────────────────────────────
  Most APIs expose no direct pitch or emphasis
  parameter. The acoustic model infers prosody from the
  TEXT — so punctuation is your control surface.

  Commas and full stops create pauses of different
  lengths. Paragraph breaks create longer ones.
  Spelling out "Dr." as "Doctor" and "1994" as
  "nineteen ninety-four" removes an entire class of
  mispronunciation.

  Practical consequence: write a SCRIPT, not a
  document. Module 7's capstone does exactly this.
─────────────────────────────────────────

5. Voice Cloning

Given a few seconds to a few minutes of a voice, modern systems can synthesise arbitrary new speech in it.

How Little It Takes
─────────────────────────────────────────
  ZERO-SHOT     3-30 seconds of reference audio.
                A speaker encoder produces a voice
                EMBEDDING, which conditions the
                acoustic model. No training.

                Exactly the IP-Adapter pattern from
                Module 3, Chapter 4 — a reference
                encoded into a conditioning vector.

  FINE-TUNED    10-60 minutes of clean audio. Higher
                fidelity, captures individual speech
                habits. The LoRA-equivalent.
─────────────────────────────────────────

That the zero-shot path needs only seconds of audio — an amount present in any voicemail, video call, or social media clip — is the fact that makes the next section necessary rather than decorative.

6. The Consent Line

This is the one place in this module where a technical chapter has to state a rule plainly.

The Rule
─────────────────────────────────────────
  Synthesising a REAL person's voice requires that
  person's INFORMED, SPECIFIC consent.

  Not the consent of whoever owns the recording.
  Not implied consent because the audio was public.
  The speaker's own consent, for this use.
─────────────────────────────────────────

Legitimate, With Consent
─────────────────────────────────────────
  - Restoring speech for someone losing their voice
  - An audiobook narrated by its own author
  - Localising a presenter into other languages
  - A synthetic brand voice built from a paid,
    contracted voice actor

Not Legitimate
─────────────────────────────────────────
  - Any voice used to impersonate someone to a person
    or a system — including voice-authentication
    bypass, which is now a mainstream fraud vector
  - Putting words in a public figure's mouth, whatever
    the stated intent
  - "It's only a demo" — a demo of a real person's
    cloned voice is the harm, not a rehearsal of it
─────────────────────────────────────────

If You Ship Voice Cloning
─────────────────────────────────────────
  - Require a recorded consent statement from the
    speaker as part of enrolment, spoken in their
    own voice
  - Watermark generated audio (Module 6, Chapter 3)
  - Disclose synthesis at the point of listening
  - Keep an audit log of what was generated, by whom
─────────────────────────────────────────

Module 6, Chapter 2 develops this across all modalities.

7. Summary & Next Steps

Key Takeaways

• Audio is compressed to spectrograms (then treated as images) or to discrete audio tokens (then modelled autoregressively) — the same "never work on raw data" principle as latent diffusion.
• Whisper is an encoder-decoder Transformer whose autoregressive decoder will fluently hallucinate on silence; VAD filtering and an initial_prompt fix the two most common failures.
• TTS splits into an acoustic model (prosody, where naturalness lives) and a vocoder; with no prosody API, punctuation and spelled-out text are your control surface.
• Zero-shot voice cloning needs only seconds of reference audio, which is precisely why informed consent from the speaker — not the recording's owner — is the operative rule.

Concept Check

1. Why does Whisper produce a confident, fluent sentence when given thirty seconds of silence?
2. A client name is consistently transcribed phonetically wrong. What is the cheapest fix, and why does it work without retraining?
3. Which technique from Module 3 is zero-shot voice cloning structurally identical to, and what is the shared mechanism?

Next Chapter

→ Chapter 2: Music & Audio Generation

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index