Generative AI

Ethics Law And Trust

Deepfakes, Consent & Misuse

Manipulated media is old. What changed is not possibility but cost.

JrCodex·7 min read

Jr Codex Generative AI Notes

Level: Intermediate Prerequisites: Chapter 1: Copyright and Training Data Time to complete: ~20 minutes

Table of Contents

  1. What Changed
  2. The Consent Standard
  3. The Four Harm Patterns
  4. Design Responses
  5. The Liar's Dividend
  6. Where the Line Sits
  7. Summary & Next Steps

1. What Changed

Manipulated media is old. What changed is not possibility but cost.

The Cost Collapse
─────────────────────────────────────────
                       Then              Now
  ─────────────────────────────────────────
  Convincing fake      a studio, weeks   minutes
  video                and specialists

  Cloned voice         hours of clean    seconds of
                       audio, experts    any audio

  Fake photograph      a skilled         one prompt
                       retoucher

  Scale                one at a time     thousands,
                                         personalised
─────────────────────────────────────────

Why Cost Is the Whole Story
─────────────────────────────────────────
  When faking someone costs a week and a studio, it
  happens to public figures, occasionally, for high
  stakes.

  When it costs ninety seconds, it happens to ordinary
  people, routinely, for low stakes — a colleague, an
  ex-partner, a classmate, a grandparent receiving a
  phone call.

  The technology did not create a new WRONG. It
  removed the friction that kept the wrong rare.
─────────────────────────────────────────

Module 4, Chapter 1 stated this for voice. It generalises to every likeness — face, body, voice, signature style.

The Standard
─────────────────────────────────────────
  Synthesising a REAL, IDENTIFIABLE person requires
  that person's INFORMED, SPECIFIC, REVOCABLE consent.

  INFORMED    they understand what will be generated
              and where it will appear
  SPECIFIC    for this use, not a blanket permission
              buried in terms of service
  REVOCABLE   they can withdraw it later, and you can
              actually honour that
─────────────────────────────────────────

Three Things That Are NOT Consent
─────────────────────────────────────────
  "The photo was public."
     Public availability is permission to LOOK, not
     permission to SYNTHESISE.

  "We own the footage."
     Owning a recording is a copyright fact. A
     person's likeness is a separate right that the
     recording's owner does not hold.

  "They agreed to the ToS."
     Buried blanket permissions fail the SPECIFIC
     test, and increasingly fail legally too.
─────────────────────────────────────────

3. The Four Harm Patterns

Being concrete about harm is more useful than a general caution, because each pattern has a different defence.

1. NON-CONSENSUAL INTIMATE IMAGERY
─────────────────────────────────────────
  The largest category of deepfake abuse by volume, and
  overwhelmingly targeted at women.
  Illegal in a growing number of jurisdictions.
  Defence: model-level and prompt-level refusal;
  never a user-configurable setting.

2. FRAUD AND SOCIAL ENGINEERING
─────────────────────────────────────────
  Cloned voice on a phone call authorising a transfer.
  Video of an executive in a meeting instructing a
  payment. Voice-authentication bypass.
  Defence: this is a SYSTEMS problem — out-of-band
  verification and callback procedures, not detection.
  Assume any voice or face can be faked.

3. DISINFORMATION
─────────────────────────────────────────
  Fabricated statements or events attributed to real
  people, at scale, timed to elections or crises.
  Defence: provenance infrastructure (Chapter 3) and
  platform policy — detection alone loses this race.

4. HARASSMENT
─────────────────────────────────────────
  Targeted synthetic content about a private
  individual. Low-profile, high-harm, and the hardest
  to catch because the victim is not famous enough
  for anyone to be checking.
  Defence: fast, credible reporting and takedown.
─────────────────────────────────────────

4. Design Responses

If you build anything that generates people, these are the mechanisms available. None is sufficient alone.

At ENROLMENT — for likeness or voice products
─────────────────────────────────────────
  - Require a recorded consent statement, spoken or
    signed by the subject themselves
  - Verify liveness so the enrolment cannot be done
    from a stolen photo or clip
  - Bind the consent record to the generated asset,
    and make revocation actually delete it

At GENERATION
─────────────────────────────────────────
  - Refuse recognisable public figures by default
  - Refuse the intimate-imagery category outright, at
    the model and the prompt layer
  - Rate-limit and log per account — abuse at scale
    looks different from ordinary use

At OUTPUT
─────────────────────────────────────────
  - Watermark and attach C2PA provenance (Chapter 3)
  - Disclose synthesis visibly where a viewer could
    otherwise be deceived
  - Retain enough audit trail to answer "who generated
    this, and when?"

At the PRODUCT level
─────────────────────────────────────────
  - A reporting path that a NON-USER can use — the
    victim of synthetic content is usually not your
    customer and cannot log in to complain
  - A takedown process with a stated response time
  - A published policy, so users know the rules before
    they test them
─────────────────────────────────────────

The non-user reporting path is the one most often missed and the one that matters most. A person harmed by content your product generated has no account, no support ticket, and no way in.

5. The Liar's Dividend

A second-order harm that is easy to miss and arguably larger than the first-order one.

The Mechanism
─────────────────────────────────────────
  Once everyone knows that convincing fakes exist,
  GENUINE evidence can be dismissed as fake.

  A real recording of real wrongdoing becomes
  deniable. The doubt is free, and it does not have
  to be justified — only raised.

  So the existence of synthesis damages trust in
  authentic media, even where no fake was ever made.
─────────────────────────────────────────

Why This Shapes the Technical Response
─────────────────────────────────────────
  You cannot solve the liar's dividend with DETECTION.
  Detection tells you "this might be fake" — it can
  never establish that something is REAL.

  Only PROVENANCE can do that: a verifiable record of
  where a piece of media came from, attached at
  capture.

  This is the single strongest argument for the
  infrastructure in Chapter 3, and the reason the
  industry has converged on signing authentic content
  rather than on catching fake content.
─────────────────────────────────────────

6. Where the Line Sits

Legitimate, With Consent
─────────────────────────────────────────
  - An actor licensing their likeness for dubbing
  - Restoring speech for someone who has lost it
  - A presenter localised into other languages
  - Historical or educational recreation, clearly
    labelled as such
  - Satire and parody where no viewer could be
    deceived, and local law protects it

Not Legitimate, With or Without Consent
─────────────────────────────────────────
  - Intimate imagery of a real person
  - Impersonation to obtain money, access or
    credentials
  - Content attributing false statements to a real
    person as though genuine
  - Synthetic media of children

The Test to Apply
─────────────────────────────────────────
  "Would the person depicted agree to THIS use, and
   would a viewer be deceived about whether it is
   real?"

  If you cannot confidently answer the first, you do
  not have consent.
  If the answer to the second is yes, you need
  disclosure — or you should not ship it.
─────────────────────────────────────────

7. Summary & Next Steps

Key Takeaways

  • Synthesis did not create a new wrong; it removed the cost that kept the wrong rare, moving the target from public figures to ordinary people.
  • Consent must be informed, specific and revocable — public availability, owning the footage, and terms-of-service acceptance are none of those things.
  • The four harm patterns have different defences: refusal, out-of-band verification, provenance, and fast takedown; a non-user reporting path is essential and usually missing.
  • The liar's dividend cannot be answered by detection, because detection can never prove something is real — only provenance can.

Concept Check

  1. Why is "we own the footage" not a valid basis for synthesising the person in it?
  2. Which harm pattern is best defended by a process change rather than by any technical measure, and why?
  3. Explain why the liar's dividend pushes the industry toward signing authentic media rather than detecting synthetic media.

Next Chapter

→ Chapter 3: Provenance and Watermarking

Jr Codex — 1-on-1 Personalized Coaching | Back to Module Index | Back to Generative AI Index