IHNYC Platform Docs

Content detection (movie, TV, or song)

Development9 min read

Table of Contents

The classifier runs in shadow mode: it logs and forwards a music or video verdict but does not change any public display. Activation waits for a labeled movie/TV soak and supervised refit. See Current status.

It runs alongside the Now Playing and Setlist system on the Pub Raspberry Pi. Signals already computed by that pipeline, plus an optional YAMNet sound-event read, produce one rolling session verdict and confidence. The intended result is to keep film and broadcast audio out of music charts and provide a gate for Now Showing and screening detection.

Sources: pi/pi_capture/content_session.py, pi/pi_capture/yamnet.py, pi/pi_capture/main.py, docs/movie-tv-detection-runbook.md, migrations/0043_heartbeat_content_status.sql

Why a session-level classifier

Per-track signals are too noisy to tell a song from a movie. A movie plays real licensed songs (a “needle-drop”); a song can sound like a score; a sports broadcast has long stretches with no identifiable music at all. Looking at any single Shazam attempt cannot separate these.

So the classifier works at the session level. It keeps a rolling window of recent cycles (default 90 s) and fuses their aggregate evidence into one verdict. Session-level is what catches the hard cases: a licensed song dropped into a dialogue-heavy film still reads video, because the surrounding minutes of the session read video even though that one track matched.

The code (pi/pi_capture/content_session.py) is deliberately a small, hand-auditable logistic — not a black box — so its weights can be read, reasoned about, and refit on real pub audio. It is pure and clock-injectable (no I/O), so it is unit-testable on synthetic evidence streams.

The input signals

Every cycle, ContentSession.observe(evidence) records one dict of evidence. Each field is already computed by the existing capture pipeline — content-detect adds almost no new work:

SignalSourceWhat it tells us
speech_probtap-select classifier (1 - music_probability)how speech-heavy the relevant pair is
matchedsong-ID attempt resultNone = no attempt this cycle (same-song lock); True = matched; False = attempted, no match
is_ostcontent.classify_albumthe match was a soundtrack / OST album
is_real_musicalbum classifierthe match was a real, non-OST music album (the concert-film carve-out)
rmslevel meterloudness of the relevant pair (dynamic-range signal)
yamnetYamnetWorker.latest(){p_music, p_speech, p_screen}, or None when YAMNet is off

A subtlety worth keeping (it caused a real misfire): the match feature is computed over attempts, not over all cycles. The same-song lock suppresses Shazam calls while a known song keeps playing, so “no attempt this cycle” must not count as “no match” — otherwise a song that is correctly identified once and then locked would slowly read as video. When there are zero attempts at all, a mild non-decisive video lean (EMPTY_SHAZAM_LOW_MATCH = 0.4, below the neutral point) is used so that absence of matches alone never crosses the threshold without speech or YAMNet support.

YAMNet — the screen-content signal

Pure DSP can tell you “this is speech-heavy,” but it can’t tell you “this is a broadcast — a television, a laughing crowd, a cheering stadium.” YAMNet (Apache-2.0, trained on AudioSet, 521 sound-event classes) can, because it has explicit Television, Laughter, Applause, Crowd, Cheering, Speech, Music, and Singing classes.

pi/pi_capture/yamnet.py collapses those 521 scores into three fusion-ready numbers per frame — each the max over a group of related classes (“any of these fired”):

OutputAudioSet classes (grouped)Read as
p_music / musMusic, Singing, instruments, pop/rock/hip-hop/electronic/jazz/classicalpulls toward music
p_speech / spkSpeech, Conversation, Narration, monologuedialogue
p_screen / scrTelevision, Radio, Laughter, Applause, Crowd, Cheering, Chatter, Audience, Sound effectthe direct “broadcast / room-of-people” tell — strongest video signal

Resilience and timing posture

  • Additive, never a hard dependency. If tflite-runtime or the model file is missing (e.g. on a dev/CI machine), YamnetClassifier.available stays False, latest() returns None, and the classifier simply uses its without-YAMNet weight set. Nothing breaks.
  • Runs on its own background thread. Inference is too slow to run inline in the capture loop without risking the lyrics-offset / song-ID timing. YamnetWorker scores a recent audio snapshot every YAMNET_CADENCE_SECONDS (default 3 s) on a daemon thread and writes its latest scores into a lock-guarded slot; content_session reads slightly-stale scores, which is fine over a multi-second session window. It never touches the song-ID path, so the main loop’s single-thread invariant holds.
  • Cheap. On the pub Pi the CPU spike test passed at ~1.3% duty — YAMNet on never threatens the song-ID / offset loop. (It is gated behind that spike precisely so it can’t.)

How the verdict is fused

features() aggregates the window, then _raw_p_video() runs a logistic regression with two weight sets — with_yamnet and without_yamnet, each with its own bias. When YAMNet is unavailable the classifier switches sets rather than dropping a term (which would skew the bias). The output is P(video); at or above the threshold (default 0.6, deliberately music-biased) the cycle is a “video cycle.”

WEIGHTS_WITH_YAMNET = {              WEIGHTS_WITHOUT_YAMNET = {
    "p_screen": 3.0,    # video          "speech_fraction": 1.5,
    "p_speech": 1.0,                      "low_match": 1.2,
    "p_music": -2.0,    # → music         "ost_hit_rate": 0.8,
    "speech_fraction": 0.8,               "loudness_var_norm": 0.6,
    "low_match": 0.8,                      "bias": -1.6,
    "ost_hit_rate": 0.5,             }
    "loudness_var_norm": 0.3,
    "bias": -2.0,
}

These weights are conservative, music-biased placeholders. They have not been fit on real pub audio. The current default with_yamnet set still mis-verdicts a clear TV show as music even when the signals separate cleanly. Refitting them is the open work — see below.

Two stabilizers sit on top of the raw logistic:

  • Hysteresis (hold). The held verdict only flips after CONTENT_HOLD consecutive cycles (default 3) on the same side, so a single stray cycle can’t toggle the displayed kind.
  • Concert-film carve-out. If the session keeps matching real (non-OST) music at decent density (≥4 attempts, ≥60% real-music, ≤20% OST), that overrides a high speech fraction toward music. A concert film or music doc keeps matching songs; an ordinary movie does not.

The window resets (reset()) on silence or a routing change back to a music input.

Current status: SHADOW + untuned

The classifier is at the shadow stage of its lifecycle:

[A] SHADOW (now) → [B] CAPTURE+LABEL → [C] REFIT → [D] APPLY weights → [E] GO ACTIVE

In shadow mode it logs what it would decide and changes nothing. A healthy shadow line (one every CONTENT_LOG_SECONDS, default 30 s) looks like:

content-detect shadow: session=music conf=0.74 p_video=0.26 set=with_yamnet carve=False |
  speech=0.06 low_match=0.40 ost=0.00 loud=0.00 scr=0.00 spk=0.65 mus=0.15 attempts=2 cycles=30

What the music soak showed

YAMNet is on at the pub (set=with_yamnet in the live log), and a ~5-hour live-music soak was captured 2026-06-24/25 and staged in the pub repo at .context/content-detect-soaks/. Over 266 confirmed-music windows the model produced zero false positives: p_video averaged 0.051 and never exceeded 0.16, comfortably under the music-biased DEFAULT_THRESHOLD = 0.6. So music is not the gap — the model already keeps a real music night out of video. The open risk is the other direction: a TV show or movie mis-read as music.

That means the refit can’t run yet. refit_content_session.py requires ≥20 labeled windows of both classes, and only the music half is banked — so the next real step is a movie/TV soak to supply the video side before [C].

Then the one-off supervised refit, not a recurring job:

  1. Soak a real contrast — the music night is banked; the missing half is a TV show / movie / game — with YAMNet on.
  2. Capture + label the shadow log (journalctl -u pi-capture | grep content-detect) against a small start,end,label CSV of what was actually on screen.
  3. Refit with python -m tools.refit_content_session --log shadow.log --labels labels.csv, which prints accuracy / recall / ROC-AUC and drop-in WEIGHTS_* dicts + a suggested threshold.
  4. Apply those dicts to content_session.py, run pytest tests/test_content_session.py, PR, merge → the Pi auto-deploys on its next ~5-min tick (still shadow, still safe).
  5. Only once the verdicts are trusted: flip CONTENT_DETECT_MODE=active (a Phase-2 code change — content_session becomes the single writer of content_kind and the old per-track miss-counter path is retired in the same PR). Until then, active deliberately behaves like shadow.

The full operator procedure lives in the repo runbook: docs/movie-tv-detection-runbook.md.

How the verdict reaches the worker

The Pi forwards a compact snapshot of the live verdict in its heartbeat (not in the now-playing ingest). Heartbeats fire every ~30 s regardless of whether a song is identified — which is exactly the “video on, no song” case (a broadcast with no matchable track). Now-playing posts are song-change / silence driven and would miss it.

__main__.py assembles the snapshot inside its own try/except so a snapshot failure can never break the core beat (and float()-casts the numpy values so they stay JSON-safe):

{ "kind": "music", "conf": 0.74, "p_video": 0.26, "set": "with_yamnet",
  "carve": false, "mode": "shadow",
  "scr": 0.0, "spk": 0.65, "mus": 0.15, "low_match": 0.40 }

Worker-side, this lands in the content_status TEXT column on heartbeats (migration 0043_heartbeat_content_status.sql, SETLIST_DB). Mirroring the tsc_status / qsys pattern, it is written by a separate tolerant UPDATE in the heartbeat handler, so a pre-0043 database never fails the core beat. The column is nullable → no behavior change until the Pi reports it.

This is currently a pipe only. Nothing consumes content_status for display yet — it is the ingest contract for the audio-confirmed sports / Now Showing fusion (a later PR, gated on the classifier going active), plus a way for the admin to watch the live verdict without SSHing into the Pi for journalctl.

.env configuration knobs

Set on the Pi at /opt/pi-capture/.env (full descriptions in pi/config.example.env):

VarDefaultMeaning
CONTENT_DETECT_MODEoffoff / shadow (log only) / active (Phase 2; currently behaves like shadow)
CONTENT_VIDEO_PAIR-1AES67 pair carrying screen audio; -1 = pick the loudest pair. At the pub this is 0 (HDMI = ch 1&2).
CONTENT_WINDOW_SECONDS90rolling session window
CONTENT_HOLD3sustained cycles before the held verdict flips
CONTENT_LOG_SECONDS30shadow summary cadence
YAMNET_ENABLEDfalseturn on the YAMNet signal (needs the CPU spike + the model file) — on at the pub
YAMNET_MODEL_PATH/opt/pi-capture/models/yamnet.tflitemodel location
YAMNET_CADENCE_SECONDS3.0how often YAMNet scores (raise to lower CPU)

To turn the whole thing off: set CONTENT_DETECT_MODE=off (or remove it) and sudo systemctl restart pi-capture. The classifier goes inert; nothing else changes.

Related: setlist content_kind tagging

The verdict here is one of two ways audio gets tagged music vs video in setlist:

  • This classifier (once active) will become the live, audio-driven writer of content_kind for ambient/broadcast content.
  • The Screenings state machine re-tags a confirmed movie’s whole time window: rows still content_kind='music' in that window flip to content_kind='video' (stamped with a stable screening_id), so a film’s soundtrack leaves the music charts.
  • Worker-side ingest also tags obvious cases server-side — ambient for sleep / white-noise / “N hours of…” background audio, and video for movie-track markers — via the AMBIENT_TRACK_MARKERS / movie-track classifiers.

Every setlist analytics surface (Top Songs, The Vibe, Persona, the /setlist feed) filters on content_kind='music', so all three paths share the same goal: keep non-music audio out of the music charts.