IHNYC Platform Docs

Now Playing and setlist system

Active8 min read

Table of Contents

  • End-to-end audio identification pipeline that surfaces the song currently playing in the pub on the TVs and at pub.ihnyc-rc.org/setlist
  • Audio is tapped at the Q-SYS Core as a 6-channel AES67 multicast stream, captured on the pub Raspberry Pi (the same Pi that bridges the AV LAN to the internet), fingerprinted against the Shazam catalog, and posted to the existing Cloudflare Worker
  • The Worker stores each track in Cloudflare D1 and fans out to TVs and the /setlist/ page via Server-Sent Events
  • Six discrete audio channels carry per-source information so each track is tagged with where it came from. The setlist labels those inputs HDMI, Wall Stereo, or Wall XLR.

Sources: worker.js (Pub "Now Playing" section banner), pi/pi_capture/, qsys/IH-Pub-1_updated-master-gain-up_20260517.qsys, docs/20260516-211208-pi-karray-song-id-office-hours.md, docs/20260520-102621-pi-aes67-bridge-office-hours.md

End-to-end chain

flowchart LR
  SP["Pub Spotify"] --> CORE
  WALL["Wall Stereo Aux"] --> CORE
  DJ["DJ Wall XLR"] --> CORE
  CORE["Q-SYS Core 110f<br/>(IH-Pub-1, fw 9.13.0)"] -->|AES67-TX-1<br/>233.254.6.0:5004<br/>L24/48000/6ch ptime=1ms| PI

  subgraph "Raspberry Pi (private AV network)"
    PI["aes67-linux-daemon<br/>+ RAVENNA kernel module"] --> ALSA["plughw:CARD=RAVENNA<br/>6-channel ALSA capture"]
    ALSA --> CAP["pi-capture (Python)<br/>RMS source-picker"]
    CAP --> ID["shazamio<br/>song-ID"]
  end

  ID -->|HTTPS POST<br/>Bearer token| WORKER["Cloudflare Worker<br/>/api/now-playing/ingest"]
  WORKER --> D1[("Cloudflare D1<br/>SETLIST_DB")]
  WORKER -->|SSE| TV["TV 'Now Playing' footer<br/>/tv/, /bar-tv-left/, /bar-tv-right/"]
  WORKER -->|SSE| SETLIST["/setlist/ page<br/>(public)"]

Sources: docs/20260516-211208-pi-karray-song-id-office-hours.md (“Q-SYS Per-Source Tap Architecture”), worker.js, pi/pi_capture/capture.py

Audio sources and channel mapping

Six channels are wired in Q-SYS Designer to a single AES67 Transmitter (AES67-TX-1). Three stereo pairs, each carrying one source:

AES67 ChannelSourceQ-SYS upstreamTag on /setlist/
1Program LMic/Line In (Pub HDMI program feed)program → “HDMI”
2Program RMic/Line In (Pub HDMI program feed)program → “HDMI”
3Wall Plate Stereo LSoftware Dante RXwallstereo → “Wall Stereo”
4Wall Plate Stereo RSoftware Dante RXwallstereo → “Wall Stereo”
5Wall Plate XLR 1Software Dante RXwallxlr → “Wall XLR”
6Wall Plate XLR 2Software Dante RXwallxlr → “Wall XLR”

The Wall Plate XLR jacks reach the Core over Dante, not the Core’s hardware Mic/Line inputs. The mapping above is what’s actually wired in the production design; if any source ever moves, the constants in pi/pi_capture/config.py (AES67_PAIR_LABELS) must move with it.

Sources: docs/20260516-211208-pi-karray-song-id-office-hours.md (channel-order section), pi/pi_capture/config.py

The source picker

Multiple sources can carry signal at the same time (e.g., a tender starts playing through the wall stereo while pub Spotify is still running through the program bus). The Pi-side aes67_extract (in pi/pi_capture/capture.py) picks the loudest non-silent stereo pair every tick using RMS energy:

# pi/pi_capture/capture.py — aes67_extract()
pair_levels = []
for i in range(3):
    pair = block[:, i * 2 : i * 2 + 2]
    rms = float(np.sqrt(np.mean(np.square(pair.astype(np.float32)))))
    pair_levels.append((rms, i))
pair_levels.sort(reverse=True)
best_rms, best_idx = pair_levels[0]
# best_idx in {0,1,2} -> program / wallstereo / wallxlr

The selected pair is summed to mono, fingerprinted, and posted with source = AES67_PAIR_LABELS[best_idx]. The Worker then stores source alongside the track and the /setlist/ page renders the matching chip via the SOURCE_LABELS map in public/setlist/index.html.

Edge case: if all three pairs are below the silence floor (SILENCE_DBFS, default -60), aes67_extract returns (None, None) and the main loop posts a silence sentinel instead of a track.

Sources: pi/pi_capture/capture.py, pi/pi_capture/config.py, public/setlist/index.html (SOURCE_LABELS)

Song-ID backend

The Pi runs shazamio (community Shazam-API client) in production. The choice is pragmatic and intentionally swappable:

BackendStatusCatalogCostNotes
shazamioproductionFull ShazamFreeToS-gray, no SLA. Fine for dev + a small community pub.
acoustidavailable~75-85% mainstreamFreeLinux-native via Chromaprint. Catalog gaps on top-40 pop.
auddavailableShazam-class~$5/moLegitimate paid API, Linux-native.
shazamkitstub onlyFull ShazamFreeApple platform only. Requires a Mac-side Swift HTTP service that the Pi posts WAV clips to. Documented in docs/20260516-211208-pi-karray-song-id-office-hours.md.

The backend is selected via SONGID_BACKEND in /opt/pi-capture/.env. The capture pipeline is identical regardless of backend; only the identifier and confidence reporting differ. See environment-variables.

Sources: pi/pi_capture/songid.py, pi/config.example.env

Worker API surface

Six endpoints exposed by worker.js for this subsystem:

EndpointMethodPurpose
/api/now-playing/ingestPOSTThe Pi posts every identified track here. Requires Authorization: Bearer <NOW_PLAYING_INGEST_TOKEN>. Returns the D1 row id on success.
/api/now-playing/currentGETServer-Sent Events stream. TVs and /setlist/ receive an initial snapshot, each new row, and a null track when playback becomes silent or stale. A deduped ingest keeps the existing row ID and does not emit another track event.
/api/now-playing/controlsWebSocketPushes the current right-TV display and per-surface lyrics settings, then sends each change.
/api/now-playing/setlistGETCursor-paginated JSON history. Bounded scopes backfill automatically; All time fetches when the scroll sentinel enters view.
/api/now-playing/snapshotGETOne-shot current-track JSON for simple pollers such as the Pi DMX controller. Silence, stale data, and video return track: null.
/api/now-playing/offsetPOSTStores the Pi’s current lyrics synchronization offset for 60 seconds. Uses the ingest bearer token.

Auth model:

  • If NOW_PLAYING_INGEST_TOKEN is set as a Wrangler secret, the Worker requires the matching Authorization: Bearer … header on /ingest and /offset. Mismatches are rejected 401.
  • If the secret is unset (for example, in local wrangler dev), those POSTs are accepted and the Worker logs a warning. Production must have the token set.
  • The read and stream endpoints are public. /controls requires a WebSocket upgrade and the TV_PAIRING Durable Object binding.

Sources: worker.js (handleNowPlayingIngest, handleNowPlayingCurrent, handleNowPlayingControlsSocket, handleNowPlayingSetlist, handleNowPlayingSnapshot, handleNowPlayingOffset)

Storage model

TableRoleImportant fields
setlistIdentified tracks and silence sentinelsstarted_at, track metadata, source, confidence, silent, content_kind, video_source, screening_id
heartbeatsLatest Pi and attached-device statelast_seen, deployed version, AVPro, projector, touchscreen, and content-classifier JSON
heartbeat_historyFive-minute liveness bucketsbucket
setlist_rollupsPrecomputed setlist surfacessurface_id, partition_key, payload, coverage, partial state, and compute timestamps
surface_configCompute and publication settingsenabled state, tier, cadence, JSON configuration, confidence floor, and recompute marker
track_enrichmentCached genre metadatanormalized track identity and provider result
pub_screeningsDetected or operator-created movie windowstitle, album key, start/end state, source, confidence, poster, track count, and shadow state

The API filters silence sentinels and non-music content out of public history and music analytics where required. Migrations 0001 through 0046 build this model; later migrations add unrelated Pub administration state.

Sources: migrations/0001_create_setlist.sql, 0002_add_setlist_silent.sql, 0003_create_heartbeats.sql, 0010_setlist_analytics.sql, 0012_track_enrichment.sql, 0016_heartbeat_history.sql, 0023_setlist_content_kind.sql, 0024_heartbeat_version.sql, 0027_heartbeat_devices.sql, 0029_heartbeat_tsc.sql, 0043_heartbeat_content_status.sql, 0046_pub_screenings.sql

Silence and live-music handling

The pipeline distinguishes three states:

  1. Track playing — A stable fingerprint above SILENCE_DBFS. Posts {artist, title, source, isrc, …} and the Worker emits a track SSE event.
  2. Silence — All three pairs below SILENCE_DBFS for SILENCE_SECONDS consecutive seconds (default 30). The Pi posts a silence sentinel; the Worker emits a track event with a null payload; the TV footer hides; and the /setlist/ Now Playing card disappears.
  3. Live music / no match — Signal is present but the song-ID backend returns no match, as with a live performance or uncatalogued track. The pipeline logs the miss and does not create a setlist row.

Sources: pi/pi_capture/main.py (main loop), pi/pi_capture/poster.py (silence sentinel), worker.js (SSE fanout), public/setlist/index.html (Now Playing card hide-on-silence)

Frontend surfaces

SurfacePathWhat it shows
TV footer/tv/, /bar-tv-left/, /bar-tv-right/Small “Now Playing” line; hides on silence. Subscribes to SSE.
Now Playing hero/setlist/ (top card)Album art, title, artist, and a Save action sheet with four music-service searches, Web Share, and copy. Hides on silence.
Setlist history/setlist/ (rows)Reverse-chronological tracks grouped by local day, with HDMI / Wall Stereo / Wall XLR labels and a streaming/share action sheet.
Public landing CTA/”Live now” button to /setlist/ while a track is playing.

All four surfaces are driven by the same SSE event stream, so they stay synchronized to within the SSE delivery window (typically <1 sec).

Sources: public/tv/script.js (search Pub "Now Playing" — SSE subscriber), public/setlist/index.html, public/index.html

Why it works this way

Why tap at the Q-SYS Core, not the Kommander amplifier?

The K-array Kommander is downstream of the Core; tapping there would only see the post-mix. Tapping at the Core preserves per-source channels, which is the whole reason the source chip on /setlist/ works.

Why a single 6-channel AES67 stream and not three separate streams?

“Variant B” — one transmitter, one multicast group, one ALSA device, one RMS-based source picker — keeps Q-SYS Designer simple and the Pi-side code small. “Variant A” (three separate transmitters) is documented as the future migration path if per-source routing or VLAN segmentation ever becomes a requirement. Migration is ~15 minutes in Designer and ~30 lines on the Pi; downstream (Worker, TV UI, /setlist/ page) doesn’t change.

Why aes67-linux-daemon + RAVENNA kernel module on the Pi instead of a simpler GStreamer pipeline?

aes67-linux-daemon honors the full AES67 receive timing profile via PTP and is what the Pi-side capture code expects as an ALSA device. A GStreamer rtpsrc + snd-aloop pipeline works for one night but drifts over hours without proper PTP-locked clock. The daemon was harder to set up but is the right shape for ship-and-leave.

Why does PTP look noisy in journalctl -u ptp4l-eth0?

The Q-SYS Core in this installation acts as a PTP Boundary Clock and advertises the Dante endpoint’s clock identity (00-1d-c1-ff-fe-9c-a9-74) as the AES67 grandmaster. That’s an ARB (Arbitrary) timescale, not UTC. ptp4l is configured with free_running 1 so it observes the master without trying to slew the Pi’s system clock to the ARB epoch. The “temporal vortex” warning in the journal is expected for this setup and harmless; the RAVENNA kernel module reads PTP state independently and clocks audio reception correctly.

See ptp-troubleshooting for the longer story.

Sources: docs/20260516-211208-pi-karray-song-id-office-hours.md (recommended approach + variants), docs/20260520-102621-pi-aes67-bridge-office-hours.md (PTP design)

Known operational issues

Q-SYS Core / Dante PTP grandmaster flap

The May-17 AES67 deploy enabled the Core’s PTPv2 master, which now contests grandmastership with an existing Dante endpoint on the private AV network via BMCA. Every flap (~every few hours) triggers an “Audio Clock Set Events, Processing overrun” hardware fault on the Core, audible as a brief click on the program bus.

Fix is in Q-SYS Designer, not on the Pi:

  • Set the Core’s PTP Priority1 below the Dante endpoint’s default (e.g., 64 vs 248) so the Core wins BMCA consistently
  • Or, move the Core’s AES67 PTP to a separate domain so it doesn’t contest

This is tracked as a follow-up Designer session. The Pi-side capture is unaffected because ptp4l ignores PTPv1 traffic from the Dante endpoint regardless of which device is master.

Sources: Q-SYS Core event log (2026-05-20), docs/20260520-102621-pi-aes67-bridge-office-hours.md