IHNYC Platform Docs

Pub operations

Active6 min read

Table of Contents

Release model

  • Pull requests must pass two independent gates: GitHub Actions runs npm ci + npm test, while Cloudflare Workers Builds uploads a non-production preview version
  • Production auto-deploys from main via Cloudflare Workers Builds; a manual wrangler deploy is a break-glass path, not the normal release workflow
  • No build step; worker and static assets are deployed as-is
  • Custom domain: pub.ihnyc-rc.org
  • Admin UI at /admin/* behind Cloudflare Access (Zero Trust)

Prerequisites

  • Node.js v18+
  • Wrangler CLI (npm install -g wrangler && wrangler login)
  • Access to the Cloudflare account that owns the worker

Secrets and variables

Required secrets

Set once per environment via wrangler secret put <NAME>:

SecretPurpose
PUB_ADMIN_TOKENAdmin API auth token (also gates Short.io QR sync). Renamed from TENDER_QR_SYNC_TOKEN; the worker reads PUB_ADMIN_TOKEN.
NOW_PLAYING_INGEST_TOKENBearer token for Pi song-ID pipeline
CF_ACCESS_POLICY_API_TOKENUsed by the Access page to update the dedicated delegated-operator policy

Optional secrets

SecretPurpose
NOTION_API_KEYNotion integration token (events source, still used for events reads)
SHORT_IO_API_KEYShort.io API for tender QR code short links
Provider and relay credentialsOptional integrations for Notion, Short.io, email, Q-SYS, lighting, and TV control

Non-secret vars in wrangler.toml

VarPurpose
TV_BUILD_IDBump on every user-facing change; triggers TV self-reload
PUBLIC_SITE_URLhttps://pub.ihnyc-rc.org
CF_ACCESS_TEAM_DOMAIN, CF_ACCESS_AUDJWT issuer and audience used for every delegated Pub operator
Access policy identifiersDedicated delegated-operator policy bridge; public identifiers, never the API token
NOTION_DATABASE_IDEvents DB
NOTION_ANNOUNCEMENTS_DATABASE_IDAnnouncements DB (legacy, being migrated to D1)
NOTION_PUB_TENDERS_DATABASE_IDPub Tenders DB (legacy, tenders now in D1)
SHORT_IO_DOMAINDomain for Short.io QR links
SETLIST_LEAD_SURFACE_ENABLEDFeature flag for Top Songs surface
SETLIST_VIBE_COMPASS_ENABLEDFeature flag for Vibe Compass
SETLIST_PERSONA_ENABLEDFeature flag for Persona AI surface
SETLIST_STORIES_ENABLEDFeature flag for Stories AI surface
SPORTS_ENABLEDLive sports schedule (keyless ESPN) → /api/sports/live + cron refresh of the sports KV cache
NOW_SHOWING_ENABLED”Now Showing” fusion (/api/now-showing) — fuses the Pi content verdict with the ESPN schedule to name a live game
SCREENINGS_ENABLEDMovie-night auto “Now Showing” + /screen archive. Values: off | shadow | on
FEEDBACK_ENABLEDPatron feedback / suggestion box (/feedback + /api/feedback); admin read at /admin/feedback is always on
PUB_SCHEDULER_V2Pub-scheduler run state-machine engine. Dormant — unset until the v2 cutover; flipping it activates the new transition engine

Bindings in wrangler.toml

BindingTypePurpose
CONTENT_DBD1Pub scheduler data (tenders, shifts, assignments, submissions, audit)
SETLIST_DBD1Song-ID data, setlist analytics, theme presets, API health
IMAGE_MIRROR_BUCKETR2Image mirror for TV/announcement/tender images
EVENT_MEDIA_BUCKETR2Event slideshow media
TUTORIALS_DATAKVAdmin-edited tutorial content blobs
TV_PAIRINGDurable ObjectTvPairingCoordinator for slot state
DJ_MODEDurable ObjectDJ Mode queue state
AIWorkers AIPersona/Stories AI inference

D1 migrations

Migrations are plain SQL files applied manually. Two databases:

CONTENT_DB (pub scheduler)

wrangler d1 execute CONTENT_DB --remote --file=./migrations/content/<file>.sql

Latest numbered migration: 0053_pub_supervisors_fairness_report.sql. Recent additions include email-log attachments (0052) and the per-supervisor fairness-report setting (0053).

SETLIST_DB (song-ID, analytics, theme, API health)

wrangler d1 execute SETLIST_DB --remote --file=./migrations/<file>.sql

Latest: 0061_pub_qsys_snapshots.sql. Migrations 0053–0061 add quiet-hours state, TV control, scoped Pub roles, delegated Access sync, Q-SYS control discovery and curation, Spotify Jam scheduling, DMX command results, and audio snapshots. Two different 0058 files exist; use the repository’s recorded migration order rather than assuming the number is unique.

Always apply migrations before deploying code that depends on new tables/columns. Check CLAUDE.md in the repo for the full ordered migration list.

Deploy

Production deploys automatically when a change lands on main via Cloudflare Workers Builds. Before merge, the GitHub Actions test check and Cloudflare preview build must both pass. GitHub Actions is test-only and has no deploy credentials; Cloudflare owns preview uploads and the production deploy.

Because main can go live immediately after merge, required D1 migrations must either be applied before merging or remain backward-compatible with both the old and new Worker. The canonical release, verification, and rollback procedure lives in the Pub repository’s docs/DEPLOYMENT.md.

Bump TV_BUILD_ID in wrangler.toml (in the same change) if TV-facing assets changed; TVs self-reload within 60 seconds when it changes.

A manual wrangler deploy is reserved for break-glass/one-off situations and must be recorded in the PR or incident — merging a green PR to main is the normal path.

Local development

# Create .dev.vars with local secrets
echo "PUB_ADMIN_TOKEN=<token>" > .dev.vars
 
# Start local dev server
wrangler dev

Local dev serves at http://localhost:8787. Workers AI (env.AI) is not available in local mode; use wrangler dev --remote for AI surface testing (bills Neurons).

Monitoring

SignalWhere to look
Worker errorsCloudflare Dashboard > Workers & Pages > Logs
TV health/admin/tv-health — live heartbeat, build match, memory, JS errors per screen
API dependencies/admin/api-health — Notion, LRCLib, Short.io, Deezer, Formbricks, Workers AI, ESPN (keyless live-sports scoreboard), TMDB (movie posters, keyless Deezer fallback)
Scheduler state/admin/pub-schedule — current week’s wizard step and submission feed
Email delivery/admin/pub-setup > Email Log
Song-ID pipelinessh pi@<pi-tailnet-hostname> 'systemctl is-active ptp4l-eth0 aes67-daemon pi-capture'
Setlist rollupsCheck /api/setlist/<surface> for fresh computedAt timestamps

TV build version

TV screens self-reload when TV_BUILD_ID changes. Bump it on every deploy that touches public/tv/, public/bar-tv-right/, or any TV-facing asset:

TV_BUILD_ID = "YYYY-MM-DD-NN"   # e.g. 2026-06-07-01

The worker injects this into TV HTML as window.__TV_BUILD_ID__. TVs poll /api/tv-build every 60 seconds and reload when the value changes.

Cloudflare Access setup

The admin UI is gated by a Pub-specific Cloudflare Access application. It is not an RC Console application.

  • Full administrators remain on the standing full-admin policy and hold *.
  • Delegated operators are kept in a dedicated email-only policy managed from /admin/access.
  • The Worker verifies the Access JWT, then reads scoped permissions from pub_admin_roles.
  • Removing an operator revokes Pub permissions and then removes the delegated Access entry.

Add a delegated operator in /admin/access, not by editing RC Console or hand-editing the main Access policy. If the policy sync fails after the D1 role changes, the Access page shows the pending retry.

Troubleshooting

TVs show “Some TVs need attention”

The hub shows amber only for genuinely missing TVs (no heartbeat for 30+ minutes). If a TV is borderline stale (5-30 min), the full TV Health page shows details. Common causes:

  • TV went to sleep or lost network
  • Cloudflare Workers had a brief outage
  • TV browser crashed (check JS errors on the health card)

If the slot is an intentionally parked QA device, open /admin/slots and turn off Monitor for that slot. Its real state remains visible, but it no longer contributes to warning counts or offline-alert email.

Scheduler email not sending

  1. Check the This Week wizard — is Step 4 (Send) active?
  2. Confirm all shifts are Approved (Draft shifts block sending)
  3. Check /admin/pub-setup > Email Log for send failures
  4. Verify PUB_ADMIN_TOKEN is set as a Cloudflare secret

Setlist surfaces show stale data

The cron dispatcher picks ONE due surface per tick. If a surface hasn’t updated:

  1. Check its surface_config.enabled = 1 in SETLIST_DB
  2. Check its cadence (e.g., Persona/Stories run weekly, Top Songs runs hourly)
  3. Check /admin/api-health — Deezer degraded blocks Vibe Compass enrichment

Tutorial edits not saving

  1. Verify TUTORIALS_DATA KV namespace is bound in wrangler.toml
  2. Check the Access session and the operator’s tutorials.edit permission
  3. Preview a history version before restoring it
  4. Check browser console for 401/403/503 errors

Audio controls are missing or locked

  1. Confirm the Pi and Q-SYS Core are reachable on the System page.
  2. Refresh discovery on /admin/audio.
  3. Confirm the control is writable and included in an operator group.
  4. Check the group and control lock state.
  5. Capture a snapshot before changing several controls.

Song-ID pipeline (Pi-side)

The pipeline runs on a Raspberry Pi reached through its approved private tailnet hostname. Three systemd services:

ServiceWhat
ptp4l-eth0.servicePTP slave syncing to Q-SYS Core
aes67-daemon.serviceRAVENNA kernel module + ALSA capture
pi-capture.serviceFingerprint + POST to worker

All enabled on boot with dependency ordering. See pi-capture-ops for the full runbook.