Raspberry Pi audio runbook

Runtime summary

Operational runbook for the pub Raspberry Pi that captures AES67 audio off the Q-SYS Core, identifies the song, and posts to the Cloudflare Worker
The Pi runs three cooperating systemd services that together carry one audio packet from the AV LAN to the production Worker
All three services are enabled on boot and have Restart=on-failure so transient hiccups recover automatically
The full architecture is documented in now-playing-system; this page is the runbook for keeping it alive

Sources: pi/README.md, pi/systemd/pi-capture.service, /etc/systemd/system/aes67-daemon.service (on Pi), /etc/systemd/system/ptp4l-eth0.service (on Pi)

Boot chain

The services start in strict dependency order so each one’s prerequisites are ready by the time it runs:

flowchart TD
  BOOT["Power on"] --> KMOD["Kernel loads MergingRavennaALSA<br/>(via /etc/modules-load.d/aes67.conf)"]
  KMOD --> NET["network-online.target"]
  NET --> PTP["ptp4l-eth0.service<br/>(linuxptp slave on eth0)"]
  PTP --> DAEMON["aes67-daemon.service<br/>(WebUI :8089, IGMP-joins 233.254.6.0)"]
  DAEMON --> CAP["pi-capture.service<br/>(sounddevice opens RAVENNA, posts to Worker)"]

Service	Requires	After	Why
`ptp4l-eth0.service`	—	`network-online.target`	Needs `eth0` link up
`aes67-daemon.service`	`ptp4l-eth0.service`	`ptp4l-eth0.service`, `network-online.target`, `sound.target`	Daemon needs PTP locked before Sink will receive
`pi-capture.service`	`aes67-daemon.service`	`aes67-daemon.service`, `network-online.target`, `sound.target`	Needs the daemon’s ALSA capture device to exist

Time-to-ready from cold boot: ~60 seconds. The slowest link is the aes67-daemon’s PTP lock (~10 sec after start).

Sources: /etc/systemd/system/*.service on the Pi

Environment variables

The pi-capture service loads its environment from /opt/pi-capture/.env. The other two services have no env file; they read /etc/linuxptp/ptp4l-eth0.conf and /opt/aes67-linux-daemon/daemon/daemon.conf respectively.

/opt/pi-capture/.env keys that matter:

Var	Production value	Notes
`WORKER_INGEST_URL`	`https://pub.ihnyc-rc.org/api/now-playing/ingest`	Where identified tracks post. Override to `http://<mac-tailscale-ip>:8787/api/now-playing/ingest` for private dev.
`WORKER_INGEST_TOKEN`	matches the Cloudflare secret	See ops. Required for prod; blank-OK for `wrangler dev`.
`CAPTURE_BACKEND`	`aes67`	`usb` for dev on a Mac with a USB interface; `aes67` for the Pi.
`AUDIO_DEVICE`	`RAVENNA`	sounddevice substring-matches PortAudio device names. Use `RAVENNA` not the numeric index for stability across reboots.
`SAMPLE_RATE`	`48000`	Matches AES67 stream. `11025` only for `usb` dev mode.
`NUM_CHANNELS`	`6`	AES67 stream is 6ch.
`SOURCE_OVERRIDE`	(empty)	Irrelevant in AES67 mode (source comes from the RMS picker). Only used in `usb` mode.
`SONGID_BACKEND`	`shazamio`	`acoustid` / `audd` / `shazamkit` are the other documented choices.
`ACOUSTID_API_KEY`	(unset)	Only needed for `SONGID_BACKEND=acoustid`.
`AUDD_API_TOKEN`	(unset)	Only needed for `SONGID_BACKEND=audd`.
`WINDOW_SECONDS`	`6.0`	Rolling buffer length. Chromaprint needs ~3s+ for a useful fingerprint.
`TICK_SECONDS`	`2.0`	Main loop fingerprint cadence; worst-case lag from song-change to POST.
`SILENCE_DBFS`	`-60.0`	RMS silence floor; raise (e.g., `-50`) if ambient room noise alone triggers fingerprints.
`FP_DIFF_THRESHOLD`	`6`	Min byte-diff over leading 24 chars of consecutive Chromaprint fingerprints to count as new song. Lower = more sensitive (more API calls).
`SILENCE_SECONDS`	`30.0`	How long below `SILENCE_DBFS` before posting a silence sentinel.

.env is loaded automatically via python-dotenv from pi/pi_capture/config.py. Real shell environment variables override .env, keeping systemd’s EnvironmentFile= path identical to dev.

Sources: pi/pi_capture/config.py, pi/config.example.env

Daily operations

Where everything lives on the Pi

/opt/pi-capture/
|-- .venv/                       # Python 3.9 venv with shazamio<0.7, numpy, sounddevice, pyacoustid, requests, python-dotenv
|-- .env                         # Local config (NOT committed; secrets live here)
|-- config.example.env           # Template — copy to .env on first deploy
|-- pi_capture/                  # Python module (rsynced from the avi-pub-landing repo's pi/)
|   |-- __main__.py              # Main loop
|   |-- audio.py                 # PortAudio wrapper + RollingBuffer
|   |-- capture.py               # usb_extract + aes67_extract (RMS source-picker lives here)
|   |-- config.py                # Env loader
|   |-- fingerprint.py           # Chromaprint
|   |-- poster.py                # POST to Worker
|   |-- songid.py                # Backend dispatch
|   `-- requirements.txt
|-- systemd/
|   `-- pi-capture.service       # Source template; installed copy is /etc/systemd/system/pi-capture.service
 
/opt/aes67-linux-daemon/         # bondagit/aes67-linux-daemon, built from source
|-- daemon/
|   |-- aes67-daemon             # Built binary
|   |-- daemon.conf              # Runtime config; http_port=8089 (NOT 8080), interface_name=eth0
|   `-- status.json              # Persisted Sink configuration; survives daemon restart
|-- 3rdparty/ravenna-alsa-lkm/
|   `-- driver/MergingRavennaALSA.ko    # Built kernel module; copied to /lib/modules/$(uname -r)/extra/
`-- webui/dist/                  # WebUI served by the daemon at http://<pi-av-host>:8089
 
/lib/modules/6.1.21-v8+/extra/MergingRavennaALSA.ko   # Auto-loaded via /etc/modules-load.d/aes67.conf
 
/etc/systemd/system/
|-- ptp4l-eth0.service
|-- aes67-daemon.service
`-- pi-capture.service
 
/etc/linuxptp/ptp4l-eth0.conf    # slaveOnly=1, time_stamping=software, free_running=1
/etc/modules-load.d/aes67.conf   # Auto-loads MergingRavennaALSA on boot

Common service operations

# Status of all three at once
systemctl status ptp4l-eth0 aes67-daemon pi-capture --no-pager | head -50
 
# Live logs (most useful one)
sudo journalctl -u pi-capture -f
 
# Live logs across all three
sudo journalctl -u ptp4l-eth0 -u aes67-daemon -u pi-capture -f
 
# Restart everything cleanly (rare; usually a single service restart is enough)
sudo systemctl restart ptp4l-eth0
sleep 10
sudo systemctl restart aes67-daemon
sleep 10
sudo systemctl restart pi-capture
 
# Just bounce pi-capture (most common during dev)
sudo systemctl restart pi-capture

Updating pi-capture code

# On your Mac, with the avi-pub-landing repo checked out at the right commit
rsync -av --exclude='.venv' --exclude='.env' --exclude='__pycache__' \
  pi/ pi@<pi-tailnet-hostname>:/opt/pi-capture/
 
# On the Pi
cd /opt/pi-capture
source .venv/bin/activate
pip install -r requirements.txt    # in case deps changed
sudo systemctl restart pi-capture
sudo journalctl -u pi-capture -f

Updating the daemon

The daemon was built from bondagit/aes67-linux-daemon source. To rebuild after an upstream update:

cd /opt/aes67-linux-daemon
git pull
git submodule update --init --recursive
cd 3rdparty/ravenna-alsa-lkm/driver
make                                # rebuild kernel module against current kernel
sudo cp MergingRavennaALSA.ko /lib/modules/$(uname -r)/extra/
sudo depmod -a
cd /opt/aes67-linux-daemon/daemon
cmake -DENABLE_TESTS=OFF -DWITH_STREAMER=OFF \
      -DCPP_HTTPLIB_DIR=/opt/aes67-linux-daemon/3rdparty/cpp-httplib \
      -DRAVENNA_ALSA_LKM_DIR=/opt/aes67-linux-daemon/3rdparty/ravenna-alsa-lkm \
      -DWITH_AVAHI=ON -DWITH_SYSTEMD=ON .
make -j$(nproc)
sudo systemctl restart aes67-daemon
sudo systemctl restart pi-capture     # follow-on, since the daemon restart drops the Sink temporarily

WITH_STREAMER=OFF skips the FAAC dependency for HTTP audio streaming; we don’t need it. ENABLE_TESTS=OFF skips test binaries.

Daemon WebUI

The daemon’s WebUI is at http://<pi-av-host>:8089 (on the AV LAN) or via a tailnet port-forward from a Mac. Obtain the current private hostname from the approved infrastructure inventory:

ssh -L 8089:localhost:8089 pi@<pi-tailnet-hostname>
# Then in browser: http://localhost:8089

Useful tabs:

PTP — current clock status, grandmaster identity, lock state
Sinks — the AES67 stream we receive; click for live packet stats
Browser — SAP/mDNS-discovered streams on the network

The Sink config is persisted in /opt/aes67-linux-daemon/daemon/status.json and survives daemon restarts. If you need to add it from scratch, see Add the sink manually below.

Sources: /opt/aes67-linux-daemon/daemon/daemon.conf, bondagit/aes67-linux-daemon README

Add the sink manually

The Q-SYS Designer doesn’t expose SAP advertisement on this firmware, but the Core does advertise via SAP. Either pick the AES67-TX-1 source from the WebUI’s “Remote Source SDP” dropdown, or paste this hand-built SDP into the “Use SDP” field. Replace <qsys-core-av-ip> with the current address from the approved infrastructure inventory:

v=0
o=- 1 1 IN IP4 <qsys-core-av-ip>
s=AES67-TX-1
c=IN IP4 233.254.6.0/32
t=0 0
m=audio 5004 RTP/AVP 96
i=AES67-TX-1
a=recvonly
a=rtpmap:96 L24/48000/6
a=ptime:1
a=mediaclk:direct=0

Sink settings to use:

Channels: 6
Audio Channels map: ALSA Input 1 → ALSA Input 6 (1:1 pass-through)
Delay (samples): 576 (12ms @ 48kHz). Bump to 768 or 960 if you see buffer underruns.
Ignore RefClk GMID: ✅ checked (our SDP has no ts-refclk line; the daemon would reject the stream otherwise)

Save. The Sink should report status receiving within a few seconds, packets-received climbing fast.

Sources: docs/20260520-102621-pi-aes67-bridge-office-hours.md (manual SDP section)

PTP troubleshooting

PTP is the trickiest part of this stack and the most common source of confusion. The journalctl -u ptp4l-eth0 output looks alarming on first read but is expected behavior for this installation.

What “normal” looks like

selected best master clock 001dc1.fffe.9ca974
foreign master not using PTP timescale
running in a temporal vortex
port 1: LISTENING to UNCALIBRATED on RS_SLAVE
rms 1779167712374750976 max … freq +3362 …

Decoded:

001dc1.fffe.9ca974 is the Dante endpoint’s clock identity (MAC 00:1d:c1:9c:a9:74), advertised through the Q-SYS Core as the AES67 grandmaster. The Core is acting as a PTP Boundary Clock.
“foreign master not using PTP timescale” and “running in a temporal vortex” are warnings, not errors. The grandmaster is using ARB (boot-relative) time, not UTC/TAI. Audio PTP doesn’t need wall-clock alignment.
The huge rms value (~1.78×10^18 ns ≈ 56 years) is the difference between the Pi’s NTP-disciplined system clock and the master’s ARB epoch. It will never close because we run with free_running 1 — ptp4l observes the master but does NOT slew the system clock.
UNCALIBRATED state can persist; that’s fine. The RAVENNA kernel module reads PTP state independently (parses Sync/FollowUp directly from eth0) and clocks audio reception correctly regardless of what ptp4l’s state machine reports.

What “broken” looks like and how to fix

Symptom	Likely cause	Fix
`ptp4l` never picks a foreign master (only `LISTENING`)	No PTP packets reaching `eth0`, OR the AV switch is blocking PTP frames	`sudo tcpdump -i eth0 -nn -c 5 'port 319 or port 320'` — should show packets from the Q-SYS Core. If empty, switch / Q-SYS issue.
`ptp4l` keeps choosing `local clock` as best master (with “defaultDS.priority1 probably misconfigured”)	Pi’s priority1 is lower than master’s; BMCA picks Pi	Verify `/etc/linuxptp/ptp4l-eth0.conf` has `priority1 255`, `priority2 255`, `clockClass 255` in `[global]`
`failed to step clock: Invalid argument` + `freq +100000000`	`free_running` not set; ptp4l trying to slew system clock to ARB time	Verify `free_running 1` in `[global]` section of `ptp4l-eth0.conf`
Audio glitches every few hours, Core event log shows “Audio Clock Set Events, Processing overrun”	Q-SYS Core / Dante PTP grandmaster flap	Designer-side fix; see now-playing-system

The canonical `/etc/linuxptp/ptp4l-eth0.conf`

[global]
slaveOnly               1
priority1               255
priority2               255
clockClass              255
free_running            1
time_stamping           software
domainNumber            0
delay_mechanism         E2E
network_transport       UDPv4
logAnnounceInterval     1
logSyncInterval         -3
announceReceiptTimeout  3
syncReceiptTimeout      3
tx_timestamp_timeout    50
 
# Declare eth0 as a managed PTP port. Empty body = use [global] defaults.
[eth0]

time_stamping software is mandatory because the Pi’s eth0 has no PHC (hardware PTP clock). phc2sys is NOT used for the same reason.

Sources: /etc/linuxptp/ptp4l-eth0.conf on the Pi, linuxptp man pages, docs/20260520-102621-pi-aes67-bridge-office-hours.md (full PTP debug history)

Other troubleshooting

`pi-capture` says `Device or resource busy` on RAVENNA

Something else has the ALSA device open. Almost always PulseAudio or PipeWire reviving via socket activation. Confirm they’re masked:

systemctl --user status pulseaudio pipewire pipewire-pulse 2>&1 | grep -E "Loaded|Active"
# Expected: masked, inactive
 
# If running, remask:
systemctl --user stop pipewire-pulse.socket pipewire-pulse.service \
                     pipewire.socket pipewire.service \
                     pulseaudio.socket pulseaudio.service
systemctl --user mask pipewire-pulse.socket pipewire-pulse.service \
                     pipewire.socket pipewire.service \
                     pulseaudio.socket pulseaudio.service
sudo systemctl restart pi-capture

`arecord -l` shows no RAVENNA card

Kernel module isn’t loaded. Verify:

lsmod | grep -i ravenna       # Should show MergingRavennaALSA + dependents
sudo modprobe MergingRavennaALSA
dmesg | tail -20              # Look for init lines from the module

If module is loaded but ALSA doesn’t see the card, the daemon hasn’t started or hasn’t talked to the kernel module yet. Check systemctl status aes67-daemon.

`pi-capture` posts time out at 10s

First POST after a long idle is sometimes slow due to TLS handshake. Subsequent posts succeed. If it happens repeatedly:

Confirm the Pi has internet on wlan0 (curl https://pub.ihnyc-rc.org/api/now-playing/setlist)
Bump the timeout in pi/pi_capture/poster.py from 10 to 15 or 20
Check Cloudflare for upstream rate-limiting or outages

The Sink is `receiving` but no song-IDs land

Confirm a source is actually playing audio above -60 dBFS (use the daemon WebUI Sink stats or arecord -D plughw:CARD=RAVENNA -c 6 -f S24_LE -r 48000 -d 5 /tmp/test.wav and inspect)
Check journalctl -u pi-capture -f for change detected lines without follow-up posted lines — that’s the song-ID backend returning empty (catalog miss on acoustid, or shazamio rate-limited)
Try changing SONGID_BACKEND temporarily to confirm the backend is the issue, not the upstream audio

Track shows up with wrong source chip on `/setlist/`

The RMS source-picker chose the wrong pair. Causes:

The actual loudest pair is correct but the listener thinks otherwise (e.g., DJ-by-XLR is louder than pub Spotify and we tagged it wallxlr, which is correct even if Spotify is “primary”)
Channel mapping in Q-SYS Designer has drifted from the documented order
AES67_PAIR_LABELS in pi/pi_capture/config.py doesn’t match the actual channel order

Verify by capturing a 5-second WAV with arecord -D plughw:CARD=RAVENNA -c 6 -f S24_LE -r 48000 -d 5 /tmp/test.wav and inspecting per-channel RMS via Audacity or ffmpeg -i /tmp/test.wav -af "channelsplit=channel_layout=hexagonal,astats=measure_perchannel=RMS_level" -f null -.

Reboot test

To confirm the full chain comes back from cold:

sudo reboot
# Wait 60-90 seconds, then ssh back:
ssh pi@<pi-tailnet-hostname> 'systemctl is-active ptp4l-eth0 aes67-daemon pi-capture'
# Expected:
# active
# active
# active
 
# Then watch a real song-ID land (with music on the program bus):
ssh pi@<pi-tailnet-hostname> 'sudo journalctl -u pi-capture -f --since "1 minute ago"'

If anything’s inactive or failed, the failure mode tells the gap:

Kernel module didn’t auto-load → check /etc/modules-load.d/aes67.conf exists and .ko is in /lib/modules/$(uname -r)/extra/
Daemon failed → check systemctl status aes67-daemon + journal; usually a port collision or missing status.json
pi-capture failed → check systemctl status pi-capture + journal; usually a sounddevice-can’t-open-RAVENNA issue

Coexistence with other Pi services

The Pi runs several services besides this pipeline. None should conflict, but worth knowing:

Service	Port(s)	Notes
`mediamtx`	1935, 8554, 8888, 8889, 8000, 8001, 8890	RTMP/RTSP/HLS. Unrelated to our work.
node app	8080	Daemon WebUI was reconfigured to 8089 to avoid this collision.
`mariadb`	3306	Unrelated.
VNC	5900	Unrelated.
Tailscale	(no fixed port)	Used for remote ssh + Worker access for dev.
hostapd / dnsmasq	(AP mode)	Pi-as-AP feature; unrelated.

The new ports in use by our pipeline:

8089 — daemon WebUI (HTTP)
8854 — daemon RTSP server (mostly unused for receive-only)
5004 — RTP multicast (233.254.6.0), inbound on eth0 only
PTP 319/320 — multicast on eth0

Sources: docs/20260516-211208-pi-karray-song-id-office-hours.md (constraints section)

Status reporter operations

The status reporter runs beside pi-capture and sends system, audio, and device health to /admin/tv-health. Its architecture, payload, and TTL are documented in Raspberry Pi status reporting.

Deploy

From pi/pi-status-reporter/ in the Pub source repository:

# Copy the script and systemd unit
sudo mkdir -p /opt/pi-status-reporter
sudo cp main.py /opt/pi-status-reporter/
sudo cp ../systemd/pi-status-reporter.service /etc/systemd/system/
 
# Create the virtual environment and install dependencies
cd /opt/pi-status-reporter
sudo python3 -m venv .venv
sudo .venv/bin/pip install psutil requests
 
# Create the environment file, which contains the shared ingest secret
sudo tee /opt/pi-status-reporter/.env >/dev/null <<'EOF'
WORKER_URL=https://pub.ihnyc-rc.org/api/pi-status
INGEST_TOKEN=<same NOW_PLAYING_INGEST_TOKEN as pi-capture>
REPORT_INTERVAL_SECONDS=30
WATCH_SCRIPTS=pi-capture,pi-status-reporter
EOF
sudo chmod 600 /opt/pi-status-reporter/.env
 
# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable --now pi-status-reporter.service
 
# Verify the service and its first reports
journalctl -u pi-status-reporter -n 20 -f

Then confirm that /admin/tv-health shows a fresh Pi card with host, CPU, memory, and last-seen data.

The unit runs as pi with Restart=on-failure, RestartSec=10, and StartLimitBurst=5 over 60 seconds. The bounded restart policy avoids a tight failure loop.

Sources: pi/pi-status-reporter/main.py (header comment block), pi/systemd/pi-status-reporter.service

Environment variables

/opt/pi-status-reporter/.env accepts:

Variable	Required	Default	Purpose
`WORKER_URL`	yes	—	Full URL including `/api/pi-status`
`INGEST_TOKEN`	yes	—	Bearer token matching the Worker secret `NOW_PLAYING_INGEST_TOKEN`
`REPORT_INTERVAL_SECONDS`	no	`30`	Reporting interval in seconds
`WATCH_SCRIPTS`	no	`""`	Comma-separated unit names, with or without `.service`; the Worker accepts at most 10
`TRANSCODE_BACKLOG_DIR`	no	`/opt/pi-capture/transcode-queue`	Directory whose entry count becomes `transcodeBacklog`; `null` if absent
`LOG_LEVEL`	no	`INFO`	Python logging level

A missing WORKER_URL or INGEST_TOKEN makes the reporter log an error and exit with status 2. The unit’s start limit prevents an unbounded restart loop.

Sources: pi/pi-status-reporter/main.py (read_env)

Verify and restart

sudo systemctl status pi-status-reporter
sudo journalctl -u pi-status-reporter -n 20 -f

A healthy report resembles:

2026-05-25 14:32:11 INFO pi-status-reporter reported cpu=18.2% mem=24.4% disk=20.7% temp=54.3

A network error is logged as WARNING pi-status-reporter post failed: <error> and the reporting loop continues.

The reporter reads .env only at startup. Restart it after changing configuration:

sudo systemctl restart pi-status-reporter

Update or pause

From the Pub repository root, deploy a new script and restart the service:

scp pi/pi-status-reporter/main.py pi@<pi-tailnet-hostname>:/opt/pi-status-reporter/main.py
ssh pi@<pi-tailnet-hostname> 'sudo systemctl restart pi-status-reporter'

The unit runs the file directly; no build or symlink change is required.

Pause reporting without uninstalling it:

sudo systemctl stop pi-status-reporter

The dashboard card expires within 10 minutes. Restart the service after the maintenance or benchmark window.

IHNYC Platform Docs

Documentation

Raspberry Pi audio runbook

Table of Contents

Runtime summary

Boot chain

Environment variables

Daily operations

Where everything lives on the Pi

Common service operations

Updating pi-capture code

Updating the daemon

Daemon WebUI

Add the sink manually

PTP troubleshooting

What “normal” looks like

What “broken” looks like and how to fix

The canonical `/etc/linuxptp/ptp4l-eth0.conf`

Other troubleshooting

`pi-capture` says `Device or resource busy` on RAVENNA

`arecord -l` shows no RAVENNA card

`pi-capture` posts time out at 10s

The Sink is `receiving` but no song-IDs land

Track shows up with wrong source chip on `/setlist/`

Reboot test

Coexistence with other Pi services

Status reporter operations

Deploy

Environment variables

Verify and restart

Update or pause

Table of Contents

IHNYC Platform Docs

Documentation

Raspberry Pi audio runbook

Table of Contents

Runtime summary

Boot chain

Environment variables

Daily operations

Where everything lives on the Pi

Common service operations

Updating pi-capture code

Updating the daemon

Daemon WebUI

Add the sink manually

PTP troubleshooting

What “normal” looks like

What “broken” looks like and how to fix

The canonical /etc/linuxptp/ptp4l-eth0.conf

Other troubleshooting

pi-capture says Device or resource busy on RAVENNA

arecord -l shows no RAVENNA card

pi-capture posts time out at 10s

The Sink is receiving but no song-IDs land

Track shows up with wrong source chip on /setlist/

Reboot test

Coexistence with other Pi services

Status reporter operations

Deploy

Environment variables

Verify and restart

Update or pause

Related

Table of Contents

The canonical `/etc/linuxptp/ptp4l-eth0.conf`

`pi-capture` says `Device or resource busy` on RAVENNA

`arecord -l` shows no RAVENNA card

`pi-capture` posts time out at 10s

The Sink is `receiving` but no song-IDs land

Track shows up with wrong source chip on `/setlist/`