IHNYC Platform Docs

Pub Scheduler v2 (state machine)

Development9 min read

Table of Contents

Built, but not active

The v2 engine ships dormant behind the PUB_SCHEDULER_V2 flag, which is OFF in production. The legacy implicit-status flow is still what runs the pub schedule today. Migration 0035 is applied to prod, but every column it adds is nullable/defaulted, so the live system behaves exactly as before until someone flips the flag during a planned cutover. This page explains the engine that will take over, not one that is running.

Why v2 exists

The original pub scheduler had no explicit notion of “what state is this week in.” A week’s progress was inferred from which timestamp columns on pub_schedule_runs happened to be null — has the form opened, were assignments generated, was it approved, was the email sent. State lived in the absence of data, spread across several columns, with no single place that said “you can’t do X from here.” That implicit timeline is what produced the class of bugs the rewrite targets: a late submission that never made it into the roster, double-sends, an action firing from a state where it shouldn’t have been legal.

v2 replaces that with one source of truth and one door:

  • A single column, v2_state, holds an explicit lifecycle state: DRAFT → COLLECTING → CLOSED → ASSIGNED → APPROVED → SENT.
  • A single function, transition(env, weekOf, to, opts) in worker/lib/pub-scheduler/state-machine.js, is the only way that state changes. Every actor — operator buttons, the autopilot cron, reopen — calls transition(). Illegal moves are rejected, not silently corrupting. There is no second code path.

The design principle is “one engine, two triggers”: the admin endpoints (when the flag is on) and the cron both go through the same transition() door, so there is never a hand-rolled state mutation that bypasses the rules.

Sources: worker/lib/pub-scheduler/state-machine.js, docs/pub-scheduler-v2-cutover.md

The run lifecycle

stateDiagram-v2
    [*] --> DRAFT: ensureRow<br/>(week created)
    DRAFT --> COLLECTING: open form<br/>(survey opened by caller)
    COLLECTING --> COLLECTING: extend deadline<br/>(self-edge)
    COLLECTING --> CLOSED: close form<br/>(lease: ACT.CLOSE)
    CLOSED --> COLLECTING: reopen<br/>(survey re-opened)
    CLOSED --> ASSIGNED: generate<br/>(lease: ACT.GENERATE)
    ASSIGNED --> ASSIGNED: regenerate in place<br/>(lease: ACT.GENERATE)
    ASSIGNED --> COLLECTING: reopen after generate<br/>(keep assignments, collect more)
    ASSIGNED --> APPROVED: operator approves
    APPROVED --> ASSIGNED: un-approve<br/>(reversible until sent)
    APPROVED --> SENT: send schedule email<br/>(lease: ACT.SEND)
    SENT --> [*]

    note right of SENT
        terminal — never re-sends.
        APPROVED and SENT write the
        legacy status column THROUGH
        ('approved' / 'sent') so a flag
        rollback lands the old engine
        on a coherent value.
    end note
StateMeaning
DRAFTThe week’s run row exists but the availability form hasn’t opened.
COLLECTINGThe form is open and tenders are submitting availability. Deadline extensions are a COLLECTING → COLLECTING self-edge.
CLOSEDThe form is closed; no more submissions accepted. Generation hasn’t run yet.
ASSIGNEDThe fairness picker has run and produced assignments. The operator reviews here.
APPROVEDThe operator approved the assignments (reversible — un-approve returns to ASSIGNED).
SENTThe schedule email went out. Terminal — any further action is rejected, never re-sends.

The states are modeled as a fixed TRANSITIONS table; each edge is either a pure status move (no side effect, a single conditional write) or a side-effecting edge that runs under a lease. extend, reopen, approve, and un-approve are pure moves; close, generate, and send are side-effecting.

Sources: worker/lib/pub-scheduler/state-machine.js (STATES, TRANSITIONS, transition)

The v2 schema additions (migration 0035)

Migration 0035_pub_runs_v2_state.sql adds columns to pub_schedule_runs. All of them are nullable or defaulted, so applying the migration is ZERO behavior change — the v2 engine treats a NULL v2_state as “not yet adopted” and seeds it lazily from the legacy status on the first transition.

ColumnPurpose
v2_state (TEXT, default NULL)The 6-state machine value. Its own column, deliberately not the legacy status.
acting_at / acting_kind (TEXT)The generalized in-flight lease. acting_at is when the claim was taken; acting_kind is which side effect is mid-flight (close | generate | send).
collecting_alert_at / unsent_alert_at (TEXT)Two independent once-per-run operator nudge claims.
autopilot (INTEGER, default 0)Per-run: when 1 the cron may auto-advance timed steps; when 0 (default) the cron only sends nudges.
kind / deadline_at / opens_atRun-shape future-proofing. kind='week' is the only v1 value; the timestamps let timing detach from the fixed cron clock.

A partial index, idx_pub_runs_v2_collecting ON pub_schedule_runs (v2_state) WHERE v2_state = 'COLLECTING', both speeds the open-form lookup and backs the “at most one COLLECTING run per survey” invariant (H3).

Why v2_state is its OWN column

This is the load-bearing decision. The base table pins the legacy column with CHECK (status IN ('pending','approved','sent')). SQLite can’t relax a CHECK without rebuilding the table — but more importantly, the cutover is flag-gated with idle-gap rollback. If the new engine wrote its new vocabulary (COLLECTING, CLOSED, ASSIGNED, …) into status, then flipping PUB_SCHEDULER_V2 off would strand a row that the old engine cannot read — it only understands pending/approved/sent.

Keeping v2_state separate means the legacy engine never sees the new states. The v2 engine writes the legacy status through on the two terminal-ish edges that have a legacy analog — APPROVED → 'approved' and SENT → 'sent' — so a rollback always lands the old engine on a coherent value. (COLLECTING/CLOSED/ASSIGNED have no legacy analog, so status simply stays 'pending' for those.)

Sources: migrations/content/0035_pub_runs_v2_state.sql, state-machine.js (LEGACY_STATUS_FOR, effectiveState, finalize)

Where 0035 sits in the run-state migration line

It’s the latest of a short series of additive run-state columns, each nullable so it ships dormant:

  • 0032 — one-shot data fix marking pre-Notion-migration weeks as sent.
  • 0033 — settings_override JSON for per-week deadline / generate-timing overrides.
  • 0034 — reopened_at, set when the operator re-opens the form after assignments were generated.
  • 0035 — the v2 state machine columns described above.

Concurrency: a lease column, not in-flight statuses

The in-flight guard is not a state like SENDING that a crash could strand. Instead it’s a lease pair on the row (acting_at + acting_kind). A side-effecting edge runs claim → act → finalize:

  1. claimLease() does a conditional UPDATE that succeeds for exactly one caller while the run is in the expected from state and the lease is free (or expired for the same kind). The atomic UPDATE is the concurrency guard.
  2. The side effect runs (close the survey / run the picker / send the email) as the transition’s act callback.
  3. On success, finalize() writes v2_state, clears the lease, writes the legacy status through, and stamps sent_at on the terminal send. On failure or throw, releaseLease() clears the lease and stamps last_error — the run stays in from and is retryable.

A stale lease (from a worker killed mid-action) is reclaimable only by the same kind: a crashed close can be reclaimed by another close, never by a send, and vice versa. This generalizes the existing single-purpose claimSendLease double-send guard (AD-12) to all three side-effecting edges. Because status is stable across a crash and there is no in-flight state to get stuck in, sent_at is only ever stamped on a genuine success.

Two extra invariants live in transition() itself:

  • H3 — at most one COLLECTING run per (single, global) survey: a second open/reopen into COLLECTING is refused while another run is already collecting.
  • H4 — un-approve (APPROVED → ASSIGNED) is refused while a send lease is live, so it can’t stomp a send it’s racing to cancel.

Sources: state-machine.js (claimLease, releaseLease, finalize, transition, leaseExpired)

Actions and the cron (the two triggers)

worker/lib/pub-scheduler/v2-actions.js maps each operator button (and each autopilot cron step) to exactly one transition(), wrapping the existing side-effect functions as the act callback:

EdgeSide effect
DRAFT/CLOSED/ASSIGNED → COLLECTINGopen/reopen survey (pre-move, not leased)
COLLECTING → CLOSEDcloseSurvey (lease: ACT.CLOSE)
CLOSED → ASSIGNEDrunGenerateShifts — the picker only; does not close the survey (lease: ACT.GENERATE)
ASSIGNED → APPROVEDapproveWeekShifts then the pure move
APPROVED → ASSIGNEDun-approve (pure move, H4-guarded)
APPROVED → SENTperformSend — the same send body the legacy path uses (lease: ACT.SEND)

worker/lib/pub-scheduler/v2-cron.js (tickV2Schedule) is the every-minute tick, and it’s a no-op unless PUB_SCHEDULER_V2 is set. When the flag is on, the legacy crons stand down and this owns time-based behavior:

  • Manual-mode nudges (the default, operator-driven path): a run the operator hasn’t advanced past its deadline gets one heads-up email — COLLECTING past the deadline → “ready to close & generate?”; APPROVED + unsent past the send window → “approved but not sent.” Each fires at most once per run, via the collecting_alert_at / unsent_alert_at claim columns. Nudges never change state.
  • Autopilot auto-advance (only for runs with autopilot=1): the cron auto-handles the boring timed steps — DRAFT → COLLECTING → CLOSED → ASSIGNED — and then stops at ASSIGNED for the operator to review and send. It never auto-approves and never auto-sends the mass email; the human keeps their hand on the thing that’s hard to unsend.

Sources: worker/lib/pub-scheduler/v2-actions.js, worker/lib/pub-scheduler/v2-cron.js

Cutover (flag-gated, idle-gap, dormant today)

Activation is operator-gated and follows docs/pub-scheduler-v2-cutover.md:

  1. Pick the idle gap — activate only when the current week is fully SENT and the next week’s form hasn’t opened, so no in-flight run crosses the boundary.
  2. Dry-run dress rehearsal (non-skippable) — with sends going only to the operator, run one complete real lifecycle through the wizard: open → extend → close → generate → reopen → regenerate → approve → send.
  3. Flip PUB_SCHEDULER_V2 ON — the v2 engine takes over; the legacy crons stand down automatically.
  4. Rollback = flip the flag OFF — safe only in the idle gap, because the old engine reads pending/approved/sent and v2_state lives in its own column.

The edge-case behavior the rehearsal and the automated tests have to cover — timing chaos, double-clicks, mid-flight data changes, external-service failures, operator mistakes, DST — is catalogued in docs/pub-scheduler-v2-scenarios.md.

Status as of this writing

The engine, actions, admin wiring, cron gating, and manual-mode nudges are all built and behind the flag, OFF in prod. Migration 0035 is applied. The critical generate-bucketing fix (H1) that started the rewrite is already live and not flag-gated. Remaining gates before full activation: the autopilot product decision (it currently stops at ASSIGNED) and the non-skippable dry-run rehearsal.

Sources: docs/pub-scheduler-v2-cutover.md, docs/pub-scheduler-v2-scenarios.md, tests/pub-scheduler-state-machine.test.js, tests/pub-scheduler-v2-actions.test.js, tests/pub-scheduler-v2-cron.test.js