​

2026-05-23 — Workflow Lifecycle & Automation Hygiene

​

Summary

​

Empirical findings (2026-05-23)

​

Decision

​

Pillar 1 — Tiered SLO + paging

​

Pillar 2 — Auto-quarantine for chronic failures

1. Disables its schedule via gh workflow disable
2. Opens an issue labeled workflow-quarantined and priority/p1 against the workflow’s CODEOWNER
3. Posts to Pushover with the offending workflow + last error step

​

Pillar 3 — Snapshot PR auto-merge contract

1. Open the PR
2. Approve from factory-cross-repo[bot] (already a bypass actor on ruleset 15843812)
3. Enable gh pr merge --auto --squash

paths:
  - docs/STATE.md
  - docs/cost/**
  - docs/conformance/**
  - docs/launch-readiness/**
  - docs/digests/**
  - docs/revenue/**
  - completion-tracker.json

​

Pillar 4 — Workflow budget gate

• retires: <existing-workflow.yml> (and the file must be deleted in the same PR), OR
• budget-exception: <reason> (and a CODEOWNER must ✅ the PR)

​

Pillar 5 — Concurrency hygiene as a typecheck

​

Pillar 6 — Visible Tier-1 health dashboard

​

Why

​

Implementation plan

​

Phase 0 — Stop the bleeding (this PR’s pre-work, hours)

​

Phase 1 — Tier registry + concurrency baseline (1 PR, ~3h)

• .github/workflows/REGISTRY.md (new) — table of all 95 workflows, tier assignment, CODEOWNER, retirement notes
• .github/workflows/_workflow-concurrency-check.yml (new) — validator that fails when a push: or pull_request: workflow lacks concurrency:
• All 47 workflows missing concurrency: — single mechanical PR adds the standard block:
  concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}
    cancel-in-progress: true
  
• .github/CODEOWNERS — add /.github/workflows/REGISTRY.md @adrper79-dot to lock the registry to CODEOWNER edits

• REGISTRY.md classifies all 95 workflows (Tier-1: ~12, Tier-2: ~25, Tier-3: ~58 by current estimate)
• workflow-concurrency-check passes on a sample PR touching no workflow files
• workflow-concurrency-check fails on a deliberately-broken test PR (manual verification)
• All 47 listed workflows have concurrency blocks
• Merge does not break any existing green workflow

​

Phase 2 — Snapshot PR contract + backlog drain (1 PR, ~3h)

• .github/snapshot-paths.yml (new) — the allowlist
• .github/scripts/snapshot-pr-helper.mjs (new) — helper invoked by each snapshot cron after opening a PR: validates paths against allowlist, approves via factory-cross-repo[bot], enables auto-merge. Refuses to act on PRs that touch out-of-allowlist paths.
• Each snapshot-generating workflow (generate-state.yml, cost-observability.yml, revenue-digest.yml, morning-digest.yml, completion-tracker.yml, platform-conformance.yml, launch-readiness.yml, track-kpis.yml, pr-queue-digest.yml) — add a final step calling snapshot-pr-helper.mjs against the opened PR
• docs/runbooks/snapshot-pr-contract.md (new) — the operator-facing runbook

• Next state-snapshot PR auto-merges within 10 minutes of opening
• An adversarial test PR with a snapshot path + an out-of-allowlist path is not auto-merged (helper rejects it)
• docs/STATE.md updates daily without human intervention
• Phase-0 backlog drained (precondition); ongoing backlog stays at 0

​

Phase 3 — Workflow Health Warden + paging (1 PR, ~4h)

• .github/workflows/workflow-health-warden.yml (new) — daily cron at 13 13 * * * (after morning-digest, before cohesion-courtesy-check)
• .github/scripts/workflow-health-warden.mjs (new) — the logic:
  1. Read REGISTRY.md for tier assignments
  2. For each workflow, fetch last 20 runs via gh api
  3. Compute consecutive-failure count
  4. For Tier-1 red >1h: Pushover P1 + create/update priority/p0 issue (dedup by title)
  5. For Tier-2 red >24h: append to daily digest issue + create/update priority/p1 issue
  6. For Tier-3 with ≥10 consecutive failures: gh workflow disable + create workflow-quarantined + priority/p1 issue
• docs/runbooks/workflow-health-warden.md (new) — runbook including how to release a quarantined workflow

• First scheduled run completes successfully
• Quarantine logic tested against a deliberately-broken Tier-3 workflow (or unit test against fixture run-history)
• Pushover alert verified end-to-end on a Tier-1 simulated failure
• Dedup logic: re-running the warden does not create duplicate issues for the same red workflow

​

Phase 4 — Budget gate + dashboard surface (1 PR, ~3h)

• .github/workflows/workflow-budget-check.yml (new) — required check on PRs touching .github/workflows/**.yml
• .github/scripts/workflow-budget-check.mjs (new) — parses PR body for retires: or budget-exception:, fails if neither is present and a new workflow file is added
• factory-status-dashboard.yml — extend to emit Tier-1 health summary into docs/STATE.md
• scripts/generate_state.py — read warden output, inject the Tier-1 status line at top
• Branch protection ruleset 15843812 — add workflow-budget-check and workflow-concurrency-check as required contexts (via .github/security/main-ruleset.json so apply-sec-hardening will self-apply)

• Adversarial PR adding a new workflow without retires: or budget-exception: is blocked
• Legitimate retirement PR (deletes one, adds one, mentions retires:) passes
• docs/STATE.md line 1 shows the Tier-1 status summary
• Ruleset change verified via apply-sec-hardening run

​

Consequences

• Tier-1 workflow red state cannot remain undetected for >1h
• Snapshot PR backlog goes to zero and stays there
• Cancelled-run noise drops sharply (47 workflows × ~average 3 cancellations/day → ~140 fewer runs/day)
• New workflows cost something (a retirement or an exception); growth is bounded
• Workflow conformance sub-score (currently 33/100) moves toward 80+ across the portfolio

• 4 new workflows added (warden, budget-check, concurrency-check, snapshot-helper as called-job) — but Pillar 4’s budget gate applies to them too; each is documented and load-bearing
• Snapshot PRs lose the (theoretical) human-review eye on bot-generated data — this is deliberate (see Pillar 3 rationale)
• Quarantining a workflow could mask a real signal if its CODEOWNER doesn’t react to the issue — mitigated by Pushover paging on quarantine event

​

Non-goals (explicit)

• Change the supervisor loop’s pickup gating or template flow
• Change the auto-merge triangulation (auto-merge-approved-prs + auto-merge-spotter + copilot-* remain as designed)
• Modify FRIDGE rules
• Touch wordis-bond, packages/, migrations/, or Stripe code paths (Red-tier per FRIDGE rule 3)
• Migrate any workflow off scheduled triggers to webhooks (separate decision)
• Flip allowed_actions: "all" → "selected" or enable sha_pinning_required: true — these are supply-chain hardening decisions worth their own RFC; this decision is about lifecycle, not perimeter

​

Revisit when

• All four phases land and Tier-1 red-state SLO is observed for 30 days, OR
• A workflow-related incident occurs that the lifecycle didn’t catch (file new decision describing the gap), OR
• Workflow count crosses 110 (the budget gate should be making this near-impossible; if it happens, the gate is being bypassed)

​

References

• docs/supervisor/FRIDGE.md — ten rules; this decision honors rules 3, 6, 8, 9
• docs/architecture/FACTORY_V1.md — canonical architecture
• docs/STATE.md — live state; will be updated in Phase 4 to surface Tier-1 health
• Incident #529 — the original branch-protection-by-hand regression that motivates apply-sec-hardening
• PR #602 — the audit that hardened apply-sec-hardening to its current shape
• .github/workflows/COORDINATION.md — existing workflow-level coordination notes (Phase 1 REGISTRY.md will supersede)