Factory Operational Patterns
Loaded by: supervisor, Claude reviewer, sub-agents, Claude Code sessionsSibling docs:
PLATFORM_STANDARDS.md (the norms — what we build) · FRIDGE.md (the non-negotiable rules — what we never break)Updated: 2026-05-23 — added patterns 7–9 from Capability Design Studio Stage C close-out This doc captures operational know-how — patterns that emerged from production debugging that the next agent (or future-you) would otherwise rediscover from scratch. Each entry is short: symptom → root cause → fix, with a code/commit reference so the fix is auditable. If you’re about to write a workflow, touch a GCP integration, or rotate a credential, search this doc first.
1. gcloud in GitHub Actions needs explicit --project
Symptom: gcloud secrets versions access latest --secret=X returns NOT_FOUND for every secret name silently. No error to stderr; the secret IS present and IAM is correctly granted.
Root cause: google-github-actions/auth@v3 authenticates the service account but does not set gcloud’s default project. Without --project, gcloud has no project context to look in.
Fix: Pass --project on every call. Default to factory-495015 (canonical Factory project), overridable via $GCP_PROJECT / $GOOGLE_CLOUD_PROJECT / $CLOUDSDK_CORE_PROJECT.
Reference: scripts/fetch_gcp_secrets.sh · landed in PR #687 (commit 69e19cd4)
2. Stale service-account keys present as invalid_grant: Invalid JWT Signature
Symptom: Workflow auth step “succeeds,” but every subsequent gcloud command fails with ERROR: There was a problem refreshing auth tokens for account <sa>: ('invalid_grant: Invalid JWT Signature.', ...). IAM bindings on the service account are correct; the secret is present; the service-account email in the key matches the granted SA.
Root cause: The user-managed JSON key in your secret store is valid format but the corresponding key in GCP IAM has been deleted or rotated. JWT signed locally, but GCP no longer recognizes the signing key.
Fix:
scripts/fetch_gcp_secrets.sh (active-account diagnostic line) · landed in PR #692
3. Direct push to main is branch-protected — workflows must use PR-per-snapshot
Symptom: Workflow that writes a docs snapshot (cost, conformance, STACK manifest, completion tracker) commits cleanly but git push fails with:
main is branch-protected. Even github-actions[bot] can’t bypass.
Fix: Workflow opens a per-run PR with the auto-merge label. Factory’s existing auto-merge bot squashes once required checks pass.
.github/workflows/completion-tracker.yml · applied to cost / conformance / stack-manifest workflows in PR #689
4. git diff --quiet docs/X/ doesn’t see new untracked files
Symptom: Workflow that writes new JSON snapshots into a directory always logs “No changes to commit” — but the files are clearly being created.
Root cause: git diff --quiet only inspects the working tree against HEAD for tracked paths. New untracked files are invisible to it.
Fix: Stage first, then diff the index.
5. Secret values with UTF-8 BOM break latin-1/ascii HTTP encoding
Symptom: Secret fetches all succeed, but every HTTP call using them fails:\xEF\xBB\xBF, invisible in most UIs). Python’s HTTP libraries use latin-1/ascii for header values — BOM is not encodable.
Fix: Strip BOM + trailing whitespace inline in your fetch helper. Don’t rely on each call site doing it.
scripts/fetch_gcp_secrets.sh · landed in PR #696
6. --admin merge as escape hatch for transient BLOCKED race
Symptom: PR is MERGEABLE / BLOCKED, all required checks SUCCESS, review APPROVED, auto-merge enabled — but it doesn’t merge. State persists.
Root cause: strict_checks: true + dismiss_stale_reviews: true combine with main advancing between check-pass and merge: each update-branch re-runs checks, which may dismiss the approval, which removes mergeability, and the auto-merge bot can’t reliably re-arm fast enough.
Fix:
- For your own PRs you’ve authored + reviewed:
gh pr merge <N> --squash --admin. Bypasses branch protection cleanly. Audit log records the bypass. - For PRs you didn’t author: prefer to wait or coordinate with the reviewer — don’t bypass other people’s PR gates.
- For automation PRs (snapshot PRs from workflows): admin-merge is the right tool; the audit trail is the commit body.
7. Staging worker that only responds on its custom domain needs stagingCustomDomain
Symptom: Admin Studio’s Overview dashboard shows HTTP 404 for a staging app health check even though the worker is deployed and serving traffic.
Root cause: healthUrlFor() in app-registry.ts falls back to {stagingWorkerName}.adrper79.workers.dev for staging. Some workers (e.g. admin-studio) are only routed via their custom domain and return 404 on the workers.dev URL.
Fix: Add the optional stagingCustomDomain field to the app’s entry in FACTORY_APPS:
healthUrlFor() prefers stagingCustomDomain over the workers.dev fallback when defined. Only add the field when you’ve confirmed the workers.dev URL actually 404s.
Reference: apps/admin-studio/src/lib/app-registry.ts · PR #914
8. GitHub Actions workflow_dispatch inputs must go through env: before shell use
Symptom: pr-review.mjs flags a workflow with “GitHub Actions script injection risk” — CHANGES_REQUESTED.
Root cause: Directly interpolating ${{ inputs.some_input }} inside a run: shell step allows script injection. An operator with write access could pass ; rm -rf / ; as an input value.
Fix: Always bind inputs to env vars in the step’s env: block, then reference them as $ENV_VAR in the shell:
with: fields (e.g. actions/upload-artifact name: and path:) are YAML context, not shell — they don’t need this treatment.
Reference: .github/workflows/dispatch-capability-provision.yml · PR #910
9. pr-review.mjs wrangler-vars secret check fires as false positive on TypeScript + workflow files
Symptom: Factory canonical reviewer posts CHANGES_REQUESTED — “No secrets in wrangler vars” — but there are no secrets in any wrangler config file in the PR.
Root cause: The check concatenates ALL added lines from ALL files and runs one regex over the blob. TypeScript type definitions (e.g. vars: string[]) introduce a vars: match, and GitHub Actions workflow env blocks (e.g. STUDIO_DISPATCH_TOKEN: ${{ secrets.X }}) introduce a TOKEN: match. The cross-file pattern fires even though no wrangler file was touched with a secret.
Fix (already applied in .github/scripts/pr-review.mjs): The check is now scoped to added lines from wrangler config files only:
vars: block.
Reference: .github/scripts/pr-review.mjs · PR #910
How to add to this doc
When you discover a new operational pattern:- Symptom (what you saw) → root cause (what was actually wrong) → fix (the durable resolution).
- Reference a commit, PR, or code path.
- Keep entries short — under 150 words ideally. If a pattern needs more than that, it belongs in an ADR or RFC.
- Add a new numbered section; don’t renumber existing ones (link stability).