Skip to main content

​
Scope

This runbook covers the gcp_backend.yml workflow for prod backend deploys. It exists to prevent wedged Cloud Run services where spec.traffic points at a failed revision while status.traffic still serves an older good revision. Product hotfix code changes are separate from deploy-process fixes. Land deploy hardening on main first; branch product fixes from current main, not stale hotfix forks.

​
Workflows

Dispatch Deploy Backend to Cloud RUN with:

​
Pre-deploy checklist (prod)

Run hermetic checks locally or in CI before dispatch:
For live read-only validation with gcloud auth:
  1. Confirm the last deploy did not wedge traffic. If unsure, run mode: repair-traffic-only first.
  2. Verify runtime env locally:
  3. For secret bindings in backend/deploy/runtime_env.yaml, preflight checks Secret Manager before creating revisions:

​
Deploy flow (mode: deploy)

Prod deploys auto-repair spec.traffic != status.traffic before creating revisions. Traffic shifts only after new revisions are Ready=True.

​
Traffic repair

If a deploy fails after creating a non-ready revision:
  1. Run mode: repair-traffic-only to align spec.traffic with status.traffic.
  2. Inspect the workflow summary Cloud Run status report for the exact serving revision and repair command.
  3. Retry mode: deploy only after repair succeeds and preflight passes.
Manual repair command format:

​
Cloud Run-only hotfixes

Use deploy_targets: cloud-run-only when the change only affects Cloud Run REST services (backend, backend-sync, backend-integration) and does not require:
  • backend-listen Helm chart changes
  • backend-secrets ExternalSecret changes
  • WebSocket / listen pipeline changes
For listen-only chart changes, use gcp_backend_listen_helm.yml instead.

​
Runtime env source of truth

Cloud Run env and secret bindings are declared in backend/deploy/runtime_env.yaml and rendered by backend/scripts/render_backend_runtime_env.py. After changing the manifest, run ./scripts/pre-deploy-check.sh. Prod Cloud Run intentionally omits SERVICE_ACCOUNT_JSON and POSTHOG_PROJECT_API_KEY bindings (#9164): those secrets are absent in prod Secret Manager and the runtime falls back to ADC / disables PostHog telemetry.

​
Dev batch STT topology

The backend, backend-sync, and backend-integration services own their routed pre-recorded STT selection and every provider dependency in backend/deploy/runtime_env.yaml. Dev Parakeet must use the isolated http://parakeet.omiapi.com endpoint—never production parakeet.omi.me; prod uses http://parakeet.omi.me. Serving policy is literal modulate-velma-2,parakeet, and the runtime validator requires both MODULATE_API_KEY and HOSTED_PARAKEET_API_URL for rendered and live Cloud Run revisions. After a dev revision is serving, run the opt-in smoke check with a valid dev authorization header:
It is dry-run by default, rejects production hosts, synthesizes a deterministic spoken fixture on macOS (or accepts --pcm-file elsewhere), and requires the batch endpoint to report a Parakeet transcript. Deploy hardening tracked in GitHub issue #9164.