OSVauco/docs/runbooks/opax-live-deploy.md

OPAX Live Deployment Runbook

Purpose

Use these scripts to deploy the two production components required for the Emma/OPAX experience:

• OPAX MCP: the conversational MCP backend and run_emma tool service.
• OPAX Web: the live web application served at https://opax.vauco.no.

Fixed Production Targets

Item	Value
Google Cloud project	propane-will-491900-m5
Region	us-central1
Artifact Registry repository	us-central1-docker.pkg.dev/propane-will-491900-m5/osvauco-repo
MCP Cloud Run service	opax-mcp
OPAX Web Cloud Run service	opax-web
Live operator URL	https://opax.vauco.no

Normal Release Sequence

Run commands from the repository root:

./scripts/deploy-mcp.sh
./scripts/deploy-opax-web.sh
./scripts/check-live-services.sh

The scripts build an image, resolve its immutable digest, and deploy that digest to the existing Cloud Run service.

MCP Deployment Behavior

OPAX-MCP Staged Deployment Workflow (Observed 2026-09-23)

During the 2026-09-23 rollout, the service traffic configuration was explicitly pinned to a revision. Untagged zero-traffic candidate revisions were observed to retire before a later promotion decision. A candidate created with --tag candidate-f003908 --no-traffic remained active, addressable, was health-checked, and was explicitly promoted. These are observations from this rollout, not general Cloud Run guarantees.

Stage A: Create and verify a tagged candidate

1. Build and push an immutable image without deploying Cloud Run. The cloudbuild.build-only.yaml configuration was used for this purpose in the observed rollout.

2. Create a tagged, no-traffic candidate revision:

   gcloud run deploy opax-mcp \
     --image <immutable-image-digest> \
     --region <region> \
     --tag candidate-<short-commit> \
     --no-traffic
   

3. Verify the candidate uses the expected image digest, is Ready and Active, has a revision tag/direct URL, and has 0% traffic while the current production revision retains 100%.

4. Health-check only the tagged candidate URL:

   curl --silent --show-error --fail \
     --max-time 10 \
     --connect-timeout 5 \
     --max-redirs 0 \
     --request GET \
     https://<candidate-tag-url>/health
   

   An empty result from a narrow Cloud Logging query means no data was returned for that query; it does not prove the absence of application errors.

Stage B: Promote traffic

Promotion is a separate, explicit traffic action after Stage A verification:

gcloud run services update-traffic opax-mcp \
  --to-revisions=<candidate-revision>=100 \
  --region <region>

Rollback

Rollback is also an explicit traffic action:

gcloud run services update-traffic opax-mcp \
  --to-revisions=<last-known-good-revision>=100 \
  --region <region>

Observed rollout record — 2026-09-23

• Source commit: f0039088c2505911c73e6a7374aa105415378277
• Cloud Build ID: 5b934520-8f8c-4a8e-89f3-f2f8377c2879
• Immutable image: us-central1-docker.pkg.dev/propane-will-491900-m5/osvauco-repo/opax-mcp@sha256:1378a7b8cd81f7f82b52dafb29c0c9ee2b2d37800b27937fbdd353618f9319bf
• Candidate tag/revision: candidate-f003908 / opax-mcp-00218-duk
• Rollback revision: opax-mcp-00214-nar
• Health result: HTTP 200 with {"status":"ok","service":"opax-mcp","version":"3.6.0"}
• Status: Observed promoted successfully on 2026-09-23.

Required OPAX-MCP Runtime Contract

• opax-mcp uses Direct VPC egress to reach the internal Emma/Ollama runtime.
• Network: default
• Subnetwork: default
• Egress: private-ranges-only
• Ollama endpoint: http://10.128.0.15:11434
• Use ./scripts/deploy-mcp.sh for MCP releases so this contract is explicitly reapplied.
• Do not use a manual deployment that omits this runtime contract.

The existing cloudbuild.deploy.yaml workflow builds and pushes an MCP image. Its internal Cloud Build deploy step can fail because the Cloud Build service account is blocked by VPC Service Controls.

deploy-mcp.sh handles that condition by resolving the image that was pushed during the build and then deploying its immutable digest directly through the authenticated local gcloud session.

A Cloud Build failure does not automatically mean the image build failed. The script stops if it cannot resolve an immutable pushed image digest.

Live Validation

After deployment:

1. Open https://opax.vauco.no.

2. Hard-refresh the browser with Ctrl+Shift+R.

3. Log in normally.

4. Confirm that the new Emma workspace UI is visible.

5. In a new conversation, send:

   Jeg heter Chris.
   

6. In the same conversation, send:

   Hva heter jeg?
   

7. Confirm that Emma uses the previous message as conversation history.

8. Start a second conversation and ask:

   Hva heter jeg?
   

9. Confirm that the second conversation does not inherit context from the first conversation.

Guardrails

• Use us-central1 only.
• Never deploy to europe-west1.
• Never deploy a service named opax.
• Deploy only the existing opax-mcp and opax-web services.
• Final Cloud Run deployment must use an immutable image digest.
• Never use a mutable tag for the final deploy.
• Never print or place secret values in scripts, logs, documentation, or Git.
• Do not deploy from Gemini without explicit human approval.
• Do not change IAM, service accounts, secrets, VPC settings, DNS, OAuth, or Cloud Run networking as part of a normal application release.

Rollback

Use one of these existing rollback methods:

1. In Cloud Run, route traffic back to the prior ready revision.
2. Re-run the relevant deployment script after replacing the image digest with a previously known good immutable digest.

Use ./scripts/check-live-services.sh to record the currently active revisions and images before a release.

---

Appendix: git-update.sh Script

Hensikt og sikkerhetsmodell

scripts/git-update.sh er en sikker wrapper for git-operasjoner mot prosjektets Gitea-repository, designet for å forhindre vanlige feil og håndheve beste praksis for versjonskontroll.

Kommandoer

Scriptet bruker et subkommando-grensesnitt:

./scripts/git-update.sh status

Viser status for repositoryet. Dette er standardvalget hvis ingen subkommando gis.

./scripts/git-update.sh commit [--allow-main] -m "<melding>" -- <fil1> [...]

Stager en eksplisitt liste filer, validerer, og kjører deretter en interaktiv commit og push i én operasjon.

• --allow-main: Valgfritt flagg som må brukes for å commite direkte til main-branchen.
• -m "<melding>": En obligatorisk commit-melding.
• -- <filer>: En eller flere filer som skal behandles.

./scripts/git-update.sh push [--allow-main]

Pusher en allerede opprettet lokal commit som ennå ikke er lastet opp til Gitea.

Sikkerhetsgarantier

1. Gitea-validering: Scriptet verifiserer at origin peker til prosjektets godkjente Gitea-repository. Det vil nekte å kjøre hvis origin er GitHub eller en ukjent URL.
2. Ingen Pre-staged Commits: Scriptet avbryter hvis det finnes filer i "staging area" før git add-kommandoen kjøres. Dette forhindrer at utilsiktede endringer blir med i en commit.
3. Eksplisitt filliste: Kun filene som listes eksplisitt etter -- blir lagt til i staging. Scriptet bruker aldri git add . eller git add -A.
4. Whitespace-sjekker: Før og etter staging kjøres git diff --check for å avdekke og stoppe ved whitespace-feil.
5. Interaktiv bekreftelse: Før en commit og push, vises en status over stagede filer, og brukeren må bekrefte med y. Hvis brukeren avbryter, forblir de eksplisitt stagede filene i staging area, men ingen commit eller push utføres.
6. Beskyttelse av main: Operasjoner (commit/push) mot main-branchen er blokkert med mindre det eksplisitte --allow-main flagget er brukt.
7. Trygg Push: Bruker git push origin <current_branch>. Bruker aldri git push --force.

Eksempler

Sjekk status:

./scripts/git-update.sh status

Commit og push til en feature-branch:

./scripts/git-update.sh commit -m "feat: Add new script" -- \
  scripts/new-script.sh \
  docs/new-doc.md

Push en allerede opprettet commit:

./scripts/git-update.sh push

Avbrudd på grunn av pre-stagede endringer:

FEIL: Repositoryet har allerede staged endringer.
Avbryter for å hindre at filer utenfor den eksplisitte fillisten blir committet.
Kjør: git diff --cached --name-status