OSVauco/docs/runbooks/opax-live-deploy.md

# OPAX Live Deployment Runbook

## Purpose

Use these scripts to deploy the two production components required for the
Emma/OPAX experience:

- OPAX MCP: the conversational MCP backend and `run_emma` tool service.
- OPAX Web: the live web application served at `https://opax.vauco.no`.

## Fixed Production Targets

| Item | Value |
|---|---|
| Google Cloud project | `propane-will-491900-m5` |
| Region | `us-central1` |
| Artifact Registry repository | `us-central1-docker.pkg.dev/propane-will-491900-m5/osvauco-repo` |
| MCP Cloud Run service | `opax-mcp` |
| OPAX Web Cloud Run service | `opax-web` |
| Live operator URL | `https://opax.vauco.no` |

## Normal Release Sequence

Run commands from the repository root:

```bash
./scripts/deploy-mcp.sh
./scripts/deploy-opax-web.sh
./scripts/check-live-services.sh
```

The scripts build an image, resolve its immutable digest, and deploy that digest
to the existing Cloud Run service.

## MCP Deployment Behavior

### OPAX-MCP Staged Deployment Workflow (Observed 2026-09-23)

During the 2026-09-23 rollout, the service traffic configuration was explicitly
pinned to a revision. Untagged zero-traffic candidate revisions were observed to
retire before a later promotion decision. A candidate created with
`--tag candidate-f003908 --no-traffic` remained active, addressable, was
health-checked, and was explicitly promoted. These are observations from this
rollout, not general Cloud Run guarantees.

#### Stage A: Create and verify a tagged candidate

1. Build and push an immutable image without deploying Cloud Run. The
   `cloudbuild.build-only.yaml` configuration was used for this purpose in the
   observed rollout.

2. Create a tagged, no-traffic candidate revision:

   ```bash
   gcloud run deploy opax-mcp \
     --image <immutable-image-digest> \
     --region <region> \
     --tag candidate-<short-commit> \
     --no-traffic
   ```

3. Verify the candidate uses the expected image digest, is `Ready` and `Active`,
   has a revision tag/direct URL, and has 0% traffic while the current
   production revision retains 100%.

4. Health-check only the tagged candidate URL:

   ```bash
   curl --silent --show-error --fail \
     --max-time 10 \
     --connect-timeout 5 \
     --max-redirs 0 \
     --request GET \
     https://<candidate-tag-url>/health
   ```

   An empty result from a narrow Cloud Logging query means no data was returned
   for that query; it does not prove the absence of application errors.

#### Stage B: Promote traffic

Promotion is a separate, explicit traffic action after Stage A verification:

```bash
gcloud run services update-traffic opax-mcp \
  --to-revisions=<candidate-revision>=100 \
  --region <region>
```

#### Rollback

Rollback is also an explicit traffic action:

```bash
gcloud run services update-traffic opax-mcp \
  --to-revisions=<last-known-good-revision>=100 \
  --region <region>
```

#### Observed rollout record — 2026-09-23

- **Source commit:** `f0039088c2505911c73e6a7374aa105415378277`
- **Cloud Build ID:** `5b934520-8f8c-4a8e-89f3-f2f8377c2879`
- **Immutable image:** `us-central1-docker.pkg.dev/propane-will-491900-m5/osvauco-repo/opax-mcp@sha256:1378a7b8cd81f7f82b52dafb29c0c9ee2b2d37800b27937fbdd353618f9319bf`
- **Candidate tag/revision:** `candidate-f003908` / `opax-mcp-00218-duk`
- **Rollback revision:** `opax-mcp-00214-nar`
- **Health result:** HTTP 200 with `{"status":"ok","service":"opax-mcp","version":"3.6.0"}`
- **Status:** Observed promoted successfully on 2026-09-23.

## Required OPAX-MCP Runtime Contract

- `opax-mcp` uses Direct VPC egress to reach the internal Emma/Ollama runtime.
- Network: `default`
- Subnetwork: `default`
- Egress: `private-ranges-only`
- Ollama endpoint: `http://10.128.0.15:11434`
- Use `./scripts/deploy-mcp.sh` for MCP releases so this contract is explicitly reapplied.
- Do not use a manual deployment that omits this runtime contract.

The existing `cloudbuild.deploy.yaml` workflow builds and pushes an MCP image.
Its internal Cloud Build deploy step can fail because the Cloud Build service
account is blocked by VPC Service Controls.

`deploy-mcp.sh` handles that condition by resolving the image that was pushed
during the build and then deploying its immutable digest directly through the
authenticated local `gcloud` session.

A Cloud Build failure does not automatically mean the image build failed. The
script stops if it cannot resolve an immutable pushed image digest.

## Live Validation

After deployment:

1. Open `https://opax.vauco.no`.
2. Hard-refresh the browser with `Ctrl+Shift+R`.
3. Log in normally.
4. Confirm that the new Emma workspace UI is visible.
5. In a new conversation, send:

   ```text
   Jeg heter Chris.
   ```

6. In the same conversation, send:

   ```text
   Hva heter jeg?
   ```

7. Confirm that Emma uses the previous message as conversation history.
8. Start a second conversation and ask:

   ```text
   Hva heter jeg?
   ```

9. Confirm that the second conversation does not inherit context from the first
   conversation.

## Guardrails

- Use `us-central1` only.
- Never deploy to `europe-west1`.
- Never deploy a service named `opax`.
- Deploy only the existing `opax-mcp` and `opax-web` services.
- Final Cloud Run deployment must use an immutable image digest.
- Never use a mutable tag for the final deploy.
- Never print or place secret values in scripts, logs, documentation, or Git.
- Do not deploy from Gemini without explicit human approval.
- Do not change IAM, service accounts, secrets, VPC settings, DNS, OAuth, or
  Cloud Run networking as part of a normal application release.

## Rollback

Use one of these existing rollback methods:

1. In Cloud Run, route traffic back to the prior ready revision.
2. Re-run the relevant deployment script after replacing the image digest with a
   previously known good immutable digest.

Use `./scripts/check-live-services.sh` to record the currently active revisions
and images before a release.

---

## Appendix: `git-update.sh` Script

### Hensikt og sikkerhetsmodell

`scripts/git-update.sh` er en sikker wrapper for git-operasjoner mot prosjektets Gitea-repository, designet for å forhindre vanlige feil og håndheve beste praksis for versjonskontroll.

### Kommandoer

Scriptet bruker et subkommando-grensesnitt:

#### **`./scripts/git-update.sh status`**

Viser status for repositoryet. Dette er standardvalget hvis ingen subkommando gis.

#### **`./scripts/git-update.sh commit [--allow-main] -m "<melding>" -- <fil1> [...]`**

Stager en eksplisitt liste filer, validerer, og kjører deretter en interaktiv `commit` og `push` i én operasjon.

- **`--allow-main`**: Valgfritt flagg som må brukes for å commite direkte til `main`-branchen.
- **`-m "<melding>"`**: En obligatorisk commit-melding.
- **`-- <filer>`**: En eller flere filer som skal behandles.

#### **`./scripts/git-update.sh push [--allow-main]`**

Pusher en *allerede opprettet* lokal commit som ennå ikke er lastet opp til Gitea.

### Sikkerhetsgarantier

1.  **Gitea-validering**: Scriptet verifiserer at `origin` peker til prosjektets godkjente Gitea-repository. Det vil nekte å kjøre hvis `origin` er GitHub eller en ukjent URL.
2.  **Ingen Pre-staged Commits**: Scriptet avbryter hvis det finnes filer i "staging area" *før* `git add`-kommandoen kjøres. Dette forhindrer at utilsiktede endringer blir med i en commit.
3.  **Eksplisitt filliste**: Kun filene som listes eksplisitt etter `--` blir lagt til i staging. Scriptet bruker aldri `git add .` eller `git add -A`.
4.  **Whitespace-sjekker**: Før og etter staging kjøres `git diff --check` for å avdekke og stoppe ved whitespace-feil.
5.  **Interaktiv bekreftelse**: Før en commit og push, vises en status over stagede filer, og brukeren må bekrefte med `y`. Hvis brukeren avbryter, forblir de eksplisitt stagede filene i staging area, men ingen commit eller push utføres.
6.  **Beskyttelse av `main`**: Operasjoner (commit/push) mot `main`-branchen er blokkert med mindre det eksplisitte `--allow-main` flagget er brukt.
7.  **Trygg Push**: Bruker `git push origin <current_branch>`. Bruker aldri `git push --force`.

### Eksempler

**Sjekk status:**

```bash
./scripts/git-update.sh status
```

**Commit og push til en feature-branch:**

```bash
./scripts/git-update.sh commit -m "feat: Add new script" -- \
  scripts/new-script.sh \
  docs/new-doc.md
```

**Push en allerede opprettet commit:**

```bash
./scripts/git-update.sh push
```

**Avbrudd på grunn av pre-stagede endringer:**

```text
FEIL: Repositoryet har allerede staged endringer.
Avbryter for å hindre at filer utenfor den eksplisitte fillisten blir committet.
Kjør: git diff --cached --name-status
```