# Engineering Brief: RAG Robustness & Operator Experience
**Date:** 2026-05-24
**Author:** Senior Staff Engineer (Gemini CLI)
**Status:** PROPOSED

## 1. Vision & Roadmap Alignment

OSVauco (OPAX) is evolving from a collection of scripts into a cohesive **AI Operating System**. The Roadmap (Phase 4-6) and Masterplan emphasize a transition from "getting it up" to "keeping it smooth, secure, and multi-client ready." 

The core of this experience is the **Operator Cockpit**—where Gemini, agents (Jason Vauger), and the TUI bridge converge to manage a complex GCP estate.

### The Role of RAG
RAG is the "long-term memory" of OPAX. It shouldn't just be a one-time setup; it must be a living repository of the operator's intent, session logs, and architectural rules.

## 2. Current RAG State (Baseline)

*   **What Works:** 
    *   Serverless corpus creation via REST (bypassing SDK bugs).
    *   Automatic regional fallback (`us-central1` -> `europe-west4`) to avoid Spanner restrictions.
    *   Initial document import from GCS.
*   **What's Fragile:**
    *   **Synchronization:** No automated way to re-sync if new files are added to GCS.
    *   **Health Checks:** No quick way to verify if the corpus is "alive" or if the embedding model is responding.
    *   **Discovery:** The corpus name is saved to Secret Manager but not easily discoverable by other agents without manual environment variable plumbing.

## 3. Proposed "Heavy Lifts"

### Lift 1: Automated RAG Sync & Maintenance Tool
**Goal:** Create `agents/rag/sync_corpus.py` (and a corresponding agent tool) that performs incremental imports and health checks.

### Lift 2: Operator "Preflight" Dashboard (CLI)
**Goal:** A new script `scripts/preflight.sh` that provides a high-signal summary for the operator at the start of a session.

### Lift 3: Jason Vauger "Cost-Lens" Tooling
**Goal:** Enhance the core agent logic to support structured cost queries.

### Lift 4: Multi-Region Infrastructure Guard
**Goal:** Refactor `infrastructure/*.sh` to be fully region-agnostic.

---

## Implementation Log

### Heavy Lift 1: Automated RAG Sync & Maintenance Tool
**Status:** COMPLETED
**Date:** 2026-05-24

*   **Changes:**
    *   Created `agents/rag/sync_corpus.py`.
    *   Implemented multi-location discovery logic (`us-central1` and `europe-west4`).
    *   Added file listing functionality to report on corpus content.
    *   Integrated incremental sync from GCS via `rag.import_files`.
    *   Added a health-check retrieval query to verify the end-to-end RAG pipeline.
*   **Why:** To ensure the RAG corpus remains a "living" repository that stays in sync with GCS-based documentation and allows operators to verify system health.
*   **How to run/test:**
    ```bash
    # 1. Ensure corpus exists (or create it)
    python3 agents/rag/setup_corpus.py
    
    # 2. Sync and check health
    python3 agents/rag/sync_corpus.py
    ```

### Heavy Lift 2: Operator Preflight Dashboard
**Status:** COMPLETED
**Date:** 2026-05-24

*   **Changes:**
    *   Created `scripts/preflight.sh`.
    *   Implemented Cloud Run service health check.
    *   Added RAG corpus discovery with automatic SDK fallback for environments missing `gcloud ai rag` commands.
    *   Integrated an "Idle Cost Risk" audit (VMs, SQL, Memory Bank).
    *   Added Secret Manager verification for critical tokens.
*   **Why:** To give the operator immediate situational awareness and prevent "billing surprises" or broken deployments before starting a session.
*   **How to run/test:**
    ```bash
    bash scripts/preflight.sh
    ```

### Heavy Lift 3: Jason Vauger "Cost-Lens" Tooling
**Status:** COMPLETED
**Date:** 2026-05-24

*   **Changes:**
    *   Created `scripts/cost-lens.sh`.
    *   Implemented "Idle Cost Risk" detection for VMs, SQL, Reasoning Engines, and Cloud Run min-instances.
    *   Added `get_price_hint` helper for rough monthly cost projections.
    *   Added 7-day cost trend visualization (with instructions for BigQuery billing export).
    *   Implemented `--heavy` mode for structured YAML/JSON output designed for LLM agent ingestion.
*   **Why:** To empower the operator and the OPAX agent (Jason Vauger) with structured, actionable cost intelligence.
*   **How to run/test:**
    ```bash
    # Human-readable report
    bash scripts/cost-lens.sh
    
    # Machine-readable "Heavy Mode"
    bash scripts/cost-lens.sh --heavy
    ```

### Heavy Lift 4: Multi-Region Infrastructure Guard
**Status:** COMPLETED
**Date:** 2026-05-24

*   **Changes:**
    *   Created `infrastructure/99-region-guard.sh` to centralize regional availability logic.
    *   Updated `infrastructure/07-rag-setup.sh` and `infrastructure/08-memorybank-setup.sh` to use the guard for automatic fallback to supported regions.
    *   Refactored `infrastructure/05-cloudrun-deploy.sh` to be fully region-agnostic, supporting `DEPLOY_REGION` overrides for cross-region deployments (e.g., clinical data in `europe-north1`).
    *   Updated `infrastructure/01-setupenv.sh` to integrate regional logging and validation.
*   **Why:** To ensure the OPAX infrastructure can scale across multiple regions and clients while respecting GCP service availability and compliance requirements (e.g., data residency).
*   **How to run/test:**
    ```bash
    # Normal setup (respects .env REGION)
    bash infrastructure/01-setupenv.sh
    
    # Deployment to a specific region
    DEPLOY_REGION=europe-north1 bash infrastructure/05-cloudrun-deploy.sh
    ```
*   **Open Follow-ups:**
    *   Expand the guard to include VPC Service Controls and other compliance-heavy services.
    *   Add automated "region discovery" to suggest the best region based on latency from the operator's current location.

## Docs Hardening – 2026-05-24
**Status:** COMPLETED

*   **Updated Docs:**
    *   `docs/LEARNINGS.md`: Added **LEARNING-007** regarding Vertex AI RAG Engine Spanner restrictions and our "graceful degradation" strategy.
    *   `docs/SECRETS-SETUP.md`: Refreshed list of essential secrets (`github-token`, `webhook-url`, `rag-corpus-name`) with updated `gcloud` commands and service account mappings.
    *   `docs/IAP-SETUP.md`: Clarified current status—OPAX is on Cloud Run, with IAP planned for Phase 5.
*   **Key Decisions:**
    *   **RAG Unavailable:** Formally recognized that RAG Engine is restricted to Spanner mode for this project. The platform now skips RAG operations cleanly rather than failing.
    *   **Operator Focus:** Shifted documentation priority towards new tools (`preflight.sh`, `cost-lens.sh`) to empower the operator despite the RAG limitation.
*   **Open Follow-ups:**
    *   Update `MASTERPLAN.md` quick-start section to include `bash scripts/preflight.sh` as the first step.
    *   Document the `infrastructure/99-region-guard.sh` logic in `ARCHITECTURE.md`.

---
*Senior Staff Engineer (Gemini CLI)*