OSVauco/protocols/sessions/2026-05-24_osvauco_rag_and_operator_experience.md

Engineering Brief: RAG Robustness & Operator Experience

Date: 2026-05-24 Author: Senior Staff Engineer (Gemini CLI) Status: PROPOSED

1. Vision & Roadmap Alignment

OSVauco (OPAX) is evolving from a collection of scripts into a cohesive AI Operating System. The Roadmap (Phase 4-6) and Masterplan emphasize a transition from "getting it up" to "keeping it smooth, secure, and multi-client ready."

The core of this experience is the Operator Cockpit—where Gemini, agents (Jason Vauger), and the TUI bridge converge to manage a complex GCP estate.

The Role of RAG

RAG is the "long-term memory" of OPAX. It shouldn't just be a one-time setup; it must be a living repository of the operator's intent, session logs, and architectural rules.

2. Current RAG State (Baseline)

• What Works:
  • Serverless corpus creation via REST (bypassing SDK bugs).
  • Automatic regional fallback (us-central1 -> europe-west4) to avoid Spanner restrictions.
  • Initial document import from GCS.
• What's Fragile:
  • Synchronization: No automated way to re-sync if new files are added to GCS.
  • Health Checks: No quick way to verify if the corpus is "alive" or if the embedding model is responding.
  • Discovery: The corpus name is saved to Secret Manager but not easily discoverable by other agents without manual environment variable plumbing.

3. Proposed "Heavy Lifts"

Lift 1: Automated RAG Sync & Maintenance Tool

Goal: Create agents/rag/sync_corpus.py (and a corresponding agent tool) that performs incremental imports and health checks.

Lift 2: Operator "Preflight" Dashboard (CLI)

Goal: A new script scripts/preflight.sh that provides a high-signal summary for the operator at the start of a session.

Lift 3: Jason Vauger "Cost-Lens" Tooling

Goal: Enhance the core agent logic to support structured cost queries.

Lift 4: Multi-Region Infrastructure Guard

Goal: Refactor infrastructure/*.sh to be fully region-agnostic.

---

Implementation Log

Heavy Lift 1: Automated RAG Sync & Maintenance Tool

Status: COMPLETED Date: 2026-05-24

• Changes:
  • Created agents/rag/sync_corpus.py.
  • Implemented multi-location discovery logic (us-central1 and europe-west4).
  • Added file listing functionality to report on corpus content.
  • Integrated incremental sync from GCS via rag.import_files.
  • Added a health-check retrieval query to verify the end-to-end RAG pipeline.
• Why: To ensure the RAG corpus remains a "living" repository that stays in sync with GCS-based documentation and allows operators to verify system health.
• How to run/test:

  # 1. Ensure corpus exists (or create it)
  python3 agents/rag/setup_corpus.py
  
  # 2. Sync and check health
  python3 agents/rag/sync_corpus.py
  

Heavy Lift 2: Operator Preflight Dashboard

Status: COMPLETED Date: 2026-05-24

• Changes:
  • Created scripts/preflight.sh.
  • Implemented Cloud Run service health check.
  • Added RAG corpus discovery with automatic SDK fallback for environments missing gcloud ai rag commands.
  • Integrated an "Idle Cost Risk" audit (VMs, SQL, Memory Bank).
  • Added Secret Manager verification for critical tokens.
• Why: To give the operator immediate situational awareness and prevent "billing surprises" or broken deployments before starting a session.
• How to run/test:

  bash scripts/preflight.sh
  

Heavy Lift 3: Jason Vauger "Cost-Lens" Tooling

Status: COMPLETED Date: 2026-05-24

• Changes:
  • Created scripts/cost-lens.sh.
  • Implemented "Idle Cost Risk" detection for VMs, SQL, Reasoning Engines, and Cloud Run min-instances.
  • Added get_price_hint helper for rough monthly cost projections.
  • Added 7-day cost trend visualization (with instructions for BigQuery billing export).
  • Implemented --heavy mode for structured YAML/JSON output designed for LLM agent ingestion.
• Why: To empower the operator and the OPAX agent (Jason Vauger) with structured, actionable cost intelligence.
• How to run/test:

  # Human-readable report
  bash scripts/cost-lens.sh
  
  # Machine-readable "Heavy Mode"
  bash scripts/cost-lens.sh --heavy
  

Heavy Lift 4: Multi-Region Infrastructure Guard

Status: COMPLETED Date: 2026-05-24

• Changes:
  • Created infrastructure/99-region-guard.sh to centralize regional availability logic.
  • Updated infrastructure/07-rag-setup.sh and infrastructure/08-memorybank-setup.sh to use the guard for automatic fallback to supported regions.
  • Refactored infrastructure/05-cloudrun-deploy.sh to be fully region-agnostic, supporting DEPLOY_REGION overrides for cross-region deployments (e.g., clinical data in europe-north1).
  • Updated infrastructure/01-setupenv.sh to integrate regional logging and validation.
• Why: To ensure the OPAX infrastructure can scale across multiple regions and clients while respecting GCP service availability and compliance requirements (e.g., data residency).
• How to run/test:

  # Normal setup (respects .env REGION)
  bash infrastructure/01-setupenv.sh
  
  # Deployment to a specific region
  DEPLOY_REGION=europe-north1 bash infrastructure/05-cloudrun-deploy.sh
  

• Open Follow-ups:
  • Expand the guard to include VPC Service Controls and other compliance-heavy services.
  • Add automated "region discovery" to suggest the best region based on latency from the operator's current location.

Docs Hardening – 2026-05-24

Status: COMPLETED

• Updated Docs:
  • docs/LEARNINGS.md: Added LEARNING-007 regarding Vertex AI RAG Engine Spanner restrictions and our "graceful degradation" strategy.
  • docs/SECRETS-SETUP.md: Refreshed list of essential secrets (github-token, webhook-url, rag-corpus-name) with updated gcloud commands and service account mappings.
  • docs/IAP-SETUP.md: Clarified current status—OPAX is on Cloud Run, with IAP planned for Phase 5.
• Key Decisions:
  • RAG Unavailable: Formally recognized that RAG Engine is restricted to Spanner mode for this project. The platform now skips RAG operations cleanly rather than failing.
  • Operator Focus: Shifted documentation priority towards new tools (preflight.sh, cost-lens.sh) to empower the operator despite the RAG limitation.
• Open Follow-ups:
  • Update MASTERPLAN.md quick-start section to include bash scripts/preflight.sh as the first step.
  • Document the infrastructure/99-region-guard.sh logic in ARCHITECTURE.md.

---

Senior Staff Engineer (Gemini CLI)