Chris Christiansen
9fcb9c354a
Some checks are pending
Check Python Version Consistency / Check Python Version (push) Waiting to run
85 lines
3.3 KiB
Markdown
85 lines
3.3 KiB
Markdown
# Plan: Secondary RAG-Dedicated Project Migration
|
|
|
|
## 1. Objective
|
|
To bypass the "Spanner mode" restriction currently blocking serverless RAG Engine corpus creation in `propane-will-491900-m5`. We will spin up a clean, dedicated project for RAG experimentation and wire OSVauco to it as a secondary data source.
|
|
|
|
## 2. New Project Setup
|
|
|
|
### Project Selection
|
|
* **Suggested Name:** `vauco-rag-lab-01` or `osvauco-memory-node`.
|
|
* **Organization:** Ensure it is created under the same billing account as the primary project.
|
|
|
|
### Step 1: Bootstrap Command
|
|
```bash
|
|
NEW_RAG_PROJECT="vauco-rag-lab-01"
|
|
BILLING_ACCOUNT=$(gcloud billing projects describe propane-will-491900-m5 --format="value(billingAccountName)")
|
|
|
|
# Create project
|
|
gcloud projects create ${NEW_RAG_PROJECT} --name="OSVauco RAG Lab"
|
|
|
|
# Link billing
|
|
gcloud billing projects link ${NEW_RAG_PROJECT} --billing-account=${BILLING_ACCOUNT}
|
|
```
|
|
|
|
### Step 2: Minimal APIs
|
|
Enable only what is needed for RAG Engine and its data source:
|
|
```bash
|
|
gcloud services enable \
|
|
aiplatform.googleapis.com \
|
|
storage.googleapis.com \
|
|
--project=${NEW_RAG_PROJECT}
|
|
```
|
|
|
|
### Step 3: Cross-Project IAM
|
|
Grant the primary OSVauco Agent (from `propane-will-491900-m5`) access to the new RAG project:
|
|
```bash
|
|
PRIMARY_SA="jason.vauger@propane-will-491900-m5.iam.gserviceaccount.com"
|
|
|
|
# Grant Vertex AI User and Storage Admin on the NEW project
|
|
for ROLE in roles/aiplatform.user roles/storage.objectAdmin; do
|
|
gcloud projects add-iam-policy-binding ${NEW_RAG_PROJECT} \
|
|
--member="serviceAccount:${PRIMARY_SA}" \
|
|
--role="${ROLE}"
|
|
done
|
|
```
|
|
|
|
## 3. RAG Setup & Sync Refactoring
|
|
|
|
We will use variants of our existing scripts, but modified to accept a `--project` override.
|
|
|
|
### `setup_corpus_remote.py`
|
|
A variant of `setup_corpus.py` that:
|
|
1. Targets `NEW_RAG_PROJECT`.
|
|
2. Attempts the same "Serverless" PATCH in `us-central1` or `europe-west4`.
|
|
3. If it succeeds, it saves the resulting corpus name to a shared secret in the **Primary** project.
|
|
|
|
### `sync_corpus_remote.py`
|
|
A variant of `sync_corpus.py` that reads from the Primary project's GCS bucket but writes/indexes to the Remote project's RAG Engine.
|
|
|
|
## 4. Dual-Project Configuration Strategy
|
|
|
|
OSVauco will be updated to handle "Project Context Switching."
|
|
|
|
### Environment Variables
|
|
The following overrides will be added to `.env`:
|
|
* `PROJECT_ID`: `propane-will-491900-m5` (Primary logic/infra)
|
|
* **`RAG_PROJECT_ID`**: `vauco-rag-lab-01` (Knowledge base source)
|
|
* **`RAG_LOCATION`**: `europe-west4` (Region for the remote corpus)
|
|
|
|
### Code Updates
|
|
1. **`agents/rag/*`**: All RAG scripts and agent tools will default to `RAG_PROJECT_ID` (falling back to `PROJECT_ID`) during `vertexai.init()`.
|
|
2. **`scripts/preflight.sh`**: Will check both projects.
|
|
* Infrastructure section -> Primary Project.
|
|
* Knowledge Base section -> Remote Project.
|
|
3. **`scripts/cost-lens.sh`**: Will aggregate costs from both projects if `HEAVY_MODE` is enabled.
|
|
|
|
## 5. Migration Timeline
|
|
|
|
1. **Project Creation**: Create `vauco-rag-lab-01` and link billing.
|
|
2. **IAM Wiring**: Establish the cross-project service account trust.
|
|
3. **Test Run**: Run `setup_corpus.py` targeting the new project to verify if "Spanner mode" restriction is absent.
|
|
4. **Integration**: Update `.env` and `preflight.sh` to recognize the secondary project.
|
|
|
|
---
|
|
*Senior Staff Engineer (Gemini CLI)*
|