# GCP Best Practices for Multi-Agent AI Systems
**OSVauco-NMTMD-GCOS | Last Updated: 2026-05**

## 1. Multi-Agent Architecture Patterns

### 1.1 Coordinator + Sub-Agent Pattern (ADK)
Use a root orchestrator agent that delegates to specialized sub-agents.

```python
root_agent = Agent(
    name="orchestrator",
    model="gemini-2.0-flash",
    sub_agents=[greeting_agent, rag_agent, memory_agent, farewell_agent],
    instruction="Delegate tasks to the appropriate sub-agent.",
    before_model_callback=before_model_callback,
    before_tool_callback=before_tool_callback,
)
```

### 1.2 Graph Workflow Pattern
- SequentialAgent: ordered pipelines
- ParallelAgent: concurrent sub-tasks
- LoopAgent: retry/polling patterns

### 1.3 Session State and Memory
- Use session.state for short-lived conversation context
- Use Vertex AI Memory Bank for persistent long-term memory
- Tag memories with user_id and session_id for retrieval scoping

## 2. Model Selection

| Use Case | Recommended Model |
|----------|------------------|
| Complex reasoning / orchestration | gemini-2.0-flash or gemini-2.5-pro |
| Fast sub-agent calls | gemini-2.0-flash |
| Non-Google models | LiteLLM via LiteLlmModel |
| Embedding generation | text-embedding-005 |
| Code generation | gemini-2.5-pro |

## 3. Tool Design

- Single-responsibility per tool function
- Full docstrings — ADK uses these for LLM instructions
- Validate inputs inside tools; never trust LLM output blindly
- Use before_tool_callback for argument guardrails
- Return structured dicts, not raw strings

## 4. Vertex AI Integration

### 4.1 RAG Engine
**CONFIRMED** (source: Vertex AI RAG Engine SDK, google/adk-python samples, 2026-05):
- Use RagCorpus for document grounding
- Embedding model: text-embedding-005
- Chunking: 512 tokens, 50-token overlap
- Retrieval: top_k=5, vector_distance_threshold=0.7

**UNKNOWN / UNDOCUMENTED** (as of 2026-05):
- Full regional availability (eu-west3 issues known)
- Whether rag.import_files() is idempotent for same GCS path
- See: docs/Doc_Gaps_and_Open_Questions.md OQ-02

### 4.2 Memory Bank
**CONFIRMED** (source: Gemini Enterprise Agent Platform Memory Bank setup, 2026-05):
- load_memory_tool + preload_memory_tool from google.adk.tools
- Required role: roles/aiplatform.user
- SDK: google-cloud-aiplatform >= 1.111.0

**UNKNOWN / UNDOCUMENTED** (as of 2026-05):
- gcloud CLI equivalent for client.agent_engines.create()
- Whether tools auto-discover Memory Bank instance via project/region
- See: docs/Doc_Gaps_and_Open_Questions.md OQ-01, OQ-06

### 4.3 Security Controls Matrix

| Component | Data Residency | CMEK | VPC-SC | Access Transparency |
|-----------|---------------|------|--------|-------------------|
| Agent Platform | YES | YES | YES | YES |
| RAG Engine | NO | NO | YES | NO |
| Vector Search | YES | NO | YES | NO |

## 5. Observability

- Cloud Trace: end-to-end request tracing
- Cloud Logging: structured JSON logs with session_id; never log PII
- Cloud Monitoring: latency p50/p95/p99, error rates
- Error Reporting: auto-alert on new exception types

## 6. Deployment

- Cloud Run services (serverless, auto-scaling)
- Artifact Registry for container images
- min-instances=1 for latency-sensitive agents
- Cloud Build for CI/CD; --no-traffic deploy + HITL traffic switch
- Label all resources: env, team, cost-center, agent

## 7. MCP Integration

```python
from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset, StdioServerParameters

tools, exit_stack = await MCPToolset.from_server(
    connection_params=StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
    )
)
```

## 8. A2A Protocol

- Agents expose AgentCard at /.well-known/agent.json
- A2A enables multi-vendor agent interoperability
- Secure A2A calls with service account tokens + VPC-SC

## 9. Handling Missing or Incomplete GCP Documentation

*Implements OSVauco-NMTMD-GCOS section 1.5.*

| Situation | Action |
|-----------|--------|
| API confirmed in GA docs or stable samples | Use it; cite source |
| API only in preview docs / blog post | Mark PREVIEW — validate before prod |
| API not found in any authoritative source | Omit; add to Open Questions |

## References
- ADK: https://google.github.io/adk-docs/
- Vertex AI RAG: https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview
- Gemini Enterprise Agent Platform: https://cloud.google.com/gemini-enterprise-agent-platform
- A2A: https://google.github.io/A2A/