OSVauco/docs/GCP_Best_Practices.md

GCP Best Practices for Multi-Agent AI Systems

OSVauco-NMTMD-GCOS | Last Updated: 2026-05

1. Multi-Agent Architecture Patterns

1.1 Coordinator + Sub-Agent Pattern (ADK)

Use a root orchestrator agent that delegates to specialized sub-agents.

root_agent = Agent(
    name="orchestrator",
    model="gemini-2.0-flash",
    sub_agents=[greeting_agent, rag_agent, memory_agent, farewell_agent],
    instruction="Delegate tasks to the appropriate sub-agent.",
    before_model_callback=before_model_callback,
    before_tool_callback=before_tool_callback,
)

1.2 Graph Workflow Pattern

• SequentialAgent: ordered pipelines
• ParallelAgent: concurrent sub-tasks
• LoopAgent: retry/polling patterns

1.3 Session State and Memory

• Use session.state for short-lived conversation context
• Use Vertex AI Memory Bank for persistent long-term memory
• Tag memories with user_id and session_id for retrieval scoping

2. Model Selection

Use Case	Recommended Model
Complex reasoning / orchestration	gemini-2.0-flash or gemini-2.5-pro
Fast sub-agent calls	gemini-2.0-flash
Non-Google models	LiteLLM via LiteLlmModel
Embedding generation	text-embedding-005
Code generation	gemini-2.5-pro

3. Tool Design

• Single-responsibility per tool function
• Full docstrings — ADK uses these for LLM instructions
• Validate inputs inside tools; never trust LLM output blindly
• Use before_tool_callback for argument guardrails
• Return structured dicts, not raw strings

4. Vertex AI Integration

4.1 RAG Engine

CONFIRMED (source: Vertex AI RAG Engine SDK, google/adk-python samples, 2026-05):

• Use RagCorpus for document grounding
• Embedding model: text-embedding-005
• Chunking: 512 tokens, 50-token overlap
• Retrieval: top_k=5, vector_distance_threshold=0.7

UNKNOWN / UNDOCUMENTED (as of 2026-05):

• Full regional availability (eu-west3 issues known)
• Whether rag.import_files() is idempotent for same GCS path
• See: docs/Doc_Gaps_and_Open_Questions.md OQ-02

4.2 Memory Bank

CONFIRMED (source: Gemini Enterprise Agent Platform Memory Bank setup, 2026-05):

• load_memory_tool + preload_memory_tool from google.adk.tools
• Required role: roles/aiplatform.user
• SDK: google-cloud-aiplatform >= 1.111.0

UNKNOWN / UNDOCUMENTED (as of 2026-05):

• gcloud CLI equivalent for client.agent_engines.create()
• Whether tools auto-discover Memory Bank instance via project/region
• See: docs/Doc_Gaps_and_Open_Questions.md OQ-01, OQ-06

4.3 Security Controls Matrix

Component	Data Residency	CMEK	VPC-SC	Access Transparency
Agent Platform	YES	YES	YES	YES
RAG Engine	NO	NO	YES	NO
Vector Search	YES	NO	YES	NO

5. Observability

• Cloud Trace: end-to-end request tracing
• Cloud Logging: structured JSON logs with session_id; never log PII
• Cloud Monitoring: latency p50/p95/p99, error rates
• Error Reporting: auto-alert on new exception types

6. Deployment

• Cloud Run services (serverless, auto-scaling)
• Artifact Registry for container images
• min-instances=1 for latency-sensitive agents
• Cloud Build for CI/CD; --no-traffic deploy + HITL traffic switch
• Label all resources: env, team, cost-center, agent

7. MCP Integration

from google.adk.tools.mcp_tool.mcp_toolset import MCPToolset, StdioServerParameters

tools, exit_stack = await MCPToolset.from_server(
    connection_params=StdioServerParameters(
        command="npx",
        args=["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
    )
)

8. A2A Protocol

• Agents expose AgentCard at /.well-known/agent.json
• A2A enables multi-vendor agent interoperability
• Secure A2A calls with service account tokens + VPC-SC

9. Handling Missing or Incomplete GCP Documentation

Implements OSVauco-NMTMD-GCOS section 1.5.

Situation	Action
API confirmed in GA docs or stable samples	Use it; cite source
API only in preview docs / blog post	Mark PREVIEW — validate before prod
API not found in any authoritative source	Omit; add to Open Questions

References

• ADK: https://google.github.io/adk-docs/
• Vertex AI RAG: https://cloud.google.com/vertex-ai/generative-ai/docs/rag-overview
• Gemini Enterprise Agent Platform: https://cloud.google.com/gemini-enterprise-agent-platform
• A2A: https://google.github.io/A2A/