Envoy AI Gateway v0.5.x

Introduces a new GatewayConfig custom resource for gateway-scoped configuration. Reference it from a Gateway via the aigateway.envoyproxy.io/gateway-config annotation to configure the external processor container with environment variables, resource requirements, and other Kubernetes container settings. Multiple Gateways can share the same GatewayConfig for consistent configuration.

Added prefix field to VersionedAPISchema for configuring API endpoint prefixes. Useful for routing to backends with OpenAI-compatible APIs that use non-standard prefixes, such as Gemini's /v1beta/openai or Cohere's /compatibility/v1.

Full support for OpenAI's new Responses API endpoint with streaming and non-streaming modes, function calling, MCP tools support, reasoning, multi-turn conversations, and native multimodal capabilities. Includes complete token usage tracking and OpenInference tracing.

Reduce costs and latency by reusing cached system prompts with Bedrock Anthropic models. The gateway handles cache point markers automatically and tracks both cache creation and cache hit tokens separately for accurate billing visibility.

Same cost-saving prompt caching now works with Claude models on GCP Vertex AI. Cache your system prompts and few-shot examples to cut input token costs on repeated requests.

Comprehensive authorization system for MCP routes. Write expressive CEL rules using request attributes (HTTP method, headers, JWT claims, tool names and call arguments), enforce access based on JWT claim values from your identity provider, or delegate to external gRPC/HTTP authorization services. Control which users can access which tools with precision.

MCP clients automatically stay in sync when you update MCPRoutes. The gateway sends notifications/tools/list_changed notifications, so clients refresh their available tools without manual intervention or reconnection.

Run command-line MCP tools (like npx-based servers) without code changes. The aigw CLI proxies stdio-based MCP servers over HTTP, letting you integrate CLI tools directly into your AI workflows.

OAuth well-known endpoints now serve at the MCPRoute path prefix, ensuring MCP clients correctly discover authorization configuration even when multiple routes have different security policies.

Apply BackendSecurityPolicy to InferencePool resources, ensuring consistent authentication across all dynamically-selected inference endpoints in your pool.

Give Gemini models access to real-time information via the google_search tool type. Filter results by domain, set blocking confidence thresholds, and restrict time ranges to ensure responses use current, relevant sources.

Use the same thinking configuration for both Anthropic and Gemini models. Write provider-agnostic code that works with extended thinking features regardless of which model backend processes the request.

Fine-tune Gemini 3 behavior with thinking_level (control reasoning depth) and media_resolution (balance image quality vs. speed). These settings gracefully degrade on older Gemini versions.

When thinking is enabled, the gateway extracts and surfaces thought summaries from Gemini responses, helping you understand how the model reached its conclusions.

Connect Gemini to your organization's search infrastructure with the enterprise_search tool type. Configure custom data sources for grounding responses in enterprise-specific knowledge.

Inject or remove JSON fields in request bodies per-backend without application changes. Use bodyMutation to add service_tier, set custom parameters, or strip internal fields before requests reach different providers. Route-level settings override backend defaults for flexible multi-backend configurations.

Choose between standard, flex, priority, and reserved for Bedrock requests. Set service_tier to prioritize latency-sensitive workloads or optimize costs for batch processing, with automatic fallback handling.

Track spending across AI providers with the new gen_ai.provider.name metric attribute. Filter dashboards and alerts by provider to identify cost drivers and optimize your multi-provider strategy.

Debug Claude model requests end-to-end with OpenInference-compliant tracing for the native /messages endpoint. View prompts, responses, and timing in Arize Phoenix or any OpenTelemetry-compatible observability platform.

Track reranking operations in your traces. Full OpenTelemetry support for Cohere's v2 rerank endpoint captures query, documents, and relevance scores for debugging RAG pipelines.

Reduced gateway latency by migrating to bytedance/sonic for JSON encoding and decoding. Expect measurable latency improvements and lower CPU usage, especially for large payloads and high request volumes.

Reduced controller reconciliation time with optimized ReferenceGrant indexing. Cross-namespace backend references now validate faster, improving startup time and configuration change responsiveness.

MCP proxy now reuses HTTP connections across requests, eliminating per-request connection overhead. This significantly improves throughput when proxying to multiple MCP backend servers. View the MCP performance blog post for details →