Envoy AI Gateway v1.1.x
v1.1.0
✨ New Features
Token Counting APIs
/tokenize across providersCount tokens for a prompt without generating a completion. Chat messages and completion prompts use the vLLM tokenize request shape against vLLM, Vertex AI Gemini, GCP Anthropic, AWS Bedrock Converse, and AWS Anthropic. Native Anthropic clients can use /anthropic/v1/messages/count_tokens; Responses API clients can use /v1/responses/input_tokens (OpenAI and Azure OpenAI).
/anthropic/v1/modelsClaude clients that call Anthropic's model list endpoint now get the gateway's configured catalog in Anthropic's response shape, scoped by hostname the same way /v1/models is.
Per-Request Credentials & Egress
credentialOverrideA trusted filter can supply the backend credential per request instead of sharing one static key in BackendSecurityPolicy. Source from Envoy dynamic metadata (preferred) or a request header the gateway strips before the backend. API keys, Anthropic, Azure, GCP, and AWS SigV4 are supported.
GatewayConfig.spec.forwardProxy tunnels upstream LLM connections through an HTTP CONNECT proxy (host:port). Bedrock requests through PrivateLink now sign over the actual upstream host, so VPC endpoints no longer fail with a silent 403.
Streaming Resilience
AIGatewayRouteRule.streamIdleTimeout bounds how long Envoy waits without upstream bytes on a streaming response. If it fires before the first token, a retry policy can fail over to the next backend; mid-stream it returns 504. Pair with a BackendTrafficPolicy whose retry covers reset.
MCP Gateway
MCPRoute.spec.hostnames scopes an MCP endpoint to specific hosts. backendSelector restricts which backends a session fans out to via CEL rules evaluated at initialize (default Deny). Initialize now advertises merged backend capabilities, and the controller shares one Envoy Gateway Backend per namespace.
Observability
Set AI_GATEWAY_TRACING_SEMCONV=gen_ai on the ext-proc to emit gen_ai.* span attributes. An example dashboard for gen_ai_* Prometheus metrics ships in examples/monitoring/grafana-dashboard.json. Controller and ext-proc logs accept --logFormat=json; MCP resource reads populate mcp_resource_uri in access-log metadata.
Provider Translation & API Compatibility
JSON-schema constrained decoding now works for Claude models on Vertex AI that advertise output_config. OpenAI reasoning_effort is forwarded to Bedrock as reasoning_config.
tool_search, unknown tools, and Codex inputsThe Responses API accepts the built-in tool_search tool and additional_tools, preserves unknown tool types, and accepts Codex-style agent input items.
Helm & Operations
The chart can emit a PDB, honour topologySpreadConstraints, and attach extra podLabels. Filter configuration is split across multiple Secrets so large Gateway configs no longer hit Kubernetes' 1 MiB Secret size limit.
🔗 API Updates
AIGatewayRouteRule.name: Optional unique name copied onto the generatedHTTPRouterule.route-not-foundis reserved.AIGatewayRouteRule.streamIdleTimeout: Optional Gateway API duration; unset means no per-try idle timeout.BackendSecurityPolicy.spec.credentialOverride: Optional per-request credential source. Exactly one offromRequestHeadersorfromDynamicMetadata.fallbackToConfigureddefaults totrue.GatewayConfig.spec.forwardProxy: Optional HTTP CONNECT proxy.addressis required (host:port).MCPRoute.spec.hostnames: Optional list of hostnames (max 16).MCPRoute.spec.backendSelector: Optional CEL selector evaluated at session initialize.defaultActiondefaults to Deny.
⚠️ Breaking Changes
- Helm controller security context defaults: The Helm chart now sets a restricted
podSecurityContextand containersecurityContexton the controller (non-root UID/GID 65532, all capabilities dropped, no privilege escalation, RuntimeDefault seccomp). Previously both were empty. If you run a custom controller image that cannot use UID 65532, overridecontroller.podSecurityContextandcontroller.securityContextin your values.
🐛 Bug Fixes
- Stale
Content-Encodingon translated error responses: Translated error bodies no longer keep the upstreamContent-Encodingheader. Clients no longer fail to decode an uncompressed error payload that still claimed to be gzip or brotli. - Anthropic ping events keep streams alive: Anthropic
pingSSE events now emit an empty chunk on translated paths, so intermediaries and clients that time out on silence no longer drop a healthy Claude stream. - Gateway local replies are not reprocessed as upstream responses: Ext-proc no longer treats Envoy local replies (rate-limit, timeout, auth) as provider payloads. Local error bodies pass through instead of being force-translated into a malformed LLM response.
- Anthropic
cache_controlon the Bedrock path: Prompt-cachingcache_controlmarkers now translate onto AWS Bedrock, including tool-result cache control and Bedrock tool cache points as their owntools[]element. Cached prefixes that previously silently dropped now actually hit Bedrock's cache. - Bedrock
reasoning_contentserialized as a plain string: Non-streaming and streaming Bedrock reasoning content is emitted as a plain string, with signatures carried viathinking_blocks. Clients that expected a string and previously received an object no longer fail to parse Claude-on-Bedrock reasoning. - Unsupported Anthropic tool types error instead of dropping: Unknown Anthropic tool types now fail the request rather than being silently stripped. You get a clear error instead of a tool-less prompt that looks like it succeeded.
- Cross-namespace
InferencePoolbackendRefhonours namespace: AnInferencePoolreferenced from another namespace is now resolved in that namespace instead of the route's. Cross-namespace inference pools that previously 404'd at reconcile now work when the ReferenceGrant allows them. - Anthropic streaming cache tokens no longer double-counted: Input and cache token counts on Anthropic
message_deltaevents are merged once, including on the OpenAI→Anthropic streaming path. Usage metrics and access logs no longer inflate cached-token cost on streaming Claude calls. - Credential leak in Envoy ext-proc debug logs: Envoy HTTP and ext-proc debug logging is capped, and credential header mutations are always redacted in debug output. Passthrough endpoints also redact user content. Debug-level logging is no longer a credential-exfiltration path.
- Gemini streaming tool-call finish reasons and reasoning tokens: Newer Gemini models such as gemini-3.5-flash now map streaming tool-call finish reasons correctly, and reasoning tokens are counted in streaming metrics. Streaming Gemini tool use no longer reports a wrong finish reason or drops thinking-token usage.
📖 Upgrade Guidance
Upgrading from v1.0 requires no CRD migrations. New fields are optional and backward compatible. Review the Helm security-context change if you override the controller image.
If the official controller image works for you, no action is needed. To restore empty security contexts:
controller:
podSecurityContext: {}
securityContext: {}
JSON logs: set controller.logFormat and extProc.logFormat to json. Enable a PDB with controller.podDisruptionBudget.enabled: true when you run more than one replica.
📦 Dependencies Versions
Continues on Go 1.26.4, same as v1.0.
Built on Envoy Gateway v1.8.1 for proven data plane capabilities.
Leveraging Envoy Proxy v1.38.1 for battle-tested networking.
Support for Gateway API v1.5.1 specifications.
Continued integration with Gateway API Inference Extension v1.0.2 for intelligent endpoint selection.
Updated from v1.6.1 to v1.7.0 for the latest Model Context Protocol features.
⏩ Patch Releases
🙏 Acknowledgements
Thank you to everyone who made v1.1 possible
🔮 What's Next
The roadmap is community-driven — join us and help shape it.