Skip to main content

Envoy AI Gateway v1.1.x

Token counting, per-request credentials, stream idle timeout, MCP hostname routing, OpenTelemetry GenAI tracing, and HTTP CONNECT egress.

v1.1.0

August 21, 2026
Tokenize APICount TokensPer-Request CredentialsStream Idle TimeoutMCP HostnamesOTel GenAI TracingForward ProxyHelm Hardening
Envoy AI Gateway v1.1.0 is the first minor release on the stable 1.x API. It adds token counting across providers, per-request upstream credentials, stream idle timeout with failover, MCP hostname routing and CEL backend selection, optional OpenTelemetry GenAI tracing, and HTTP CONNECT egress. Upgrading from v1.0 needs no CRD migrations; the only operator-facing change is Helm's restricted controller security context.

✨ New Features​

Token Counting APIs

vLLM-compatible /tokenize across providers

Count tokens for a prompt without generating a completion. Chat messages and completion prompts use the vLLM tokenize request shape against vLLM, Vertex AI Gemini, GCP Anthropic, AWS Bedrock Converse, and AWS Anthropic. Native Anthropic clients can use /anthropic/v1/messages/count_tokens; Responses API clients can use /v1/responses/input_tokens (OpenAI and Azure OpenAI).

Anthropic-format /anthropic/v1/models

Claude clients that call Anthropic's model list endpoint now get the gateway's configured catalog in Anthropic's response shape, scoped by hostname the same way /v1/models is.

Per-Request Credentials & Egress

Per-request upstream credentials via credentialOverride

A trusted filter can supply the backend credential per request instead of sharing one static key in BackendSecurityPolicy. Source from Envoy dynamic metadata (preferred) or a request header the gateway strips before the backend. API keys, Anthropic, Azure, GCP, and AWS SigV4 are supported.

HTTP CONNECT forward proxy

GatewayConfig.spec.forwardProxy tunnels upstream LLM connections through an HTTP CONNECT proxy (host:port). Bedrock requests through PrivateLink now sign over the actual upstream host, so VPC endpoints no longer fail with a silent 403.

Streaming Resilience

Stream idle timeout with failover

AIGatewayRouteRule.streamIdleTimeout bounds how long Envoy waits without upstream bytes on a streaming response. If it fires before the first token, a retry policy can fail over to the next backend; mid-stream it returns 504. Pair with a BackendTrafficPolicy whose retry covers reset.

MCP Gateway

Hostname routing and backend selection

MCPRoute.spec.hostnames scopes an MCP endpoint to specific hosts. backendSelector restricts which backends a session fans out to via CEL rules evaluated at initialize (default Deny). Initialize now advertises merged backend capabilities, and the controller shares one Envoy Gateway Backend per namespace.

Observability

OpenTelemetry GenAI conventions and Grafana dashboard

Set AI_GATEWAY_TRACING_SEMCONV=gen_ai on the ext-proc to emit gen_ai.* span attributes. An example dashboard for gen_ai_* Prometheus metrics ships in examples/monitoring/grafana-dashboard.json. Controller and ext-proc logs accept --logFormat=json; MCP resource reads populate mcp_resource_uri in access-log metadata.

Provider Translation & API Compatibility

Structured output for Claude on Vertex AI

JSON-schema constrained decoding now works for Claude models on Vertex AI that advertise output_config. OpenAI reasoning_effort is forwarded to Bedrock as reasoning_config.

Responses API tool_search, unknown tools, and Codex inputs

The Responses API accepts the built-in tool_search tool and additional_tools, preserves unknown tool types, and accepts Codex-style agent input items.

Helm & Operations

Controller hardening and sharded config secrets

The chart can emit a PDB, honour topologySpreadConstraints, and attach extra podLabels. Filter configuration is split across multiple Secrets so large Gateway configs no longer hit Kubernetes' 1 MiB Secret size limit.

🔗 API Updates​

  • AIGatewayRouteRule.name: Optional unique name copied onto the generated HTTPRoute rule. route-not-found is reserved.
  • AIGatewayRouteRule.streamIdleTimeout: Optional Gateway API duration; unset means no per-try idle timeout.
  • BackendSecurityPolicy.spec.credentialOverride: Optional per-request credential source. Exactly one of fromRequestHeaders or fromDynamicMetadata. fallbackToConfigured defaults to true.
  • GatewayConfig.spec.forwardProxy: Optional HTTP CONNECT proxy. address is required (host:port).
  • MCPRoute.spec.hostnames: Optional list of hostnames (max 16).
  • MCPRoute.spec.backendSelector: Optional CEL selector evaluated at session initialize. defaultAction defaults to Deny.

⚠️ Breaking Changes​

  • Helm controller security context defaults: The Helm chart now sets a restricted podSecurityContext and container securityContext on the controller (non-root UID/GID 65532, all capabilities dropped, no privilege escalation, RuntimeDefault seccomp). Previously both were empty. If you run a custom controller image that cannot use UID 65532, override controller.podSecurityContext and controller.securityContext in your values.

🐛 Bug Fixes​

  • Stale Content-Encoding on translated error responses: Translated error bodies no longer keep the upstream Content-Encoding header. Clients no longer fail to decode an uncompressed error payload that still claimed to be gzip or brotli.
  • Anthropic ping events keep streams alive: Anthropic ping SSE events now emit an empty chunk on translated paths, so intermediaries and clients that time out on silence no longer drop a healthy Claude stream.
  • Gateway local replies are not reprocessed as upstream responses: Ext-proc no longer treats Envoy local replies (rate-limit, timeout, auth) as provider payloads. Local error bodies pass through instead of being force-translated into a malformed LLM response.
  • Anthropic cache_control on the Bedrock path: Prompt-caching cache_control markers now translate onto AWS Bedrock, including tool-result cache control and Bedrock tool cache points as their own tools[] element. Cached prefixes that previously silently dropped now actually hit Bedrock's cache.
  • Bedrock reasoning_content serialized as a plain string: Non-streaming and streaming Bedrock reasoning content is emitted as a plain string, with signatures carried via thinking_blocks. Clients that expected a string and previously received an object no longer fail to parse Claude-on-Bedrock reasoning.
  • Unsupported Anthropic tool types error instead of dropping: Unknown Anthropic tool types now fail the request rather than being silently stripped. You get a clear error instead of a tool-less prompt that looks like it succeeded.
  • Cross-namespace InferencePool backendRef honours namespace: An InferencePool referenced from another namespace is now resolved in that namespace instead of the route's. Cross-namespace inference pools that previously 404'd at reconcile now work when the ReferenceGrant allows them.
  • Anthropic streaming cache tokens no longer double-counted: Input and cache token counts on Anthropic message_delta events are merged once, including on the OpenAI→Anthropic streaming path. Usage metrics and access logs no longer inflate cached-token cost on streaming Claude calls.
  • Credential leak in Envoy ext-proc debug logs: Envoy HTTP and ext-proc debug logging is capped, and credential header mutations are always redacted in debug output. Passthrough endpoints also redact user content. Debug-level logging is no longer a credential-exfiltration path.
  • Gemini streaming tool-call finish reasons and reasoning tokens: Newer Gemini models such as gemini-3.5-flash now map streaming tool-call finish reasons correctly, and reasoning tokens are counted in streaming metrics. Streaming Gemini tool use no longer reports a wrong finish reason or drops thinking-token usage.

📖 Upgrade Guidance​

Upgrading from v1.0 requires no CRD migrations. New fields are optional and backward compatible. Review the Helm security-context change if you override the controller image.

If the official controller image works for you, no action is needed. To restore empty security contexts:

controller:
podSecurityContext: {}
securityContext: {}

JSON logs: set controller.logFormat and extProc.logFormat to json. Enable a PDB with controller.podDisruptionBudget.enabled: true when you run more than one replica.

📦 Dependencies Versions​

Go 1.26.4

Continues on Go 1.26.4, same as v1.0.

Envoy Gateway v1.8.1

Built on Envoy Gateway v1.8.1 for proven data plane capabilities.

Envoy v1.38

Leveraging Envoy Proxy v1.38.1 for battle-tested networking.

Gateway API v1.5.1

Support for Gateway API v1.5.1 specifications.

Gateway API Inference Extension v1.0.2

Continued integration with Gateway API Inference Extension v1.0.2 for intelligent endpoint selection.

MCP Go SDK v1.7.0

Updated from v1.6.1 to v1.7.0 for the latest Model Context Protocol features.

⏩ Patch Releases​

🙏 Acknowledgements​

Thank you to everyone who made v1.1 possible

🔮 What's Next​

The roadmap is community-driven — join us and help shape it.