Eden AI
Unified API aggregator for AI services across providers
Envoy AI Gateway
Unified gateway for secure, scalable generative AI traffic
Envoy AI Gateway routes, authenticates, and rate‑limits traffic to multiple LLM providers, offering a two‑tier architecture for centralized control and fine‑grained model access.
Overview
Overview
Envoy AI Gateway provides a cloud‑native, Envoy‑based solution for routing client requests to a wide range of generative AI services. By handling authentication, top‑level routing, and global rate limiting at a centralized Tier One gateway, it gives operators a single control point for policy enforcement across providers such as OpenAI, Azure OpenAI, Google Gemini, Anthropic, and many others.
Architecture
A second Tier Two gateway sits closer to self‑hosted model clusters, offering fine‑grained access control and an endpoint picker that can direct inference traffic to the most suitable model instance. The two‑tier pattern lets organizations combine SaaS LLM APIs with on‑premise deployments while maintaining consistent security and observability. Deployment follows standard Envoy Gateway practices on Kubernetes, and the project includes quick‑start guides, documentation, and an active CNCF‑aligned community.
Highlights
Two‑tier gateway pattern separating entry‑point and model‑specific routing
Built‑in authentication, global rate limiting, and top‑level routing
Supports 20+ LLM providers out of the box
Endpoint picker for inference optimization on self‑hosted clusters
Pros
- Works with any Envoy Gateway deployment
- Extensive provider support reduces integration effort
- Fine‑grained control via Tier Two gateway
- Community‑driven CNCF project with active Slack channel
Considerations
- Requires familiarity with Envoy configuration
- Complexity of two‑tier setup may be overkill for simple use‑cases
- Limited to Kubernetes environments for self‑hosted models
- Performance depends on underlying Envoy deployment
Managed products teams compare with
When teams consider Envoy AI Gateway, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Fit guide
Great for
- Enterprises needing unified access control across multiple LLM services
- Teams deploying self‑hosted model clusters alongside SaaS providers
- Developers seeking standardized authentication and rate limiting for AI APIs
- Organizations adopting cloud‑native, CNCF‑aligned infrastructure
Not ideal when
- Small projects with a single LLM provider and minimal routing
- Teams without Envoy or Kubernetes expertise
- Use‑cases requiring out‑of‑the‑box UI dashboards
- Scenarios where latency overhead of an extra proxy layer is unacceptable
How teams use it
Enterprise AI platform consolidating OpenAI, Azure, and internal models
Provides a single entry point with unified auth, rate limits, and routing, simplifying policy enforcement.
Multi‑tenant SaaS offering LLM‑powered features
Isolates tenant traffic via Tier Two gateways, enabling per‑tenant model selection and usage quotas.
Edge deployment routing requests to on‑premise LLM cluster
Uses Tier Two gateway to pick optimal inference endpoints, reducing latency and cost.
Rapid prototyping across diverse AI providers
Allows developers to switch providers without code changes, leveraging the same gateway configuration.
Tech snapshot
Go91%
MDX7%
CSS1%
TypeScript1%
Makefile1%
Smarty1%
Frequently asked questions
What languages or frameworks are required to run Envoy AI Gateway?
The gateway runs on Envoy Gateway and is configured via YAML/Envoy resources; the control plane is written in Go but any language can interact through standard HTTP/HTTPS.
Does Envoy AI Gateway provide built‑in authentication mechanisms?
Yes, the Tier One gateway supports authentication plugins (e.g., JWT, OAuth) that can be configured to validate incoming client requests.
Can I add a new LLM provider not listed in the documentation?
Custom providers can be integrated by defining a new upstream cluster and routing rules, leveraging Envoy’s extensible filter architecture.
Is there a managed SaaS version of Envoy AI Gateway?
Currently the project is self‑hosted; no official managed service is offered.
How does the endpoint picker improve inference performance?
It selects the most appropriate model endpoint based on routing criteria such as latency, version, or load, allowing fine‑tuned optimization.
Project at a glance
Active- Stars
- 1,329
- Watchers
- 1,329
- Forks
- 155
LicenseApache-2.0
Repo age1 year old
Last commit17 hours ago
Primary languageGo
Last synced 12 hours ago