Comet
Experiment tracking, model registry & production monitoring for ML teams
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Unified, high-performance gateway for industrial-grade LLM applications
TensorZero provides a fast, extensible gateway, observability, optimization, evaluation, and experimentation stack for LLMs, supporting dozens of providers, streaming, multimodal, and high-throughput workloads.
TensorZero is a modular stack that lets developers access any major LLM provider through a single, high-performance gateway. Built in Rust, the gateway adds less than 1 ms p99 overhead and can sustain over 10 k queries per second, while supporting streaming, tool use, batch, embeddings, multimodal inputs, and caching.
All inferences and optional feedback are stored in a user‑provided database (e.g., ClickHouse) and can be inspected via the TensorZero UI or programmatically. The platform automatically builds datasets, replays historic calls with new prompts or models, and exports OpenTelemetry traces. Integrated metrics and human‑feedback loops enable prompt, model, and strategy optimization.
TensorZero includes out‑of‑the‑box A/B testing, routing, retries, fallbacks, and granular rate‑limiting. It can be deployed with Docker, accessed via a Python client, patched OpenAI SDKs, or any HTTP client, making it language‑agnostic. Teams can adopt individual components incrementally and combine them with existing tooling.
When teams consider TensorZero, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Real‑time chat assistant with multi‑model fallback
Seamlessly route requests between OpenAI and Anthropic, maintaining sub‑millisecond latency and automatic retries on failures.
Batch embedding generation for recommendation engine
Process millions of texts via the gateway’s batch endpoint, store embeddings in ClickHouse, and monitor throughput via the UI.
A/B testing new prompt designs
Deploy two prompt variants, collect user feedback, and use built‑in metrics to identify the higher‑performing version.
Debugging and replaying production inferences
Query historical calls from the UI, edit prompts, and re‑run them to evaluate model updates without affecting live traffic.
Add the provider in the TensorZero configuration; any OpenAI‑compatible endpoint can be registered, and many major providers are pre‑supported.
You configure your own database (e.g., ClickHouse) where inferences, metrics, and feedback are persisted.
Yes, you can patch the OpenAI client or point the SDK to the gateway’s base URL to route calls through TensorZero.
Custom rate limits can be defined with granular scopes such as user tags, and the gateway enforces them per request.
Currently TensorZero is self‑hosted via Docker; no managed SaaS offering is provided.
Project at a glance
ActiveLast synced 4 days ago
