Confident AI
DeepEval-powered LLM evaluation platform to test, benchmark, and safeguard apps
OpenLIT
Unified observability and management platform for LLM applications
OpenLIT streamlines AI development by providing OpenTelemetry-native tracing, cost tracking, prompt versioning, secret vaults, and dashboards for LLMs, vector DBs, and GPUs—all deployable via Docker or Helm.
Overview
Audience
OpenLIT is aimed at engineers and ops teams building production‑grade generative AI services who need end‑to‑end visibility, cost control, and secure management of prompts and API keys.
Capabilities
The platform offers OpenTelemetry‑native SDKs for tracing and metrics, an analytics dashboard that surfaces performance, cost, and exception data, a Prompt Hub for versioned prompt storage, and a Vault for encrypted secret handling. OpenGround lets you experiment with multiple LLM providers side‑by‑side, while built‑in cost tracking supports custom pricing files for fine‑tuned models.
Deployment
Deploy the full stack with a single docker compose up -d command or via Helm on Kubernetes. After installing the SDK (pip install openlit), a one‑line openlit.init() call starts sending telemetry to the OpenTelemetry collector, which stores data in ClickHouse and feeds the UI at localhost:3000.
Highlights
OpenTelemetry‑native SDKs for vendor‑agnostic tracing and metrics
Analytics dashboard with cost, performance, and exception monitoring
Prompt Hub and Vault for secure prompt versioning and API key management
OpenGround for side‑by‑side LLM experimentation
Pros
- Vendor‑neutral observability integrates with existing OpenTelemetry stacks
- Single‑line SDK init accelerates development
- Built‑in cost tracking helps budgeting for custom and fine‑tuned models
- Self‑hostable via Docker or Helm for on‑premise control
Considerations
- Requires running the OpenLIT stack (ClickHouse, collector) adding infrastructure overhead
- Currently limited to Python and TypeScript SDKs
- Dashboard UI may need customization for large teams
- Advanced guardrails and auto‑evaluation are still in roadmap
Managed products teams compare with
When teams consider OpenLIT, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Fit guide
Great for
- Teams building production LLM services that need traceability
- Developers wanting real‑time cost visibility for custom models
- Organizations requiring secure storage of API keys and prompts
- Ops teams already using OpenTelemetry for other services
Not ideal when
- Projects unable to allocate extra infrastructure for ClickHouse
- Users seeking a fully managed SaaS observability solution
- Environments limited to languages beyond Python or TypeScript
- Small scripts where full observability overhead outweighs benefits
How teams use it
Monitor LLM latency and token usage in production
Identify performance bottlenecks and optimize model selection, reducing response times by up to 30%.
Track cost of fine‑tuned models across multiple deployments
Maintain budgets with real‑time cost dashboards and custom pricing files.
Version and share prompts securely across microservices
Ensure consistent prompt behavior and prevent accidental key leaks via the Prompt Hub and Vault.
Compare alternative LLM providers during experimentation
Use OpenGround to run side‑by‑side tests, selecting the best model before committing to production.
Tech snapshot
Python55%
TypeScript31%
Go13%
Shell1%
Dockerfile1%
JavaScript1%
Frequently asked questions
How do I start collecting telemetry?
Install the OpenLIT SDK (pip install openlit or npm package) and call openlit.init() with your OTLP endpoint; traces are sent to the OpenLIT collector.
Do I need to run ClickHouse?
The default Docker compose includes ClickHouse for storing metrics; you can replace it with another OTLP‑compatible backend if preferred.
Can I use OpenLIT with existing OpenTelemetry exporters?
Yes, the SDK follows OpenTelemetry semantic conventions, so you can point the exporter to any OTLP endpoint your stack already uses.
Is secret management compliant with best practices?
API keys and secrets are stored in the Vault component, encrypted at rest and accessed only through the SDK, avoiding hard‑coded credentials.
What is the roadmap for auto‑evaluation?
Future releases will add programmatic evaluation metrics and human‑feedback loops; these are marked as 'Coming Soon' in the roadmap.
Project at a glance
Active- Stars
- 2,155
- Watchers
- 2,155
- Forks
- 232
LicenseApache-2.0
Repo age1 year old
Last commit5 days ago
Primary languagePython
Last synced yesterday