Amazon SageMaker
Fully managed machine learning service to build, train, and deploy ML models at scale
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Run any LLM locally behind an OpenAI-compatible API
OpenLLM lets developers serve any open‑source LLM (Llama, Qwen, Phi, etc.) as an OpenAI‑compatible API with a single command, plus a chat UI and Docker/K8s deployment tools.
OpenLLM is designed for developers, data scientists, and enterprises that want to self‑host large language models without building custom inference stacks. With a single openllm serve command you can launch a model such as Llama 3.3, Qwen 2.5, or Phi‑4 and instantly expose OpenAI‑compatible endpoints for chat, completions, and embeddings.
The framework includes a built‑in web chat UI, supports a growing catalog of state‑of‑the‑art models, and integrates with Docker, Kubernetes, and BentoCloud for production‑grade deployments. Model weights are fetched from Hugging Face at runtime, requiring only an HF token for gated models. Once running, the server is reachable at http://localhost:3000 (or any configured host) and can be consumed by any client library that speaks the OpenAI API.
OpenLLM streamlines the workflow from local experimentation to scalable cloud services, letting teams iterate quickly while retaining full control over data and infrastructure.
When teams consider OpenLLM, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Chatbot prototype
Launch a functional chat API in minutes for internal testing or demos.
Internal microservice
Expose a secure, self‑hosted LLM endpoint that integrates with existing backend services.
Model benchmarking
Run multiple models behind the same API to compare latency, cost, and quality.
Educational labs
Provide students with hands‑on experience deploying and querying LLMs without external cloud costs.
OpenLLM ships with dozens of models including Llama 3.1/3.2/3.3, Qwen 2.5, Phi‑4, Mistral, Gemma, and more. Custom model repositories can also be added.
No. Weights are fetched from Hugging Face at runtime. A valid HF_TOKEN is required for gated models.
The API key is optional; you can use a dummy value (`na`) for local testing.
Yes. The project provides Docker images and Helm charts that integrate with standard K8s workflows.
Project at a glance
ActiveLast synced 4 days ago
