Better Stack (Log Management)
Cloud-based log management solution for aggregating, searching, and analyzing application logs at scale
logkit
Unified log and metric collector with web console
logkit is a high‑performance server agent that gathers logs and metrics from files, databases, message queues and network sources, then forwards them to Pandora via an intuitive web UI.
Overview
Overview
logkit is a high‑performance server agent designed for teams that need to centralize logs and metrics from heterogeneous environments. It reads from over a dozen native sources—plain files, CSV, nginx logs, Elasticsearch, MongoDB, MySQL, Microsoft SQL Server, PostgreSQL, Kafka, Redis, TCP/UDP sockets, HTTP POST, custom scripts, and SNMP—then forwards the normalized data to Pandora or any compatible endpoint. The built‑in web console lets users create, edit, and monitor runners in real time, providing instant visibility into pipeline health.
Deployment
The tool ships as pre‑compiled binaries for Linux, macOS, and Windows, a Docker image (wonderflow/logkit:nightly), and a ready‑to‑apply Kubernetes manifest. Configuration is driven by a simple logkit.conf file and per‑runner JSON definitions, which can be managed directly through the UI or by watching a configuration directory. With support for multi‑core processing, debug levels, and self‑log cleanup, logkit scales from a single‑node test setup to large‑scale production clusters.
Highlights
Supports 15+ data sources including files, databases, Kafka, Redis, SNMP
Web console for real‑time monitoring and runner configuration
Cross‑platform binaries, Docker image, and Kubernetes deployment
Highly configurable runners with Go‑based performance
Pros
- Broad source support covering files, databases, message queues, and network protocols
- Intuitive web UI simplifies pipeline creation and health monitoring
- Available as native binaries, Docker image, and K8s manifest for flexible deployment
- Written in Go, offering high throughput and low resource usage
Considerations
- Primary output target is Pandora; other destinations need custom plugins
- Configuration relies on JSON files; complex transformations may require code changes
- No built‑in alerting or dashboarding; external tools are required for alerts
- Community edition may lack dedicated enterprise support channels
Managed products teams compare with
When teams consider logkit, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Fit guide
Great for
- DevOps teams needing unified log and metric collection across diverse systems
- Enterprises with mixed technology stacks that require a single collector agent
- Organizations already using Qiniu Pandora as a data lake or analytics platform
- Teams preferring a self‑hosted, open‑source solution with a web management UI
Not ideal when
- Environments that must ship data to non‑Pandora services without custom development
- Users seeking advanced log parsing DSLs out‑of‑the‑box
- Small scripts‑only projects where a heavyweight agent is unnecessary
- Teams without resources to maintain custom runner configurations
How teams use it
Centralized web server logs
Collect nginx and Apache logs from multiple hosts, normalize them, and ship to Pandora for long‑term analysis.
Database change monitoring
Poll MySQL and PostgreSQL for new rows, transform into metric events, and forward to monitoring dashboards.
Real‑time Kafka stream ingestion
Consume Kafka topics, enrich messages, and deliver to Pandora for downstream processing.
SNMP device metrics collection
Periodically query network devices via SNMP, convert to structured logs, and store in Pandora for capacity planning.
Tech snapshot
Go93%
JavaScript6%
Python1%
CSS1%
HTML1%
Batchfile1%
Frequently asked questions
How do I add a new data source?
Create a runner JSON file in the configured `confs_path` or use the web console’s “Add Runner” wizard, selecting the desired source type and connection details.
Can logkit send data to destinations other than Pandora?
By default it ships to Pandora, but custom output plugins can be developed in Go or the HTTP runner can POST to any endpoint.
What are the system requirements?
A Linux, macOS, or Windows host with at least 2 CPU cores is recommended; binaries run without a separate Go runtime.
How is high availability achieved?
Deploy multiple logkit instances behind a load balancer; each instance reads from the same source configuration and forwards independently.
Is there support for TLS/SSL?
Yes, source and destination connections can be configured with TLS certificates and authentication parameters in the runner definition.
Project at a glance
Dormant- Stars
- 1,364
- Watchers
- 1,364
- Forks
- 262
LicenseApache-2.0
Repo age8 years old
Last commit2 years ago
Primary languageGo
Last synced 4 hours ago