Apify
Web automation & scraping platform powered by serverless Actors
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Fast, elegant web scraping framework for Go developers
Colly offers a clean, high‑performance API for building crawlers and scrapers in Go, handling concurrency, delays, cookies, robots.txt, caching, and distributed scraping out of the box.
Colly is a Go library that lets developers create powerful web crawlers and scrapers with minimal boilerplate. Its declarative API lets you define handlers for HTML elements, requests, and responses, while the framework automatically manages request throttling, domain‑level concurrency, and session cookies.
Built for speed, Colly can process over 1,000 requests per second on a single CPU core and supports synchronous, asynchronous, and parallel execution modes. Features like robots.txt compliance, automatic encoding, caching, and distributed scraping make it suitable for large‑scale data collection, archiving, and monitoring tasks. Integration is straightforward: add the module to your go.mod, import the package, and run your scraper on any Go‑compatible environment, from local machines to cloud containers.
Whether you are a solo developer prototyping a data‑mining script or a team building a production‑grade crawling service, Colly provides the performance and flexibility needed without imposing heavy dependencies.
When teams consider Colly, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Website content archiving
Capture and store static snapshots of target sites for preservation
Price monitoring
Continuously scrape e‑commerce pages to detect price changes
Research data mining
Extract structured information from public directories for analysis
SEO competitor analysis
Crawl competitor sites respecting robots.txt to gather link and keyword data
Yes, Colly manages concurrency per domain and is safe for parallel use.
Colly does not execute JavaScript; you need to integrate a headless browser if required.
Yes, its distributed scraping feature enables multi‑node deployments.
Colly is released under the Apache‑2.0 license.
Add `github.com/gocolly/colly/v2` to your `go.mod` and run `go get`.
Project at a glance
ActiveLast synced 4 days ago
