Acunetix
Web vulnerability scanner for automated security testing of websites and web apps
Open-Source Projects
Discover top open-source software, updated regularly with real-world adoption signals.
Discover top open-source software, updated regularly with real-world adoption signals.
Headless Chrome crawler that harvests high-quality URLs for security testing
CrawlerGo leverages headless Chrome to render pages, auto‑fill and submit forms, trigger DOM events, de‑duplicate URLs, and export results in JSON or push to scanners.
CrawlerGo is a browser‑based crawler that runs Chrome in headless mode to collect URLs with full DOM rendering. It automatically fills and submits forms, triggers JavaScript events, and extracts resources from scripts, comments, robots.txt and common path fuzzing. A built‑in de‑duplication module filters pseudo‑static URLs, delivering a concise, high‑quality request list.
Designed for security researchers, penetration testers, and DevOps teams integrating site mapping into CI pipelines, CrawlerGo can be built from source, run via Docker, or executed directly with a Chromium binary. Results are available in JSON or can be pushed to passive vulnerability scanners through a proxy interface. The tool supports custom headers, request proxies, host binding, and automatic Referer handling, making it adaptable to varied testing environments.
Install Chromium, compile with make build (or use the Docker image), then launch with the -c flag pointing to the Chrome executable. Use -t to set concurrent tabs, -o json for output, and --push-to-proxy to forward findings to downstream scanners.
When teams consider crawlergo, these hosted platforms usually appear on the same shortlist.
Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.
Enrich passive vulnerability scanners
Feed high‑quality URL lists directly into scanners for deeper analysis
Automated form discovery
Uncover hidden endpoints by auto‑filling and submitting web forms
Dockerized CI pipeline
Run crawling as a container step to detect new URLs on each build
Proxy‑based internal reconnaissance
Route crawl traffic through a SOCKS5 proxy to test internal services
A Chromium/Chrome binary compatible with your OS and the Go toolchain to compile, or use the provided Docker image.
Use the `-o json` flag to output JSON, or `--push-to-proxy` to send results to a passive scanner.
Yes, it includes default techniques to bypass common headless‑mode detection checks.
You can build for the current platform with `make build` or for all platforms with `make build_all`; Docker works on any platform with Docker installed.
Install the required libraries listed in the Troubleshooting section (e.g., libasound2, libgtk-3-0 on Ubuntu).
Project at a glance
StableLast synced 4 days ago
