PickYourTechPickYourTech home
crawlergo logo

crawlergo

Headless Chrome crawler that harvests high-quality URLs for security testing

Application Security Testing (SAST/DAST/SCA)

CrawlerGo leverages headless Chrome to render pages, auto‑fill and submit forms, trigger DOM events, de‑duplicate URLs, and export results in JSON or push to scanners.

Overview

Overview

CrawlerGo is a browser‑based crawler that runs Chrome in headless mode to collect URLs with full DOM rendering. It automatically fills and submits forms, triggers JavaScript events, and extracts resources from scripts, comments, robots.txt and common path fuzzing. A built‑in de‑duplication module filters pseudo‑static URLs, delivering a concise, high‑quality request list.

Who It Serves & Deployment

Designed for security researchers, penetration testers, and DevOps teams integrating site mapping into CI pipelines, CrawlerGo can be built from source, run via Docker, or executed directly with a Chromium binary. Results are available in JSON or can be pushed to passive vulnerability scanners through a proxy interface. The tool supports custom headers, request proxies, host binding, and automatic Referer handling, making it adaptable to varied testing environments.

Getting Started

Install Chromium, compile with make build (or use the Docker image), then launch with the -c flag pointing to the Chrome executable. Use -t to set concurrent tabs, -o json for output, and --push-to-proxy to forward findings to downstream scanners.

Highlights

Chrome headless rendering for accurate page analysis
Intelligent form filling and automated submission
Smart URL de‑duplication to reduce noise
Proxy support and direct push to passive scanners

Pros

  • High fidelity crawling using a real browser engine
  • Automatic discovery of hidden endpoints via form handling
  • De‑duplication produces concise, actionable URL lists
  • Flexible output: JSON, Docker, or proxy push

Considerations

  • Requires a compatible Chromium installation
  • Higher resource consumption than lightweight crawlers
  • May need additional system libraries on Linux
  • Configuration can be complex for custom headers or cookies

Managed products teams compare with

When teams consider crawlergo, these hosted platforms usually appear on the same shortlist.

Acunetix logo

Acunetix

Web vulnerability scanner for automated security testing of websites and web apps

AppCheck logo

AppCheck

Automated web application and infrastructure vulnerability scanning platform

Burp Suite logo

Burp Suite

Web application security testing platform

Looking for a hosted option? These are the services engineering teams benchmark against before choosing open source.

Fit guide

Great for

  • Security researchers needing comprehensive URL enumeration
  • Penetration testers integrating crawling with vulnerability scanners
  • CI/CD pipelines that require automated site mapping
  • Teams that want to push crawl results directly to passive scanners

Not ideal when

  • Environments with strict memory or CPU limits
  • Simple static sites where JavaScript rendering is unnecessary
  • Users unable to install or update Chromium
  • Real‑time high‑throughput crawling where speed outweighs depth

How teams use it

Enrich passive vulnerability scanners

Feed high‑quality URL lists directly into scanners for deeper analysis

Automated form discovery

Uncover hidden endpoints by auto‑filling and submitting web forms

Dockerized CI pipeline

Run crawling as a container step to detect new URLs on each build

Proxy‑based internal reconnaissance

Route crawl traffic through a SOCKS5 proxy to test internal services

Tech snapshot

Go99%
Makefile1%
Shell1%

Tags

crawlergoheadlesschromedpchrome-devtoolsheadless-chromevulnerability-scannerblackhatgolangarsenalweb-vulnerability-scannerscrawler

Frequently asked questions

What do I need to run CrawlerGo?

A Chromium/Chrome binary compatible with your OS and the Go toolchain to compile, or use the provided Docker image.

How can I get the crawl results?

Use the `-o json` flag to output JSON, or `--push-to-proxy` to send results to a passive scanner.

Does CrawlerGo avoid headless detection?

Yes, it includes default techniques to bypass common headless‑mode detection checks.

Which platforms are supported?

You can build for the current platform with `make build` or for all platforms with `make build_all`; Docker works on any platform with Docker installed.

What if Chrome reports missing dependencies?

Install the required libraries listed in the Troubleshooting section (e.g., libasound2, libgtk-3-0 on Ubuntu).

Project at a glance

Stable
View repo
Stars
3,016
Watchers
3,016
Forks
501
LicenseGPL-3.0
Repo age6 years old
Last commit10 months ago
Primary languageGo

Last synced 2 days ago