URLs longer than 120 characters are harder for users to read, share, and remember. Long URLs often signal poor site architecture - deep nesting, unnecessary parameters, or overly descriptive slugs. Search engines can handle long URLs, but shorter, cleaner URLs tend to perform better in rankings and get more clicks in search results.

This guide shows you how to find every long URL on your website using the crawler.sh CLI.

Step 1: Install crawler.sh CLI

Install the CLI with a single command:

curl -fsSL https://install.crawler.sh | sh

This downloads the correct binary for your operating system and architecture, places it in ~/.crawler/bin/, and adds it to your PATH. Restart your terminal or run source ~/.bashrc (or ~/.zshrc) to pick up the new PATH entry.

Verify the installation:

crawler --version

Step 2: Crawl the target website

Run a full crawl of the website you want to audit:

crawler crawl https://example.com

The crawler records the full URL for every page it discovers. Results are saved as an NDJSON file (.crawl) in the current directory. For larger sites:

crawler crawl https://example.com --max-pages 5000

Step 3: Run SEO audit

Run the SEO analysis on your crawl data:

crawler seo example-com.crawl

The long URLs check flags any URL that exceeds 120 characters.

Step 4: Identify long URLs

Look for the Long URLs section in the SEO report. Common patterns that produce long URLs:

Deep directory nesting (/category/subcategory/sub-subcategory/page/)
Query parameters and session IDs appended to URLs
Excessively descriptive slugs that mirror the full page title
CMS-generated paths that include dates, categories, and tags
URL-encoded characters that inflate the length

Step 5: Fix and re-crawl

Shorten long URLs by:

Flattening the directory structure - reduce unnecessary nesting levels
Shortening slugs - use 3–5 keywords instead of the full page title
Removing unnecessary parameters - clean up tracking and session parameters
Setting up 301 redirects from old URLs to the new shorter versions

After fixing, re-crawl to verify:

crawler crawl https://example.com
crawler seo example-com.crawl

Why long URLs matter for SEO

Short, descriptive URLs are easier for users to understand at a glance, more likely to be shared on social media, and more likely to be clicked in search results. Google has stated that URL length is a minor ranking factor and recommends keeping URLs simple. Cleaning up long URLs also often reveals and fixes underlying site architecture issues.