Guide
Configuring the crawler
Crawl limits, excluded paths, and auditing password-protected staging sites.
3 min read
Crawl limits by plan
Each plan has a maximum pages-per-audit limit: Free crawls up to 20 pages, Pro up to 50 pages, and Agency up to 80 pages. The crawler always prioritises pages closest to the homepage, so your most important content is audited first. You can set a lower per-project cap in Settings if you want faster audits on large sites.
Exclude high-volume sections to get the most from your limit
Blog posts, news archives, tag pages, author pages, and search result pages add hundreds of URLs but rarely surface unique structural issues. Add patterns like /blog/*, /news/*, /tag/*, /author/*, or /page/* to the excluded paths list so the crawler focuses its budget on your core pages: homepage, service pages, landing pages, and key conversion flows.
Exclude paths you do not need
Add excluded path patterns, like /tag/*, /author/*, or a language folder, to skip sections that would waste the page budget. A pattern ending in /* also matches the base directory itself.
Audit staging sites behind basic auth
Enable Crawler Basic Auth and add the username and password. The crawler authenticates on every request, including screenshots and contrast checks, so protected staging environments audit exactly like production.
Skip translated duplicates
Use the Exclude translations helper to add common language-code paths in one click, so multilingual sites do not burn the entire page budget on duplicate content in other languages.