Map (URL Discovery)

Map API — discover all URLs on a website via sitemap parsing and link extraction

The Map API discovers all URLs on a website. It fetches the page, parses sitemaps, and extracts links to build a complete URL inventory. Use it for site audits, content indexing, or as a precursor to Crawl.

Features

• Sitemap + link extraction — combines sitemap.xml parsing with in-page link discovery
• Subdomain support — optionally include links to subdomains
• Search filter — filter discovered URLs by keyword
• Browser rendering — use headless Chrome for JS-rendered pages
• Up to 5,000 URLs per request

Endpoints