Crawl

Crawl API — scrape multiple pages from a website with pattern-based filtering

The Crawl API scrapes multiple pages from a website. It first discovers URLs (like Map), then scrapes each page and returns the content. Use it when you need the actual content from multiple pages, not just the URLs.

Features

• Multi-page scraping — crawl up to 100 pages in a single job
• Pattern filtering — include/exclude URLs by glob patterns
• Browser rendering — use headless Chrome for JS-heavy sites
• Per-page results — get HTML and markdown for each crawled page
• Proxy routing — route through 207 countries

Endpoints