Web Scraper

Custom high-throughput scrapers for structured data extraction

Web Scraper

The Web Scraper runs custom-built, high-throughput scraper configurations to extract structured data from websites. Unlike the URL Scraper which returns raw HTML/markdown, the Web Scraper returns data in a predefined schema specific to each scraper.

Features

  • Structured output — returns data in a schema defined by the scraper configuration
  • Custom scrapers — purpose-built scrapers for specific sites and data types
  • High throughput — optimized for speed and reliability at scale
  • Async processing — submit jobs and poll for results

Endpoints

POST/v1/web-scraper

Submit Scrape Job — extract structured data using a custom scraper
GET/v1/web-scraper/{id}

Get Scrape Result — retrieve job status and extracted data