Glossary
Web scraping and AI-agent data terms
Plain, one-sentence definitions for the terms that come up when you build data pipelines and AI agents on the web. Each is written to be quoted on its own.
- Web scraping API
- An HTTP API that fetches a web page and returns its content as clean data, so you never run or maintain a scraper yourself.
- Web data API
- An API that returns structured data from the public web, whether the source site has its own API or not.
- Scrape
- A single request that loads one URL and returns its content as Markdown, HTML, JSON, or a screenshot.
- Crawl
- Following links from a starting URL to fetch many pages of a site in one job, with depth and include/exclude rules.
- Map
- Discovering every URL on a website without downloading each page, useful for planning a crawl or an index.
- Markdown extraction
- Converting a rendered page into clean Markdown that drops navigation and ads and keeps the main content.
- Structured extraction
- Pulling specific fields from a page into a defined JSON shape, usually with a schema or an AI model.
- Search API
- A web search endpoint that returns the full content of each result page, not just a title and a link.
- Agentic search
- A multi-step research pipeline that runs several searches, reads the sources, and returns a cited answer instead of a link list.
- Wire action
- A pre-built, parameterized operation against a specific site (read or write) that hides that site's auth, anti-bot, and parsing behind one endpoint.
- Anti-bot
- The systems a website uses to detect and block automated traffic, such as fingerprint checks, challenges, and rate limits.
- Bot mitigation
- Vendor products like Cloudflare, Akamai, and DataDome that sit in front of a site and decide whether a request looks human.
- TLS fingerprinting
- Identifying a client from the low-level details of its TLS handshake, used to tell a real browser from a scripting library.
- Browser fingerprinting
- Identifying a visitor from browser and device signals (fonts, canvas, screen, WebGL) rather than cookies.
- Headless browser
- A real browser engine run without a visible window, used to render JavaScript pages programmatically.
- Stealth browser
- A headless browser tuned to remove the automation signals that anti-bot systems look for.
- CAPTCHA solving
- Automatically clearing challenges such as reCAPTCHA, hCaptcha, or Turnstile so a scrape can continue.
- Proxy
- An intermediary server that forwards your request so the target site sees the proxy's IP address instead of yours.
- Residential proxy
- A proxy that routes through a real consumer IP address, which is harder for a site to flag than a datacenter IP.
- Datacenter proxy
- A proxy hosted in a data center, faster and cheaper than residential but easier for sites to detect and block.
- Proxy rotation
- Automatically switching the outbound IP address between requests to spread load and avoid per-IP rate limits.
- Rate limiting
- A cap on how many requests a client may make in a window, enforced by the target site or by the API you call.
- Web unblocker
- A service that handles proxies, browser rendering, and anti-bot bypass so a request to a hard site simply returns the page.
- JavaScript rendering
- Executing a page's scripts before extraction so content that loads client-side (as in a single-page app) is captured.
- Single-page app (SPA)
- A site that renders its content in the browser with JavaScript, so a plain HTTP fetch returns an empty shell.
- Robots.txt
- A file at a site's root that tells crawlers which paths they may or may not fetch, honored by well-behaved bots.
- Sitemap.xml
- A machine-readable list of a site's URLs that helps search and AI crawlers discover pages.
- llms.txt
- A Markdown file at a site's root that gives language models a concise, curated map of the site.
- Screenshot capture
- Rendering a page and returning an image of it, either the viewport or the full scrollable page.
- Storage state
- A saved bundle of cookies and local storage that lets an automated session resume as a logged-in user.
- Session warming
- Building up cookies and browsing history on a session before a scrape so it looks like an established visitor.
- CDP (Chrome DevTools Protocol)
- The protocol used to drive and inspect a Chromium browser, the transport most browser-automation tools speak.
- Playwright
- An open-source library for driving real browsers, commonly used to render and interact with pages for scraping.
- MCP (Model Context Protocol)
- An open standard that lets AI assistants call external tools and data sources through a uniform interface.
- MCP tool
- A single capability an MCP server exposes to an AI assistant, such as scrape a URL or run a search.
- MCP server
- A process that exposes a set of tools over the Model Context Protocol for an assistant like Claude to call.
- RAG (Retrieval-Augmented Generation)
- Feeding a language model retrieved documents at query time so its answer is grounded in real, current sources.
- Webhook
- A URL you register so the API can POST a result to your server when a long-running job finishes, instead of you polling.
- Credit
- The unit of usage billing for the API; most basic scrapes cost one credit and heavier operations cost more.
- AI visibility (GEO)
- Generative Engine Optimization: measuring and improving whether AI answer engines mention and cite your brand.