Polling Jobs

How to poll for async job results across all endpoints

Polling Jobs

Most AnakinScraper endpoints process jobs asynchronously. After submitting a request, you receive a jobId and poll a GET endpoint until the job completes.

How it works

Submit a POST request — you receive a jobId with status pending
Poll the corresponding GET endpoint with the jobId
Check the status field — repeat until completed or failed

Product	Submit	Poll
URL Scraper	`POST /v1/url-scraper`	`GET /v1/url-scraper/{id}`
URL Scraper (batch)	`POST /v1/url-scraper/batch`	`GET /v1/url-scraper/{id}`
Web Scraper	`POST /v1/web-scraper`	`GET /v1/web-scraper/{id}`
Agentic Search	`POST /v1/agentic-search`	`GET /v1/agentic-search/{id}`

Search API (POST /v1/search) is synchronous — results are returned immediately, no polling needed.

Recommended polling interval

Product	Interval	Typical completion
URL Scraper	2–5 seconds	3–15 seconds
Web Scraper	2–5 seconds	3–10 seconds
Agentic Search	10 seconds	1–5 minutes

Polling examples

import requests
import time

def poll_job(endpoint, job_id, api_key, interval=5, timeout=300):
    """Poll a job until completed or failed."""
    elapsed = 0
    while elapsed < timeout:
        result = requests.get(
            f'https://api.anakin.io/v1/{endpoint}/{job_id}',
            headers={'X-API-Key': api_key}
        )
        data = result.json()

        if data['status'] == 'completed':
            return data
        if data['status'] == 'failed':
            raise Exception(data.get('error', 'Job failed'))

        time.sleep(interval)
        elapsed += interval

    raise TimeoutError('Job polling timed out')

# URL Scraper
result = poll_job('url-scraper', 'job_abc123xyz', 'your_api_key', interval=3)
print(result['markdown'])

# Agentic Search (longer interval)
result = poll_job('agentic-search', 'agentic_abc123xyz', 'your_api_key', interval=10, timeout=600)
print(result['markdown'])

async function pollJob(endpoint, jobId, apiKey, interval = 5000, timeout = 300000) {
  const start = Date.now();

  while (Date.now() - start < timeout) {
    const res = await fetch(`https://api.anakin.io/v1/${endpoint}/${jobId}`, {
      headers: { 'X-API-Key': apiKey }
    });
    const data = await res.json();

    if (data.status === 'completed') return data;
    if (data.status === 'failed') throw new Error(data.error || 'Job failed');

    await new Promise(r => setTimeout(r, interval));
  }

  throw new Error('Job polling timed out');
}

// URL Scraper
const result = await pollJob('url-scraper', 'job_abc123xyz', 'your_api_key', 3000);
console.log(result.markdown);

// Agentic Search (longer interval)
const report = await pollJob('agentic-search', 'agentic_abc123xyz', 'your_api_key', 10000, 600000);
console.log(report.markdown);

Status values

Status	Description
`pending`	Job is queued, not yet started
`queued`	Job is waiting for a worker (agentic search only)
`processing`	Job is actively being processed
`completed`	Results are ready — stop polling
`failed`	Job encountered an error — stop polling