Elixir

Scrape your first page from Elixir using Req — the de facto modern HTTP client, single-dependency, ergonomic.

Submit a scrape, poll for the result, and handle transient errors — using Req, the modern Elixir HTTP client (already bundled with Phoenix 1.7+).

Authentication

Set your API key as an environment variable. Get a key from the Dashboard.

export ANAKIN_API_KEY=ak-your-key-here

The base URL is https://api.anakin.io/v1. Every request authenticates via the x-api-key header.

Install

Req is the de facto modern HTTP client in Elixir — JSON encode/decode, retry, and connection pooling all built in. Phoenix 1.7+ already bundles it; standalone projects need one dep:

Add to mix.exs:

defp deps do
  [
    {:req, "~> 0.5"}
  ]
end

Then:

mix deps.get

Scrape a page

Save as lib/quickstart.ex:

defmodule Quickstart do
  @base "https://api.anakin.io/v1"

  defp api_key do
    System.get_env("ANAKIN_API_KEY") || raise "ANAKIN_API_KEY is not set"
  end

  defp request(method, path, body \\ nil) do
    Req.request(
      method: method,
      url: @base <> path,
      headers: [{"x-api-key", api_key()}, {"content-type", "application/json"}],
      json: body,
      receive_timeout: 30_000
    )
  end

  def scrape(url) do
    {:ok, %{body: submitted}} = request(:post, "/url-scraper", %{url: url})
    job_id = submitted["jobId"]

    Enum.reduce_while(1..60, nil, fn _, _ ->
      case request(:get, "/url-scraper/#{job_id}") do
        {:ok, %{body: %{"status" => "completed"} = job}} ->
          {:halt, job}
        {:ok, %{body: %{"status" => "failed", "error" => err}}} ->
          raise "scrape failed: #{err}"
        _ ->
          Process.sleep(3_000) # retry transient errors
          {:cont, nil}
      end
    end)
    |> case do
      nil -> raise "timed out after 3 minutes"
      job -> job
    end
  end
end

job = Quickstart.scrape("https://example.com")
IO.puts(job["markdown"])

Run it:

mix run lib/quickstart.ex

What this does

1. Submits https://example.com to /url-scraper and gets back a jobId.
2. Polls /url-scraper/{jobId} every 3 seconds (up to 60 attempts = 3 minutes).
3. Retries transient network errors silently — only surfaces real failures.
4. Prints the final markdown when the job completes.

Most jobs finish in 3–15 seconds.

Go further

Extract structured JSON with AI

Replace the submit body with generateJson: true to have AI return structured data:

{:ok, %{body: submitted}} = request(:post, "/url-scraper", %{
  url: "https://news.ycombinator.com",
  generateJson: true
})

The completed response includes a generatedJson field with structured data inferred from the page.

Scrape JavaScript-heavy sites

For SPAs and dynamically-loaded pages, add useBrowser: true:

{:ok, %{body: submitted}} = request(:post, "/url-scraper", %{
  url: "https://example.com/spa",
  useBrowser: true
})

Only use browser mode when needed — standard scraping is faster and cheaper.

Use it from Phoenix

Wrap scrape/1 in a GenServer or schedule via Oban — the polling loop blocks for up to 3 minutes per URL. For Phoenix LiveView, kick off scraping in a Task.async/1 and update the LiveView when the result arrives.

Next steps