Expand AI logo
DocsDocs
Glow Active
API Reference
Login

Documentation

Get Started

OverviewWhy ExpandQuickstartWays to Use Expand

Agent Quickstarts

OverviewExpand SkillClaude CodeCursorCodexOpenCodeSkill-Based AgentsOther MCP Clients

Fetch

OverviewHow Fetch WorksOutput ModelInclude OptionsBrowser BehaviorHighlightsPlayground & ReplayBatched Fetch

Reference

API ReferenceCLI CommandsMCP Tools & ResourcesTypeScript SDKPython SDK

Account & Billing

Pricing & UsageTiersFAQ

Machine-Readable Docs

start.mdllms.txtllms-full.txtDocs as Markdown
Browse docs

Get Started

OverviewWhy ExpandQuickstartWays to Use Expand

Agent Quickstarts

OverviewExpand SkillClaude CodeCursorCodexOpenCodeSkill-Based AgentsOther MCP Clients

Fetch

OverviewHow Fetch WorksOutput ModelInclude OptionsBrowser BehaviorHighlightsPlayground & ReplayBatched Fetch

Reference

API ReferenceCLI CommandsMCP Tools & ResourcesTypeScript SDKPython SDK

Account & Billing

Pricing & UsageTiersFAQ

Machine-Readable Docs

start.mdllms.txtllms-full.txtDocs as Markdown

Batched Fetch

Fetch many URLs asynchronously and poll for results.

Start many browser captures, poll one run ID, and consume paginated Fetch results as they finish. Batched Fetch is the async fan-out path for processing a list of URLs that share the same settings, without holding open one request per page.

# Start a run
curl -X POST https://api.expand.ai/v1/fetch/batched \
  -H "x-expand-api-key: $EXPAND_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com","https://example.com/about"]}'

# Poll the run ID it returns
curl "https://api.expand.ai/v1/fetch/batched/019...?limit=10&offset=0" \
-H "x-expand-api-key: $EXPAND_API_KEY"
URLs
-> POST /v1/fetch/batched
-> { id }
-> GET /v1/fetch/batched/{id}
-> status + paginated results
Use Batched Fetch whenUse single Fetch when
You have many URLs with the same settings.You need one URL now.
Your pipeline can poll asynchronously.You need screenshots, summaries, State JSON, Appendix, or Highlights.
You want results paged from one run ID.You need the full single-request output model.

Jump to: Start a Run · Poll Results · API Reference

Batched Fetch is API/SDK-only today. There is no CLI command or MCP tool for starting a batch.

Start a Run

Send all the URLs in one POST. The call returns once Hatchet durably accepts the run command; billing reservation, lifecycle projection, and browser captures continue asynchronously in the background.

curl -X POST https://api.expand.ai/v1/fetch/batched \
  -H "x-expand-api-key: $EXPAND_API_KEY" \
  -H "x-idempotency-key: fastenersolutions:2026-07" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com",
      "https://example.com/about"
    ]
  }'
{
  "id": "019..."
}

urls is required and must be non-empty. include is optional, and the only public browserConfig field at start is scrollFullPage. Unknown top-level keys are rejected.

  • Every submitted URL uses the same shared settings. The start body does not accept per-URL settings.
  • Duplicate URLs are deduped by resolved URL, so the same page is captured once.
  • Production callers should authenticate with an API key. Anonymous sessions cannot start batched runs.

Keep the run ID. Every status check and every page of results is read back through it.

An immediate poll can return QUEUED with no results while the accepted command is being projected. This is expected. Billing is checked authoritatively during that initialization, so a run can later become FAILED even though the start request was accepted.

Safe retries

When x-idempotency-key is omitted, the API generates a unique key. Send a stable key when the caller needs to retry a separate request or recover after a process restart. Retrying the same payload with the same explicit key is safe: the API returns a successful response with the same run ID and does not create or charge for another run.

Only reusing the same key with a different request body returns 409 BatchedIdempotencyConflict:

{
  "_tag": "BatchedIdempotencyConflict",
  "existingRunId": "019...",
  "reason": "payload_mismatch"
}

Treat payload_mismatch as a client bug rather than polling a run created for different input.

  • Keys are scoped to your organization and may be up to 255 characters.
  • The API generates a key when omitted. Persist or deterministically reconstruct an explicit key when deduplication must survive separate calls or process restarts.
  • Include a logical period in recurring-job keys, such as fastenersolutions:2026-07:<digest>.

Poll Results

Read status and results back with the run ID. Page through results with limit and offset.

curl "https://api.expand.ai/v1/fetch/batched/019...?limit=10&offset=0" \
  -H "x-expand-api-key: $EXPAND_API_KEY"

Keep polling while batchedStatus is QUEUED or RUNNING. Stop once it reaches a terminal status.

const terminal = new Set(["COMPLETED", "FAILED", "CANCELLED"])

let status = await client.getBatched(run.id)
while (!terminal.has(status.batchedStatus)) {
  await new Promise((resolve) => setTimeout(resolve, 1000))
  status = await client.getBatched(run.id)
}
  • Use a short delay or backoff between polls instead of a tight loop.
  • Handle API errors from the poll call the way you handle any request.
  • FAILED or CANCELLED means stop polling the run.
  • After the run is terminal, inspect each result's status before using it.

Status Model

batchedStatus describes the whole run. The top-level status describes the current page of results, and each result has its own persisted status.

FieldMeaningHow to use it
batchedStatusOverall run status.Decide whether to keep polling the run.
statusCurrent page status, including extraction quality.Decide whether this page is stable and its requested content was extracted.
results[].status
batchedStatus = whole run
status        = current page
  • QUEUED means the run was accepted but may not be executing yet.
  • RUNNING means work is still in progress.
  • COMPLETED on batchedStatus means the run is done, not that every URL produced Markdown.
  • FAILED and CANCELLED are terminal.
  • A page reads COMPLETED only when its rows are stable and requested content extraction succeeded. Requested content can legitimately be empty.
  • A page can read FAILED after a completed capture if snapshot content could not be read or rendered. Successful siblings on that page remain usable.
  • Failed extraction outcomes are cached for one minute. Re-poll the same failed page with backoff to retry extraction without starting or charging for another browser capture.
  • A page can read COMPLETED while the whole run is still RUNNING. Keep the two fields separate in your code; do not collapse them into one status.

Response Shape

A poll returns the run status, pagination metadata, and a page of per-URL results.

{
  "id": "019...",
  "status": "COMPLETED",
  "batchedStatus": "RUNNING",
  "totalUrls": 20,
  "pagination": {
    "total": 12,

Handle each result independently. A batch can finish even when one URL was blocked, failed, or redirected. Use results[].status rather than missing Markdown to identify failed items. A blocked result remains SUCCEEDED and carries data.blocked; fully populated siblings remain usable.

If no poll include.markdown is provided, Batched Fetch normalizes results to Markdown by default. For the exact per-result schema, see the API Reference.

Pagination

Batched results are paginated. Fetch page 1 with offset=0, then keep increasing offset by limit while pagination.hasMore is true.

offset 0  -> 10 results
offset 10 -> 10 results
offset 20 -> ...
  • limit defaults to 10 and accepts 1 through 100.
  • offset defaults to 0 and must be non-negative.
  • pagination.total is the number of available result rows for the run.
  • pagination.hasMore tells you whether to request the next page.
curl "https://api.expand.ai/v1/fetch/batched/019...?limit=100&offset=100" \
  -H "x-expand-api-key: $EXPAND_API_KEY"
  • Polling only the first page can miss later results.
  • Page-level status can differ across pages while the run is still active.

Partial Failures

Batched Fetch is item-oriented. Treat the batch as a container for many Fetch attempts, then inspect each result before using it.

  • Blocked target pages can appear as per-item data.blocked, for example blockedType: "botProtection".
  • data.response.originStatusCode helps classify each outcome, for example 403 on a blocked page.
  • Pending rows can initially return only minimal data.response.
  • A URL-only, unblocked result on a FAILED page can indicate an extraction failure; missing Markdown on a COMPLETED page can be legitimate empty content.
  • Failed extraction outcomes are cached for one minute. Re-poll the same limit/offset page with backoff to retry reading existing capture artifacts without another capture or charge.
  • Terminal COMPLETED on the run does not override a page-level extraction failure.
  • Store per-URL outcome state so a retry collects only the URLs that need it.

Retry a start request with the same key and payload after an ambiguous timeout. The API returns the same run ID if the first request already created the run.

Supported Options

CapabilityBatched Fetch supportNotes
MarkdownSupportedDefault result content.
HTMLSupported on completed resultsRequest through include.
MetaSupportedIncluded by default when available.

Polling rejects screenshot, json, and appendix include fields with BatchedIncludeUnsupported.

Batched Fetch is narrower than single Fetch by design. Use it when many URLs and asynchronous collection matter more than every single Fetch artifact. For the full set of fields and nested defaults, see Include Options.

SDK Examples

The SDKs wrap the same two calls: batched to start a run, getBatched to poll it.

TypeScript

import { ExpandClient } from "@expandai/sdk"

const client = new ExpandClient()

const run = await client.batched({
  urls: ["https://example.com", "https://example.com/about"],
})

let page = await client.getBatched(run.id, { limit: "10", offset: "0" })
while (page.batchedStatus === "QUEUED" || page.batchedStatus === "RUNNING") {
  • When omitted, batched() generates one idempotencyKey and reuses it for the client's configured retry policy.
  • Supply a stable idempotencyKey to deduplicate separate calls or calls made after a process restart.
  • The same key and payload return the same run ID. Catch ExpandClientApiError with status 409 only for a different payload; its body contains existingRunId and reason: "payload_mismatch".
  • The generated getBatched query params are strings today, so the examples use "10" and "0".

Python

from expandai import BatchedParams, Expand

client = Expand()

run = client.batched(BatchedParams(
    urls=["https://example.com", "https://example.com/about"],
))

status = client.get_batched(run.id)
print(status.results)
  • When omitted, batched() generates one idempotency_key and reuses it for the client's configured retry policy.
  • Supply a stable idempotency_key to deduplicate separate calls or calls made after a process restart.
  • The same key and payload return the same run ID. Catch ExpandAPIError with status code 409 only for a different payload; its body contains existingRunId and reason: "payload_mismatch".

Operational Notes

  • Batched Fetch queues its child browser captures under the organization's browser concurrency rather than cancelling each other.
  • Cancellation is cooperative. A successful cancel request returns CANCELLING while running items drain, or CANCELLED when no work remains; queued items are cancelled immediately.
  • Higher tiers can have higher browser concurrency. Exact tier details live in Tiers and Pricing & Usage.
  • Duplicate submitted URLs are deduped by resolved URL.
  • Anonymous sessions cannot start batched runs; API-key auth is the normal production setup.
  • There is no Batched-specific public rate-limit number to plan against today.

Common Mistakes

  • Polling once, seeing no results, and assuming the run failed.
  • Stopping because a page status is COMPLETED while batchedStatus is still RUNNING.
  • Ignoring pagination and reading only the first page.
  • Assuming COMPLETED means every item has Markdown.
  • Requesting unsupported include fields (screenshot, json, appendix) on polling.
  • Looking for a CLI or MCP batch command. There is none today.
  • Expecting an automatically generated key to deduplicate separate calls or calls made after a process restart.
  • Treating payload_mismatch as a recoverable duplicate instead of fixing key reuse.

Next steps

  • Start Batched Fetch API Reference: exact create schema.
  • Get Batched Fetch API Reference: exact poll and result schema.
  • Include Options: the full include surface and limits.
  • Browser Behavior: capture and scrollFullPage behavior.
  • Highlights: search a single Fetch or existing snapshot.
PreviousPlayground & Replay
NextAPI Reference

On This Page

Start a RunSafe retriesPoll ResultsStatus ModelResponse ShapePaginationPartial FailuresSupported OptionsSDK ExamplesTypeScriptPythonOperational NotesCommon MistakesNext steps
Persisted state for one URL.
Handle successful, failed, queued, running, and cancelled URLs independently.
"limit": 10,
"offset": 0,
"hasMore": true
},
"results": [
{
"status": "SUCCEEDED",
"data": {
"response": {
"url": "https://example.com",
"originStatusCode": 200
},
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples."
}
},
{
"status": "SUCCEEDED",
"data": {
"response": {
"url": "https://blocked.example",
"originStatusCode": 403
},
"blocked": {
"blockedType": "botProtection"
}
}
},
{
"status": "FAILED",
"data": {
"response": {
"url": "https://unavailable.example"
}
}
}
]
}
LinksSupported on completed resultsRequest through include.links.
Response infoBasic metadata supportedBatched polling returns URL/status metadata, but not response headers today.
browserConfig.scrollFullPageSupported at startApplies to every child fetch.
ScreenshotsNot supported in batched resultsUse single Fetch.
State JSON / jsonNot supported in batched resultsUse single Fetch or JSON Mode.
AppendixNot supported in batched resultsUse single Fetch.
Highlights/searchNot supportedUse Highlights on a single Fetch or existing snapshot.
await new Promise((resolve) => setTimeout(resolve, 1000))
page = await client.getBatched(run.id, { limit: "10", offset: "0" })
}
console.log(page.results)