Expand AI logo
DocsDocs
Glow Active
API Reference
Login

Documentation

Get Started

OverviewMCP ServerQuickstartWhy ExpandWays to Use Expand

Fetch

OverviewHow Fetch WorksOutput ModelProgressive DisclosureInclude OptionsDocumentsYouTubeBrowser BehaviorHighlightsPlayground & ReplayBatched Fetch

Reference

MCP Tools & ResourcesCLI, Skill & HooksCLI CommandsAPI ReferenceTypeScript SDK

Account & Billing

Pricing & UsageTiersFAQ

Machine-Readable Docs

start.mdllms.txtllms-full.txtDocs as Markdown
Browse docs

Get Started

OverviewMCP ServerQuickstartWhy ExpandWays to Use Expand

Fetch

OverviewHow Fetch WorksOutput ModelProgressive DisclosureInclude OptionsDocumentsYouTubeBrowser BehaviorHighlightsPlayground & ReplayBatched Fetch

Reference

MCP Tools & ResourcesCLI, Skill & HooksCLI CommandsAPI ReferenceTypeScript SDK

Account & Billing

Pricing & UsageTiersFAQ

Machine-Readable Docs

start.mdllms.txtllms-full.txtDocs as Markdown

TypeScript SDK

Use @expandai/sdk to call Fetch from TypeScript and read Main Markdown plus State JSON in code.

Use @expandai/sdk to call Fetch from TypeScript: Main Markdown, State JSON, Highlights, Batched Fetch, and citation helpers.

npm install @expandai/sdk
pnpm add @expandai/sdk
bun add @expandai/sdk
import { ExpandClient, isFetchOriginError } from '@expandai/sdk'

const expand = new ExpandClient({ apiKey: process.env.EXPAND_API_KEY })
const page = await expand.fetchJson({
url: 'https://news.ycombinator.com',
})
if (isFetchOriginError(page)) {
console.error(page.kind, page.url, page.retryable)
} else {
const mainMarkdown = page.markdown
const stateJson = page.json
const snapshotId = page.meta.snapshotId
const playground = page.meta.playground
}

Use fetchJson() for application code that needs Main Markdown plus State JSON. Use fetch() when you only want the Markdown string.

Jump to: JSON Mode · Markdown-only Fetch · Highlights · Batched Fetch · Citation Helpers · Errors · Effect

Install and auth

Install the package, then set your API key. The SDK reads EXPAND_API_KEY from the environment by default.

export EXPAND_API_KEY="xpnd_..."
import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient({
  apiKey: process.env.EXPAND_API_KEY,
})

The SDK sends your key on every request as the x-expand-api-key header.

OptionDefaultDescription
apiKeyprocess.env.EXPAND_API_KEYExpand API key.
baseUrlhttps://api.expand.aiAPI base URL.

Method index

MethodUse whenReturns
fetchJson(params, options?)App code needs Main Markdown, State JSON, snapshot metadata, or inline Highlights.FetchJsonResponse: captured content or FetchOriginError.
fetch(params, options?)You only need the Markdown string.string

JSON Mode with fetchJson

fetchJson() is the recommended SDK method for applications. It returns either captured content — Main Markdown, State JSON, snapshot metadata, and optional search results — or a typed FetchOriginError when the origin could not be reached. Use isFetchOriginError() before reading content fields; it narrows only this top-level network-error case. An HTTP 4xx or 5xx still has captured content and reports its outcome in page.originError.

const page = await expand.fetchJson({
  url: 'https://example.com',
})

if (isFetchOriginError(page)) {
  console.error(page.kind, page.url, page.retryable)
} else {
  console.log(page.markdown)
  console.log(page.json)
  console.log(page.meta.snapshotId)
  console.log(page.meta.playground)
}

The result shape, abbreviated:

type FetchJsonResponse =
  | {
      meta: {
        snapshotId: string
        playground: string
        url: string
        capturedAt: string
        // ...
      }
      markdown: string
      json: Array<unknown>
      data?: {
        search?: {
          query: string
          snippets: Array<{
            source: 'markdown' | 'appendix' | 'statejson'
            text: string
            json?: unknown
  • markdown is the Main Markdown.
  • json is State JSON and extracted evidence.
  • meta.snapshotId is the handle for later Highlights with fetchSearch.
  • meta.playground is the human inspection link.
  • data.search appears when inline search is requested.
  • originError appears on captured content when the origin answered with HTTP 400 or above; it is not a top-level union member.

This shape is not exhaustive. See Output Model and the API Reference for exact fields.

Markdown-only Fetch with fetch

fetch() is the convenience method for scripts and agents that only need the Markdown string.

const markdown = await expand.fetch({
  url: 'https://example.com',
})

console.log(markdown)
  • The return type is string.
  • This calls /v1/fetch.
  • Use fetchJson() if the application needs State JSON, snapshot metadata, or structured search results.

Include options

Both fetch and fetchJson accept query include options through the second argument.

const markdownWithAppendix = await expand.fetch(
  { url: 'https://example.com' },
  { include: 'appendix,statejson' },
)
const objectMode = await expand.fetchJson(
  { url: 'https://example.com' },
  { include: 'appendix' },
)
  • include is a query option passed as the second argument.
  • fetchJson also accepts include options in the request body when you need exact structured control.
  • Passing { include: null } omits the include query param.

Exact include semantics belong to Include Options and the API Reference.

Highlights

Highlights ("search") run two ways: inline with a fresh Fetch, or against a stored snapshot.

Inline Highlights with fetchJson

const page = await expand.fetchJson({
  url: 'https://docs.example.com',
  search: {
    query: 'authentication limits',
    maxResults: 5,
    minScore: 0.6,
  },
})

if (isFetchOriginError(page)) {
  console.error(page.kind, page.url, page.retryable)
} else {
  const snippets = page.data?.search?.snippets ?? []
}
  • This starts a new Fetch.
  • Snippets live under data.search.snippets.
  • State JSON snippets may include json.
  • Raw SDK snippets expose location, not the MCP-only citationUrl field.

Snapshot Highlights with fetchSearch

const page = await expand.fetchJson({
  url: 'https://docs.example.com',
})

if (isFetchOriginError(page)) {
  console.error(page.kind, page.url, page.retryable)
} else {
  const result = await expand.fetchSearch({
    snapshotId: page.meta.snapshotId,
    search: {
      query: 'authentication limits',
      maxResults: 5,
      minScore: 0.6,
    },
  • fetchSearch calls /v1/fetch/search.
  • It searches stored artifacts without recapturing the URL.
  • Use it to refine results after a previous Fetch.
  • The exact request schema belongs to the API Reference. See Highlights for behavior.

Citation Helpers

Raw SDK results expose snippet location. Use Playground helpers when you want to turn a snippet into a user-visible citation link.

import { citationUrl, resolvePlaygroundHost } from '@expandai/sdk/Playground'
if (isFetchOriginError(page)) {
  console.error(page.kind, page.url, page.retryable)
} else {
  const host = resolvePlaygroundHost(page.meta.playground, 'https://expand.land')

  for (const snippet of page.data?.search?.snippets ?? []) {
    const url = citationUrl(page.meta.snapshotId, snippet.location, host)
    console.log(snippet.text, url)
  }
}
HelperPurpose
playgroundBase(snapshotId, host?)Build the whole-snapshot Playground URL.
playgroundOrigin(metaPlayground)Extract the origin from meta.playground.
resolvePlaygroundHost(metaPlayground, fallback)Prefer the server-provided Playground host, fall back if missing.

SDK API responses do not already contain citationUrl. MCP adds that field; SDK users build it with these helpers. See Playground & Replay for URL semantics.

Progressive disclosure with snapshots

fetchJson returns meta.resources when a snapshot was saved. Its snapshotId is the public snapshot-link ID; the snapshots methods read the catalog, collections, and artifact previews behind it without an API key and without refetching the page. Catalog links are paths on the API origin, so resolve them against meta.resources.url before fetching bytes.

const page = await expand.fetchJson({ url: 'https://example.com' })
if (isFetchOriginError(page) || !page.meta.resources) throw new Error('no snapshot to disclose')

const snapshotId = page.meta.resources.snapshotId
const catalog = await expand.snapshots.get({ snapshotId })
const network = await expand.snapshots.network({ snapshotId })
// Archives over the server budget answer 413 here; the content link below still works.
const preview = await expand.snapshots
  .preview({ snapshotId, artifactId:

The full walkthrough is in Progressive Disclosure.

Batched Fetch

const run = await expand.batched({
  urls: [
    'https://example.com',
    'https://example.com/about',
  ],
})

let status = await expand.getBatched(run.id, { limit: '10', offset: '0' })

while (status.batchedStatus === 'QUEUED' || status.batchedStatus === 'RUNNING') {
  await new Promise((resolve) => setTimeout(resolve, 1000))
  • batched() returns a run ID.
  • getBatched() polls the run.
  • Successful results include Main Markdown at data.markdown and State JSON at data.json by default when extraction succeeds.
  • Each result includes its persisted status alongside data, so handle failed or cancelled URLs independently.
  • When omitted, the SDK generates one idempotencyKey and reuses it for the configured retry policy.
  • Supply a stable idempotencyKey to deduplicate separate calls or calls made after a process restart.
  • Reusing a key with a different payload returns 409 BatchedIdempotencyConflict with reason: 'payload_mismatch'.
  • The current generated TypeScript getBatched query params use strings for limit, offset, and include.

The full lifecycle belongs to Batched Fetch.

Effect Service

Use ExpandService when your application is already Effect-native. It exposes the same operations as ExpandClient, but failures are typed in the Effect error channel.

import { NodeRuntime } from '@effect/platform-node'
import { Effect } from 'effect'
import { ExpandService, isFetchOriginError } from '@expandai/sdk'

const program = Effect.gen(function* () {
  const expand = yield* ExpandService
  const page = yield* expand.fetchJson({ url: 'https://example.com' })
  if (isFetchOriginError(page)) {
    yield* Effect.logWarning(page.kind, page.url, page.retryable)

You do not need this section to use the SDK from plain Promise code.

Errors and retries

Promise-side errors, thrown by ExpandClient:

ErrorMeaning
ExpandClientErrorBase Promise-side SDK error and invalid client options.
ExpandClientApiErrorNon-2xx API response. Inspect status and body.
ExpandClientConnectionErrorNetwork failure.

Effect-side errors, surfaced in the ExpandService error channel:

ErrorMeaning
ExpandSdkErrorInvalid service options or SDK setup error.
ExpandApiErrorNon-2xx API response.
ExpandConnectionErrorNetwork failure.
ExpandTimeoutError

Retry and timeout behavior:

  • Default timeoutMs is 60000.
  • Default maxRetries is 2.
  • Retryable failures include timeouts, connection errors, 408, 409, 429, and 5xx.
  • FetchNavigationFailed (502) and FetchCaptureTimeout (504) are exempt from automatic retries. A 504 means the capture started but no result arrived in time; it does not prove the page caused the timeout. Skipping automatic retries keeps an Expand-side incident from being amplified; a deliberate manual retry can still make sense.
  • FetchCapacityTimeout (529) means the request was still waiting for capacity when its deadline passed. It is retryable, and the SDK honors it through the normal retry policy; the response advertises Retry-After: 5.
import { ExpandClientApiError } from '@expandai/sdk'

try {
  await expand.fetchJson({ url: 'https://example.com' })
} catch (error) {
  if (error instanceof ExpandClientApiError) {
    console.error(error.status, error.body)
  }
  throw error
}

Choosing timeoutMs

timeoutMs is a total wall-clock budget across all SDK attempts, and it is enforced client-side. Synchronous Fetch has one 150-second budget per request that covers queueing, rendering, and internal retries, and it never moves. Expand then takes at most five seconds to decide the outcome, so use 180 seconds to leave 25 seconds for the response to reach you:

const page = await expand.fetchJson(
  { url: 'https://example.com' },
  { timeoutMs: 180_000, maxRetries: 0 },
)
  • 180000 covers the budget, the outcome decision, and delivery headroom. A result that arrives while Expand is deciding the outcome is still returned successfully; otherwise the API produces a typed FetchCaptureTimeout/FetchCapacityTimeout by 155 seconds. The remaining 25 seconds are for the HTTP response and network.
  • The 60000 default is sized for typical pages; it aborts long captures with ExpandClientTimeoutError before Expand answers.
  • The budget covers retries and backoff too, so raise timeoutMs per additional attempt you allow, or set maxRetries: 0 when you only want one verdict.
  • FetchCaptureTimeout.timeoutMs is 150000 by default; it is the request ceiling, not the time remaining. The tag means the capture started and did not finish in time; a request that never started is reported separately as FetchCapacityTimeout.

Related pages

TopicWhere to go
First SDK or API setupQuickstart
Choosing SDK vs CLI/MCP/APIWays to Use Expand
Product behaviorFetch Overview
Main Markdown and State JSON
PreviousAPI Reference
NextPricing & Usage

On This Page

Install and authMethod indexJSON Mode with fetchJsonMarkdown-only Fetch with fetchInclude optionsHighlightsInline Highlights with fetchJsonSnapshot Highlights with fetchSearchCitation HelpersProgressive disclosure with snapshotsBatched FetchEffect ServiceErrors and retriesChoosing timeoutMsRelated pages
timeoutMs
60000
Total wall-clock budget across attempts.
maxRetries2Retry attempts for retryable failures.
fetchSearch(params, options?)You already have snapshotId and want Highlights from stored artifacts.Snapshot Highlights result.
batched(params, options?)You want to start many async Fetch jobs. The SDK generates options.idempotencyKey when omitted.{ id }
getBatched(id, options?)You want to poll a Batched Fetch run.Batched status/results.
snapshots.get({ snapshotId })You have meta.resources.snapshotId and want the public catalog of saved artifacts and collections.SnapshotCatalog
snapshots.network({ snapshotId })You want the captured network requests for a snapshot.unknown: the collection JSON { items, next? }; narrow before use.
snapshots.preview({ snapshotId, artifactId, maxBytes? })You want a bounded look at an artifact before downloading it.unknown: { contentType, encoding: "base64", body, bytes, totalBytes, truncated }; narrow before use.
score: number
location?: unknown
}>
}
}
originError?: {
_tag: 'FetchOriginError'
url: string
kind: 'http'
originStatusCode: number
retryable: boolean
}
}
| {
_tag: 'FetchOriginError'
url: string
kind:
| 'dns'
| 'tls'
| 'connectionRefused'
| 'connectionReset'
| 'connectionClosed'
| 'addressUnreachable'
| 'emptyResponse'
| 'connectionTimeout'
upstreamError?: string
retryable: boolean
}
include: {
markdown: true,
json: true,
appendix: false,
},
})
console.log(result.search.snippets)
}
citationUrl(snapshotId, location, host?)
Build a citation link from location.evidenceId.
'html.original'
, maxBytes: 512 })
.catch(() => undefined)
const har = catalog.artifacts.find((artifact) => artifact.id === 'har')
if (har) {
const bytes = await fetch(new URL(har.url, page.meta.resources.url)).then((response) => response.arrayBuffer())
}
status = await expand.getBatched(run.id, { limit: '10', offset: '0' })
}
console.log(status.results)
} else {
yield* Effect.log(page.markdown)
}
})
NodeRuntime.runMain(
program.pipe(
Effect.provide(
ExpandService.layer({ apiKey: process.env.EXPAND_API_KEY }),
),
),
)
ExpandClientTimeoutError
Request exceeded timeoutMs.
Request exceeded timeoutMs.
Output Model
Include semanticsInclude Options
Highlights behaviorHighlights
Citation URLs and replayPlayground & Replay
Batched lifecycleBatched Fetch
Snapshot catalog, collections, artifactsProgressive Disclosure
Exact endpoint schemasAPI Reference