{ "openapi": "3.1.0", "info": { "title": "Botnom API", "version": "1.0.0", "summary": "LLM-ready web data: scrape, crawl, map, search, extract and batch-scrape any site.", "description": "Botnom turns web pages into clean, LLM-ready data. One API covers the whole pipeline:\n\n| Operation | What it does | Credits |\n| --- | --- | --- |\n| **Scrape** | One URL → markdown, HTML, links, summary and/or screenshot | 1 |\n| **Crawl** | Follow links from a start URL and scrape every page found | 1 per page |\n| **Map** | Discover a site's URLs (sitemaps, with on-page link fallback) | 1 |\n| **Search** | Web search, optionally scraping every hit | 2 per 10 results |\n| **Extract** | LLM-powered structured extraction from up to 100 URLs | 5 |\n| **Batch** | Scrape up to 1,000 URLs in one job | 1 per URL |\n\n## Authentication\n\nEvery request needs an organization API key (they start with `bn_`), sent as a Bearer token:\n\n```\nAuthorization: Bearer bn_live_xxxxxxxxxxxxxxxxxxxxxxxx\n```\n\nCreate and revoke keys in the dashboard. A missing, malformed, unknown or revoked key yields `401 {\"error\":\"unauthorized\"}`.\n\n## Asynchronous job model\n\nAll six operations are asynchronous:\n\n1. `POST` the operation. Input is validated synchronously and the job is queued; the response is `202 Accepted` with `{\"id\":\"\",\"status\":\"queued\"}`.\n2. Poll `GET /api/v1/jobs/{id}` (or the operation-specific `GET`) until `status` is terminal: `succeeded`, `failed` or `canceled`. Jobs move `queued → running → succeeded | failed`, or to `canceled` when cancelled.\n3. Read the output:\n * **scrape, map, extract** store a single result — `GET /api/v1/jobs/{id}/result` (returns `404` until `hasResult` is `true`).\n * **crawl, batch, search** store one row per page — page through `GET /api/v1/crawl/{id}`, `GET /api/v1/batch/scrape/{id}` (search jobs are paged through `/api/v1/crawl/{id}`, which works for any multi-page job) using `skip` / `limit`. Rows appear while the job is still running.\n * Screenshots are served as PNG from `GET /api/v1/jobs/{id}/files/{slot}`; each document's `screenshot` field holds that path.\n\nWebhooks (configured in the dashboard) deliver `.started`, `.page`, `.completed` and `.failed` events, so polling is optional.\n\n## Credits and plans\n\nCredits are charged when a job **succeeds** (failed jobs are free) and count against a monthly quota that resets on the first of each month (UTC). When the quota is exhausted, new jobs are refused with `429 {\"error\":\"quota_exceeded\"}` and `Retry-After: 3600`.\n\n| Plan | Monthly credits | Requests / minute | Max pages per crawl |\n| --- | --- | --- | --- |\n| free | 1,000 | 20 | 100 |\n| starter | 50,000 | 100 | 5,000 |\n| pro | 500,000 | 500 | 50,000 |\n\n## Rate limits\n\nTwo token buckets apply to every job-creating `POST` (each refilling continuously at the per-minute rate, with a one-minute burst capacity):\n\n* **Per organization** — your plan's requests/minute.\n* **Per target domain** — 120 requests/minute per registrable domain (e.g. `example.com`), shared across all Botnom customers, so no single site is hammered.\n\nWhen either bucket is empty the request is refused with `429 {\"error\":\"rate_limited\",\"scope\":\"org:\" | \"domain:\"}` and a `Retry-After` header (seconds, currently `60`). Back off for at least that long before retrying. No job is created and nothing is charged.\n\n## Errors\n\nErrors are JSON objects with a machine-readable `error` code:\n\n| HTTP | `error` | Meaning |\n| --- | --- | --- |\n| 400 | `invalid_request` | Body failed validation; `issues` lists each problem (`path`, `message`, `code`) |\n| 401 | `unauthorized` | Missing or invalid API key |\n| 403 | `js_actions_disabled` | `executeJavascript` actions are disabled on this deployment |\n| 404 | `not_found` | Unknown job, job owned by another organization, or no result/file yet |\n| 429 | `quota_exceeded` | Monthly credit quota used up (`Retry-After: 3600`) |\n| 429 | `rate_limited` | Per-org or per-domain request budget exhausted (`Retry-After`) |\n| 500 | `internal_error` | Unexpected failure; `detail` carries a short message. Also returned for a request body that is not valid JSON |\n\nValidation runs first, then the JavaScript-action check, then rate limits, then the quota check.", "contact": { "name": "Botnom", "url": "https://botnom.com" } }, "servers": [ { "url": "https://botnom.com", "description": "Production" } ], "security": [ { "bearerAuth": [] } ], "tags": [ { "name": "Scrape", "description": "Scrape a single URL into markdown, HTML, links, a summary and/or a screenshot. Costs 1 credit." }, { "name": "Crawl", "description": "Crawl a site starting from one URL, scraping every page discovered. Costs 1 credit per page crawled. Results are paged; crawls can be cancelled." }, { "name": "Map", "description": "Discover the URLs of a site from its sitemaps (falling back to on-page links). Costs 1 credit." }, { "name": "Search", "description": "Run a web search and optionally scrape each result. Costs 2 credits per 10 results (rounded up, minimum 2); scraping the hits does not add to the cost." }, { "name": "Extract", "description": "LLM-powered structured data extraction from one or more URLs, guided by a prompt and/or a JSON Schema. Costs 5 credits per job." }, { "name": "Batch", "description": "Scrape a list of up to 1,000 URLs in a single job. Costs 1 credit per URL submitted. Results are paged." }, { "name": "Jobs", "description": "Type-agnostic job status, single results and stored files (screenshots)." } ], "paths": { "/api/v1/scrape": { "post": { "tags": [ "Scrape" ], "operationId": "createScrape", "summary": "Scrape a URL", "description": "Queue a scrape of a single URL. Pass scrape options (`ScrapeOptions`) alongside `url` to choose output formats, strip boilerplate, wait for dynamic content or perform browser actions first.\n\n**Cost:** 1 credit.\n\nPoll `GET /api/v1/jobs/{id}` until `status` is `succeeded`, then fetch the `Document` from `GET /api/v1/jobs/{id}/result`. If `screenshot` was requested, the document's `screenshot` field is the path of the PNG, served by `GET /api/v1/jobs/{id}/files/document.png`.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/ScrapeRequest" }, "examples": { "markdown": { "summary": "Markdown + links", "value": { "url": "https://example.com/blog/hello-world", "formats": [ "markdown", "links" ], "onlyMainContent": true } }, "screenshotWithActions": { "summary": "Dismiss a cookie banner, then screenshot", "value": { "url": "https://example.com/pricing", "formats": [ "markdown", "screenshot" ], "waitFor": 1000, "actions": [ { "type": "click", "selector": "#accept-cookies" }, { "type": "wait", "milliseconds": 500 }, { "type": "scroll", "direction": "down" } ] } } } } } }, "responses": { "202": { "$ref": "#/components/responses/JobAccepted" }, "400": { "$ref": "#/components/responses/InvalidRequest" }, "401": { "$ref": "#/components/responses/Unauthorized" }, "403": { "$ref": "#/components/responses/JsActionsDisabled" }, "429": { "$ref": "#/components/responses/TooManyRequests" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/crawl": { "post": { "tags": [ "Crawl" ], "operationId": "createCrawl", "summary": "Start a crawl", "description": "Queue a crawl that starts at `url`, follows links up to `maxDepth` hops away and scrapes at most `limit` pages. `limit` is silently capped at your plan's *max pages per crawl* (free 100, starter 5,000, pro 50,000).\n\nScrape options (formats, `onlyMainContent`, actions, ...) apply to every page. They may be given at the top level of the body or nested under `scrapeOptions`; when `scrapeOptions` is present it is used as-is and top-level scrape options are ignored.\n\n**Cost:** 1 credit per page crawled (minimum 1), charged when the crawl succeeds.\n\nPoll `GET /api/v1/crawl/{id}` — pages become available while the crawl is still running. Cancel with `DELETE /api/v1/crawl/{id}`.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/CrawlRequest" }, "examples": { "docsSection": { "summary": "Crawl a docs section", "value": { "url": "https://example.com/docs", "maxDepth": 3, "limit": 200, "includePaths": [ "*/docs/*" ], "excludePaths": [ "*/docs/changelog/*" ], "strategy": "bfs", "scrapeOptions": { "formats": [ "markdown", "links" ], "onlyMainContent": true } } }, "flat": { "summary": "Scrape options at the top level", "value": { "url": "https://example.com", "limit": 20, "maxDepth": 1, "formats": [ "markdown" ] } } } } } }, "responses": { "202": { "$ref": "#/components/responses/JobAccepted" }, "400": { "$ref": "#/components/responses/InvalidRequest" }, "401": { "$ref": "#/components/responses/Unauthorized" }, "403": { "$ref": "#/components/responses/JsActionsDisabled" }, "429": { "$ref": "#/components/responses/TooManyRequests" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/crawl/{id}": { "parameters": [ { "$ref": "#/components/parameters/JobId" } ], "get": { "tags": [ "Crawl" ], "operationId": "getCrawl", "summary": "Get crawl status and pages", "description": "Returns the job's status fields merged with a page of its results. Pages are ordered by discovery time and appear while the crawl is still `running`, so you can stream results by polling with an increasing `skip`.\n\n* `total` — number of pages stored so far.\n* `completed` — the final billable page count; `0` until the job has `succeeded`.\n* `next` — the `skip` value for the next page, or `null` when you have reached the end of what is stored so far.\n\nThis endpoint works for any multi-page job (crawl, batch scrape or search), not only crawls.", "parameters": [ { "$ref": "#/components/parameters/Skip" }, { "$ref": "#/components/parameters/Limit" } ], "responses": { "200": { "description": "Job status plus one page of results.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/PagedJob" }, "examples": { "running": { "summary": "Crawl still running (first 2 of 2 pages so far)", "value": { "id": "cmg7p4x2a0001l80bq3h9z1kd", "type": "crawl", "status": "running", "creditsUsed": 0, "pageCount": 0, "error": null, "input": { "url": "https://example.com/docs", "maxDepth": 3, "limit": 100, "includePaths": [ "*/docs/*" ], "allowExternalLinks": false, "strategy": "bfs" }, "createdAt": "2026-10-06T09:12:44.120Z", "startedAt": "2026-10-06T09:12:45.004Z", "finishedAt": null, "hasResult": false, "total": 2, "completed": 0, "data": [ { "url": "https://example.com/docs", "markdown": "# Documentation\n\nWelcome to the docs...", "links": [ "https://example.com/docs/quickstart", "https://example.com/docs/api" ], "metadata": { "title": "Documentation", "description": "Everything you need to get started.", "language": "en" }, "statusCode": 200 }, { "url": "https://example.com/docs/quickstart", "markdown": "# Quickstart\n\nInstall the CLI...", "links": [ "https://example.com/docs" ], "metadata": { "title": "Quickstart", "language": "en" }, "statusCode": 200 } ], "next": null } } } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "$ref": "#/components/responses/NotFound" }, "500": { "$ref": "#/components/responses/InternalError" } } }, "delete": { "tags": [ "Crawl" ], "operationId": "cancelCrawl", "summary": "Cancel a crawl", "description": "Marks the job `canceled`. A `queued` job will not start. A `running` job is not interrupted mid-request, but it stays `canceled` when its current work finishes and no credits are charged for it; pages it stored remain readable. Jobs that have already `succeeded` or `failed` cannot be cancelled and respond `404`. Cancelling an already-cancelled job succeeds again (idempotent).\n\nAlthough it lives under `/crawl`, this works for a job of any type.", "responses": { "200": { "description": "The job is now cancelled.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/CancelResult" }, "example": { "status": "canceled" } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "description": "No such job in your organization, or the job already finished (`succeeded` / `failed`) and can no longer be cancelled.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "not_found" } } } }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/map": { "post": { "tags": [ "Map" ], "operationId": "createMap", "summary": "Map a site's URLs", "description": "Queue a map job that lists the URLs of a site. Botnom reads `/sitemap.xml` and every `Sitemap:` entry in `robots.txt` (following one level of sitemap indexes); if no sitemap yields anything it falls back to extracting the links on the page itself. Results are restricted to the same host (and its subdomains when `includeSubdomains` is true), optionally filtered by `search`, and truncated to `limit`.\n\n**Cost:** 1 credit.\n\nFetch the result from `GET /api/v1/jobs/{id}/result` once the job has `succeeded`: `{\"links\": [\"https://...\", ...]}`.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/MapRequest" }, "examples": { "basic": { "summary": "Whole site", "value": { "url": "https://example.com" } }, "filtered": { "summary": "Only blog URLs, no subdomains", "value": { "url": "https://example.com", "search": "/blog/", "limit": 500, "includeSubdomains": false } } } } } }, "responses": { "202": { "$ref": "#/components/responses/JobAccepted" }, "400": { "$ref": "#/components/responses/InvalidRequest" }, "401": { "$ref": "#/components/responses/Unauthorized" }, "429": { "$ref": "#/components/responses/TooManyRequests" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/batch/scrape": { "post": { "tags": [ "Batch" ], "operationId": "createBatchScrape", "summary": "Scrape many URLs", "description": "Queue one job that scrapes every URL in `urls` (1–1,000) with the same scrape options, given at the top level of the body. URLs are processed in order; a URL that fails is recorded as a `{url, error}` row and does not fail the job.\n\n**Cost:** 1 credit per URL submitted (minimum 1), charged when the job succeeds.\n\nPage through the results with `GET /api/v1/batch/scrape/{id}`.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/BatchScrapeRequest" }, "example": { "urls": [ "https://example.com/products/1", "https://example.com/products/2", "https://example.com/products/3" ], "formats": [ "markdown", "summary" ], "onlyMainContent": true, "timeout": 30000 } } } }, "responses": { "202": { "$ref": "#/components/responses/JobAccepted" }, "400": { "$ref": "#/components/responses/InvalidRequest" }, "401": { "$ref": "#/components/responses/Unauthorized" }, "403": { "$ref": "#/components/responses/JsActionsDisabled" }, "429": { "$ref": "#/components/responses/TooManyRequests" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/batch/scrape/{id}": { "parameters": [ { "$ref": "#/components/parameters/JobId" } ], "get": { "tags": [ "Batch" ], "operationId": "getBatchScrape", "summary": "Get batch status and pages", "description": "Returns the job's status fields merged with a page of its results, in submission order. Each row is either a `Document` or, for a URL that could not be scraped, `{\"url\": ..., \"error\": ...}`. Rows appear while the job is still `running`.\n\n* `total` — rows stored so far.\n* `completed` — the final billable count (number of URLs); `0` until the job has `succeeded`.\n* `next` — the `skip` for the next page, or `null`.", "parameters": [ { "$ref": "#/components/parameters/Skip" }, { "$ref": "#/components/parameters/Limit" } ], "responses": { "200": { "description": "Job status plus one page of results.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/PagedJob" }, "examples": { "succeeded": { "summary": "Finished batch with one failed URL", "value": { "id": "cmg7p9k1z0003l80bx7v2m4qa", "type": "batch_scrape", "status": "succeeded", "creditsUsed": 3, "pageCount": 3, "error": null, "input": { "urls": [ "https://example.com/products/1", "https://example.com/products/2", "https://example.com/products/3" ] }, "createdAt": "2026-10-06T09:20:01.511Z", "startedAt": "2026-10-06T09:20:02.090Z", "finishedAt": "2026-10-06T09:20:19.730Z", "hasResult": false, "total": 3, "completed": 3, "data": [ { "url": "https://example.com/products/1", "markdown": "# Product 1\n\n...", "summary": "Product 1 is a widget.", "metadata": { "title": "Product 1", "description": "Product 1 is a widget." }, "statusCode": 200 }, { "url": "https://example.com/products/2", "error": "Error: crawl failed for https://example.com/products/2: net::ERR_NAME_NOT_RESOLVED" }, { "url": "https://example.com/products/3", "markdown": "# Product 3\n\n...", "summary": "Product 3 is a gadget.", "metadata": { "title": "Product 3" }, "statusCode": 200 } ], "next": null } } } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "$ref": "#/components/responses/NotFound" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/search": { "post": { "tags": [ "Search" ], "operationId": "createSearch", "summary": "Search the web", "description": "Queue a web search for `query`. Each hit is stored as a row with `url`, `title` and `description`. Set `scrape: true` to also scrape every hit with the given scrape options (top level or nested under `scrapeOptions`); a hit whose scrape fails keeps its search metadata.\n\n**Cost:** 2 credits per 10 results (rounded up, minimum 2), charged when the job succeeds. Scraping the hits does not change the cost.\n\nPage through the results with `GET /api/v1/crawl/{id}` (the paged endpoint accepts any multi-page job id).", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/SearchRequest" }, "examples": { "basic": { "summary": "Top 10 results", "value": { "query": "web scraping api for llms", "limit": 10 } }, "scraped": { "summary": "Localised search, scraping each hit to markdown", "value": { "query": "beste pizza berlin", "limit": 5, "country": "de", "lang": "de", "scrape": true, "scrapeOptions": { "formats": [ "markdown" ], "onlyMainContent": true } } } } } } }, "responses": { "202": { "$ref": "#/components/responses/JobAccepted" }, "400": { "$ref": "#/components/responses/InvalidRequest" }, "401": { "$ref": "#/components/responses/Unauthorized" }, "403": { "$ref": "#/components/responses/JsActionsDisabled" }, "429": { "$ref": "#/components/responses/TooManyRequests" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/extract": { "post": { "tags": [ "Extract" ], "operationId": "createExtract", "summary": "Extract structured data", "description": "Queue an LLM extraction over 1–100 URLs. Each page is rendered to markdown and passed to the model with your `prompt` (natural-language instructions) and/or `schema` (a JSON Schema describing the object you want back). With a `schema` the model returns data shaped like it; with only a `prompt` it returns free-form JSON blocks. Provide at least one of the two.\n\n**Cost:** 5 credits per job, regardless of the number of URLs.\n\nFetch the output from `GET /api/v1/extract/{id}` (status + data in one call) or `GET /api/v1/jobs/{id}/result`.", "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/ExtractRequest" }, "examples": { "schema": { "summary": "Schema-guided extraction", "value": { "urls": [ "https://example.com/pricing" ], "prompt": "Extract every pricing plan with its monthly price in USD.", "schema": { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "priceUsd": { "type": "number" }, "features": { "type": "array", "items": { "type": "string" } } }, "required": [ "name", "priceUsd" ] } } }, "required": [ "plans" ] } } }, "promptOnly": { "summary": "Prompt-only extraction across several pages", "value": { "urls": [ "https://example.com/team", "https://example.com/about" ], "prompt": "List the people mentioned with their job titles." } } } } } }, "responses": { "202": { "$ref": "#/components/responses/JobAccepted" }, "400": { "$ref": "#/components/responses/InvalidRequest" }, "401": { "$ref": "#/components/responses/Unauthorized" }, "429": { "$ref": "#/components/responses/TooManyRequests" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/extract/{id}": { "parameters": [ { "$ref": "#/components/parameters/JobId" } ], "get": { "tags": [ "Extract" ], "operationId": "getExtract", "summary": "Get extract status and data", "description": "Returns the job's status fields plus `data`: the stored `ExtractResult` once the job has `succeeded`, otherwise `null`. Convenient single call for polling extractions; `GET /api/v1/jobs/{id}/result` returns the same `ExtractResult` on its own.", "responses": { "200": { "description": "Job status, with the extraction output when available.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/ExtractJob" }, "examples": { "succeeded": { "summary": "Finished extraction", "value": { "id": "cmg7pd3rt0005l80bk2n8w6ye", "type": "extract", "status": "succeeded", "creditsUsed": 5, "pageCount": 1, "error": null, "input": { "urls": [ "https://example.com/pricing" ], "prompt": "Extract every pricing plan with its monthly price in USD.", "schema": { "type": "object", "properties": { "plans": { "type": "array" } } } }, "createdAt": "2026-10-06T09:30:10.000Z", "startedAt": "2026-10-06T09:30:10.900Z", "finishedAt": "2026-10-06T09:30:24.410Z", "hasResult": true, "data": { "data": { "plans": [ { "name": "Free", "priceUsd": 0, "features": [ "1,000 credits" ] }, { "name": "Starter", "priceUsd": 29, "features": [ "50,000 credits" ] } ] }, "sources": [ "https://example.com/pricing" ] } } }, "pending": { "summary": "Still running", "value": { "id": "cmg7pd3rt0005l80bk2n8w6ye", "type": "extract", "status": "running", "creditsUsed": 0, "pageCount": 0, "error": null, "input": { "urls": [ "https://example.com/pricing" ], "prompt": "Extract every pricing plan with its monthly price in USD." }, "createdAt": "2026-10-06T09:30:10.000Z", "startedAt": "2026-10-06T09:30:10.900Z", "finishedAt": null, "hasResult": false, "data": null } } } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "$ref": "#/components/responses/NotFound" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/jobs/{id}": { "parameters": [ { "$ref": "#/components/parameters/JobId" } ], "get": { "tags": [ "Jobs" ], "operationId": "getJob", "summary": "Get job status", "description": "Status of any job (scrape, crawl, map, search, extract or batch_scrape). Poll this until `status` is `succeeded`, `failed` or `canceled`. `hasResult` tells you whether `GET /api/v1/jobs/{id}/result` will return a body; `creditsUsed` and `pageCount` are filled in when the job succeeds; `error` is set when it fails.", "responses": { "200": { "description": "The job.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/JobStatus" }, "examples": { "queued": { "summary": "Just created", "value": { "id": "cmg7p1h3b0000l80b5r8d2x9f", "type": "scrape", "status": "queued", "creditsUsed": 0, "pageCount": 0, "error": null, "input": { "url": "https://example.com/blog/hello-world" }, "createdAt": "2026-10-06T09:05:00.000Z", "startedAt": null, "finishedAt": null, "hasResult": false } }, "succeeded": { "summary": "Scrape finished", "value": { "id": "cmg7p1h3b0000l80b5r8d2x9f", "type": "scrape", "status": "succeeded", "creditsUsed": 1, "pageCount": 1, "error": null, "input": { "url": "https://example.com/blog/hello-world" }, "createdAt": "2026-10-06T09:05:00.000Z", "startedAt": "2026-10-06T09:05:00.420Z", "finishedAt": "2026-10-06T09:05:03.870Z", "hasResult": true } }, "failed": { "summary": "Scrape failed", "value": { "id": "cmg7p1h3b0000l80b5r8d2x9f", "type": "scrape", "status": "failed", "creditsUsed": 0, "pageCount": 0, "error": "crawl failed for https://example.com/blog/hello-world: Page.goto: Timeout 60000ms exceeded.", "input": { "url": "https://example.com/blog/hello-world" }, "createdAt": "2026-10-06T09:05:00.000Z", "startedAt": "2026-10-06T09:05:00.420Z", "finishedAt": "2026-10-06T09:06:01.112Z", "hasResult": false } } } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "$ref": "#/components/responses/NotFound" }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/jobs/{id}/result": { "parameters": [ { "$ref": "#/components/parameters/JobId" } ], "get": { "tags": [ "Jobs" ], "operationId": "getJobResult", "summary": "Get a job's single result", "description": "The stored result of a single-result job, by job type:\n\n| Job type | Body |\n| --- | --- |\n| `scrape` | `Document` |\n| `map` | `MapResult` — `{\"links\": [...]}` |\n| `extract` | `ExtractResult` — `{\"data\": ..., \"sources\": [...]}` |\n\nResponds `404` until the job has produced its result (check `hasResult` on `GET /api/v1/jobs/{id}`), and always `404` for `crawl`, `batch_scrape` and `search` jobs, which store per-page rows instead — page through those with `GET /api/v1/crawl/{id}` or `GET /api/v1/batch/scrape/{id}`.\n\nThe three result kinds above are returned as `application/json`. Any other stored format (none are produced today; reserved for future result types) is streamed back verbatim as `application/octet-stream`.", "responses": { "200": { "description": "The job's result.", "content": { "application/json": { "schema": { "oneOf": [ { "$ref": "#/components/schemas/Document" }, { "$ref": "#/components/schemas/MapResult" }, { "$ref": "#/components/schemas/ExtractResult" } ] }, "examples": { "scrape": { "summary": "Scrape result (Document)", "value": { "url": "https://example.com/blog/hello-world", "markdown": "# Hello, world\n\nThis is the first post on our blog...", "links": [ "https://example.com/", "https://example.com/blog" ], "screenshot": "/api/v1/jobs/cmg7p1h3b0000l80b5r8d2x9f/files/document.png", "metadata": { "title": "Hello, world — Example Blog", "description": "This is the first post on our blog.", "language": "en", "og:title": "Hello, world", "og:image": "https://example.com/og/hello.png" }, "statusCode": 200 } }, "map": { "summary": "Map result", "value": { "links": [ "https://example.com/", "https://example.com/about", "https://example.com/blog", "https://example.com/blog/hello-world", "https://example.com/pricing" ] } }, "extract": { "summary": "Extract result", "value": { "data": { "plans": [ { "name": "Free", "priceUsd": 0 }, { "name": "Starter", "priceUsd": 29 } ] }, "sources": [ "https://example.com/pricing" ] } } } }, "application/octet-stream": { "schema": { "type": "string", "format": "binary", "description": "Raw bytes of a non-JSON stored result (reserved; not produced by current job types)." } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "description": "Unknown job, job owned by another organization, no result stored yet (job still queued/running or failed), or a multi-page job type (`crawl`, `batch_scrape`, `search`) that has no single result.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "not_found" } } } }, "500": { "$ref": "#/components/responses/InternalError" } } } }, "/api/v1/jobs/{id}/files/{slot}": { "parameters": [ { "$ref": "#/components/parameters/JobId" }, { "name": "slot", "in": "path", "required": true, "description": "Name of the stored file within the job. Screenshots use the slot `document.png` for scrape jobs and `pages/{n}.png` (zero-based) for crawl, batch and search pages — exactly the value found after `/files/` in a document's `screenshot` field. The slot is a single path segment, so a `/` inside it must be percent-encoded (`pages%2F0.png`); OpenAPI-generated clients do this automatically. The job's JSON blobs (`document.json`, `pages/{n}.json`, `map.json`, `extract.json`) are also addressable here and are returned as `application/octet-stream`.", "schema": { "type": "string", "examples": [ "document.png", "pages/0.png" ] } } ], "get": { "tags": [ "Jobs" ], "operationId": "getJobFile", "summary": "Download a stored file (screenshot)", "description": "Streams a file stored by a job — primarily the PNG screenshots produced by the `screenshot` format. Files ending in `.png` are served as `image/png`; everything else as `application/octet-stream`. Responds `404` if the job does not exist in your organization or the slot has no file (for example a screenshot that was not requested, or a page the job has not reached yet).\n\nThe response is the raw file, not JSON — note that an `` tag cannot send the `Authorization` header, so download the bytes server-side or via `fetch` and display them as a blob/data URL.", "responses": { "200": { "description": "The file bytes.", "content": { "image/png": { "schema": { "type": "string", "format": "binary" } }, "application/octet-stream": { "schema": { "type": "string", "format": "binary" } } } }, "401": { "$ref": "#/components/responses/Unauthorized" }, "404": { "description": "Unknown job, job owned by another organization, or no file stored under this slot.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "not_found" } } } }, "500": { "$ref": "#/components/responses/InternalError" } } } } }, "components": { "securitySchemes": { "bearerAuth": { "type": "http", "scheme": "bearer", "bearerFormat": "Organization API key (bn_...)", "description": "Send your organization API key as a Bearer token: `Authorization: Bearer bn_...`. Keys are created in the dashboard and are scoped to one organization; all jobs and results are visible only to the organization that owns the key." } }, "parameters": { "JobId": { "name": "id", "in": "path", "required": true, "description": "Job id, as returned in the `id` field of the `202 Accepted` response that created it.", "schema": { "type": "string", "examples": [ "cmg7p4x2a0001l80bq3h9z1kd" ] } }, "Skip": { "name": "skip", "in": "query", "required": false, "description": "Number of result rows to skip (offset pagination). Use the `next` value from the previous response.", "schema": { "type": "integer", "minimum": 0, "default": 0 } }, "Limit": { "name": "limit", "in": "query", "required": false, "description": "Maximum number of result rows to return in this page. Defaults to 100; a value of `0` or a non-number also falls back to 100. There is no server-side maximum, but large pages mean large responses (each row can carry full page content).", "schema": { "type": "integer", "minimum": 1, "default": 100 } } }, "headers": { "RetryAfter": { "description": "Seconds to wait before retrying. `60` for `rate_limited`, `3600` for `quota_exceeded`.", "schema": { "type": "integer", "examples": [ 60, 3600 ] } } }, "responses": { "JobAccepted": { "description": "The job was validated and queued. Poll `GET /api/v1/jobs/{id}` for progress.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/JobAccepted" }, "example": { "id": "cmg7p4x2a0001l80bq3h9z1kd", "status": "queued" } } } }, "InvalidRequest": { "description": "The request body failed validation. `issues` lists every problem with its JSON `path`.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "invalid_request", "issues": [ { "validation": "url", "code": "invalid_string", "message": "Invalid url", "path": [ "url" ] }, { "code": "too_big", "maximum": 10, "type": "number", "inclusive": true, "exact": false, "message": "Number must be less than or equal to 10", "path": [ "maxDepth" ] } ] } } } }, "Unauthorized": { "description": "The `Authorization: Bearer bn_...` header is missing, malformed, unknown or revoked.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "unauthorized" } } } }, "JsActionsDisabled": { "description": "The request contains an `executeJavascript` action, and arbitrary JavaScript execution is disabled on this deployment. Remove the action (the other action types — `wait`, `click`, `write`, `press`, `scroll` — are always allowed).", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "js_actions_disabled" } } } }, "NotFound": { "description": "No job with this id exists in your organization.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "not_found" } } } }, "TooManyRequests": { "description": "Refused before any job was created — nothing is charged. Either a rate limit (`rate_limited`, with `scope` naming the exhausted bucket: `org:` for your plan's requests/minute or `domain:` for the shared 120 requests/minute per target domain) or your monthly credit quota (`quota_exceeded`). Wait at least `Retry-After` seconds before retrying.", "headers": { "Retry-After": { "$ref": "#/components/headers/RetryAfter" } }, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "examples": { "rateLimitedOrg": { "summary": "Per-organization limit (Retry-After: 60)", "value": { "error": "rate_limited", "scope": "org:cmg7nz0aa0000l80b1c2d3e4f" } }, "rateLimitedDomain": { "summary": "Per-target-domain limit (Retry-After: 60)", "value": { "error": "rate_limited", "scope": "domain:example.com" } }, "quotaExceeded": { "summary": "Monthly quota used up (Retry-After: 3600)", "value": { "error": "quota_exceeded" } } } } } }, "InternalError": { "description": "Unexpected server error. `detail` carries a short diagnostic message. Also returned when the request body is not valid JSON.", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" }, "example": { "error": "internal_error", "detail": "SyntaxError: Unexpected token } in JSON at position 42" } } } } }, "schemas": { "Format": { "type": "string", "description": "An output format a scrape can produce. Each requested format adds the matching field to the resulting Document.\n\n* `markdown` — the page as Markdown (`markdown` field).\n* `html` — cleaned HTML (`html`).\n* `rawHtml` — the original HTML as received (`rawHtml`).\n* `links` — absolute URLs linked from the page (`links`).\n* `screenshot` — a PNG screenshot, served from `/api/v1/jobs/{id}/files/{slot}` (`screenshot` holds the path).\n* `summary` — a short summary of the page, taken from its meta description (`summary`).", "enum": [ "markdown", "html", "rawHtml", "links", "screenshot", "summary" ] }, "Action": { "description": "A browser action executed in the headless browser before the page content is captured, in the order given. Use actions to dismiss banners, open menus, fill search boxes or trigger lazy loading.", "oneOf": [ { "$ref": "#/components/schemas/WaitAction" }, { "$ref": "#/components/schemas/ClickAction" }, { "$ref": "#/components/schemas/WriteAction" }, { "$ref": "#/components/schemas/PressAction" }, { "$ref": "#/components/schemas/ScrollAction" }, { "$ref": "#/components/schemas/ExecuteJavascriptAction" } ], "discriminator": { "propertyName": "type", "mapping": { "wait": "#/components/schemas/WaitAction", "click": "#/components/schemas/ClickAction", "write": "#/components/schemas/WriteAction", "press": "#/components/schemas/PressAction", "scroll": "#/components/schemas/ScrollAction", "executeJavascript": "#/components/schemas/ExecuteJavascriptAction" } } }, "WaitAction": { "type": "object", "title": "wait", "description": "Pause before continuing. With `selector`, wait until an element matching the CSS selector is present (this becomes the page's wait condition); otherwise sleep for `milliseconds`.", "required": [ "type" ], "properties": { "type": { "type": "string", "const": "wait" }, "milliseconds": { "type": "integer", "minimum": 1, "description": "How long to sleep. Ignored when `selector` is given.", "examples": [ 1000 ] }, "selector": { "type": "string", "description": "CSS selector to wait for.", "examples": [ "#results .item" ] } }, "examples": [ { "type": "wait", "milliseconds": 1500 }, { "type": "wait", "selector": "main article" } ] }, "ClickAction": { "type": "object", "title": "click", "description": "Click the first element matching the CSS selector (no-op if it does not exist).", "required": [ "type", "selector" ], "properties": { "type": { "type": "string", "const": "click" }, "selector": { "type": "string", "description": "CSS selector of the element to click.", "examples": [ "button#accept-cookies" ] } } }, "WriteAction": { "type": "object", "title": "write", "description": "Set the value of an input (the element matching `selector`, or the currently focused element) and dispatch an `input` event.", "required": [ "type", "text" ], "properties": { "type": { "type": "string", "const": "write" }, "text": { "type": "string", "description": "Text to write into the field.", "examples": [ "botnom" ] }, "selector": { "type": "string", "description": "CSS selector of the input. Defaults to the active element.", "examples": [ "input[name=q]" ] } } }, "PressAction": { "type": "object", "title": "press", "description": "Dispatch a `keydown` event for `key` to the focused element.", "required": [ "type", "key" ], "properties": { "type": { "type": "string", "const": "press" }, "key": { "type": "string", "description": "A `KeyboardEvent.key` value.", "examples": [ "Enter", "Escape", "ArrowDown" ] } } }, "ScrollAction": { "type": "object", "title": "scroll", "description": "Scroll the window by one viewport height.", "required": [ "type" ], "properties": { "type": { "type": "string", "const": "scroll" }, "direction": { "type": "string", "enum": [ "up", "down" ], "default": "down" } } }, "ExecuteJavascriptAction": { "type": "object", "title": "executeJavascript", "description": "Run arbitrary JavaScript in the page. Disabled by default: on deployments where it is not enabled, any request containing this action is rejected with `403 js_actions_disabled`.", "required": [ "type", "script" ], "properties": { "type": { "type": "string", "const": "executeJavascript" }, "script": { "type": "string", "description": "JavaScript source to execute.", "examples": [ "document.querySelectorAll('.ad').forEach(e => e.remove());" ] } } }, "ScrapeOptions": { "type": "object", "description": "Options controlling how a page is rendered and what is returned. Shared by scrape, crawl, batch scrape and search (with `scrape: true`). Every field is optional; unknown fields are ignored.", "properties": { "formats": { "type": "array", "items": { "$ref": "#/components/schemas/Format" }, "default": [ "markdown" ], "description": "Output formats to produce. Each adds the matching field to the Document.", "examples": [ [ "markdown", "links" ] ] }, "onlyMainContent": { "type": "boolean", "default": true, "description": "Keep only the main content, dropping navigation, footers and other boilerplate (implemented as a word-count threshold on content blocks). Set `false` to keep everything." }, "includeTags": { "type": "array", "items": { "type": "string" }, "description": "CSS selectors to restrict extraction to. When given, only content inside matching elements is returned.", "examples": [ [ "article", "main" ] ] }, "excludeTags": { "type": "array", "items": { "type": "string" }, "description": "HTML tag names to remove before extraction.", "examples": [ [ "nav", "footer", "aside" ] ] }, "waitFor": { "type": "integer", "minimum": 0, "maximum": 60000, "default": 0, "description": "Milliseconds to wait after the page loads before capturing content; useful for client-rendered pages." }, "timeout": { "type": "integer", "minimum": 1, "maximum": 300000, "default": 60000, "description": "Page load timeout in milliseconds. A page that does not load in time fails the scrape (or that page of a crawl/batch)." }, "headers": { "type": "object", "additionalProperties": { "type": "string" }, "description": "Extra HTTP request headers to send to the target. Accepted and validated; reserved — not yet applied by the rendering engine.", "examples": [ { "Accept-Language": "en-GB" } ] }, "actions": { "type": "array", "items": { "$ref": "#/components/schemas/Action" }, "description": "Browser actions to perform, in order, before content is captured." }, "mobile": { "type": "boolean", "default": false, "description": "Emulate a mobile viewport. Accepted and validated; reserved — not yet applied by the rendering engine." }, "fullPage": { "type": "boolean", "default": false, "description": "Capture the full scrollable page in the `screenshot` format instead of the viewport. Accepted and validated; reserved — not yet applied by the rendering engine." }, "location": { "type": "object", "description": "Geographic/language hints for the target site. Accepted and validated; reserved — not yet applied by the rendering engine.", "properties": { "country": { "type": "string", "minLength": 2, "maxLength": 2, "description": "ISO 3166-1 alpha-2 country code.", "examples": [ "US" ] }, "languages": { "type": "array", "items": { "type": "string" }, "description": "Preferred languages, most preferred first (BCP 47 tags).", "examples": [ [ "en-US", "en" ] ] } } } } }, "ScrapeRequest": { "title": "ScrapeRequest", "description": "Body of `POST /api/v1/scrape`: the URL to scrape plus any scrape options at the top level.", "allOf": [ { "$ref": "#/components/schemas/ScrapeOptions" }, { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "format": "uri", "description": "Absolute URL to scrape (http or https).", "examples": [ "https://example.com/blog/hello-world" ] } } } ] }, "CrawlRequest": { "title": "CrawlRequest", "description": "Body of `POST /api/v1/crawl`. Scrape options may be nested under `scrapeOptions` or given at the top level; when `scrapeOptions` is present it takes precedence and top-level scrape options are ignored.", "allOf": [ { "$ref": "#/components/schemas/ScrapeOptions" }, { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "format": "uri", "description": "Absolute URL to start crawling from.", "examples": [ "https://example.com/docs" ] }, "maxDepth": { "type": "integer", "minimum": 0, "maximum": 10, "default": 2, "description": "How many link hops away from the start URL to follow. `0` crawls only the start URL." }, "limit": { "type": "integer", "minimum": 1, "default": 100, "description": "Maximum number of pages to crawl. Silently capped at your plan's max pages per crawl (free 100, starter 5,000, pro 50,000); the capped value is what appears in the job's `input.limit`." }, "includePaths": { "type": "array", "items": { "type": "string" }, "description": "URL glob patterns; when given, only URLs matching at least one pattern are crawled (the start URL is always fetched).", "examples": [ [ "*/docs/*", "*/guides/*" ] ] }, "excludePaths": { "type": "array", "items": { "type": "string" }, "description": "URL glob patterns; URLs matching any pattern are skipped.", "examples": [ [ "*/login*", "*?page=*" ] ] }, "allowExternalLinks": { "type": "boolean", "default": false, "description": "Follow links to other domains. Off by default, which keeps the crawl on the start URL's site." }, "strategy": { "type": "string", "enum": [ "bfs", "dfs", "best_first" ], "default": "bfs", "description": "Traversal order: breadth-first (`bfs`), depth-first (`dfs`) or relevance-ordered (`best_first`). With a `limit` smaller than the site, the strategy decides which pages make the cut." }, "scrapeOptions": { "$ref": "#/components/schemas/ScrapeOptions", "description": "Scrape options applied to every crawled page. Takes precedence over top-level scrape options when present." } } } ] }, "MapRequest": { "type": "object", "title": "MapRequest", "description": "Body of `POST /api/v1/map`.", "required": [ "url" ], "properties": { "url": { "type": "string", "format": "uri", "description": "Any URL on the site to map; its origin is used to locate `robots.txt` and `/sitemap.xml`.", "examples": [ "https://example.com" ] }, "search": { "type": "string", "description": "Case-insensitive substring filter applied to each discovered URL.", "examples": [ "/blog/" ] }, "limit": { "type": "integer", "minimum": 1, "maximum": 100000, "default": 5000, "description": "Maximum number of URLs to return." }, "includeSubdomains": { "type": "boolean", "default": true, "description": "Also keep URLs on subdomains (and parent domains) of the target host. When `false`, only URLs on exactly the same host are kept." } } }, "BatchScrapeRequest": { "title": "BatchScrapeRequest", "description": "Body of `POST /api/v1/batch/scrape`: the URLs plus scrape options at the top level (applied to every URL).", "allOf": [ { "$ref": "#/components/schemas/ScrapeOptions" }, { "type": "object", "required": [ "urls" ], "properties": { "urls": { "type": "array", "minItems": 1, "maxItems": 1000, "items": { "type": "string", "format": "uri" }, "description": "Absolute URLs to scrape, processed in order.", "examples": [ [ "https://example.com/products/1", "https://example.com/products/2" ] ] } } } ] }, "SearchRequest": { "title": "SearchRequest", "description": "Body of `POST /api/v1/search`. Scrape options (used only when `scrape` is `true`) may be nested under `scrapeOptions` or given at the top level; `scrapeOptions` takes precedence when present.", "allOf": [ { "$ref": "#/components/schemas/ScrapeOptions" }, { "type": "object", "required": [ "query" ], "properties": { "query": { "type": "string", "minLength": 1, "maxLength": 500, "description": "Search query.", "examples": [ "web scraping api for llms" ] }, "limit": { "type": "integer", "minimum": 1, "maximum": 100, "default": 10, "description": "Maximum number of results. Cost is 2 credits per 10 results actually returned." }, "country": { "type": "string", "description": "Country to localise results to (ISO 3166-1 alpha-2, lower or upper case).", "examples": [ "us", "de" ] }, "lang": { "type": "string", "description": "Interface/result language (ISO 639-1).", "examples": [ "en", "de" ] }, "scrape": { "type": "boolean", "default": false, "description": "Also scrape every result with the scrape options. Each row then carries the full Document fields in addition to `title` and `description`." }, "scrapeOptions": { "$ref": "#/components/schemas/ScrapeOptions", "description": "Scrape options for the hits when `scrape` is `true`. Takes precedence over top-level scrape options when present." } } } ] }, "ExtractRequest": { "type": "object", "title": "ExtractRequest", "description": "Body of `POST /api/v1/extract`. Provide `prompt`, `schema` or both.", "required": [ "urls" ], "properties": { "urls": { "type": "array", "minItems": 1, "maxItems": 100, "items": { "type": "string", "format": "uri" }, "description": "Pages to extract from. Each page is processed separately; with one URL `data` is that page's object, with several it is an array in URL order.", "examples": [ [ "https://example.com/pricing" ] ] }, "prompt": { "type": "string", "description": "Natural-language instructions for what to extract.", "examples": [ "Extract every pricing plan with its monthly price in USD." ] }, "schema": { "type": "object", "additionalProperties": true, "description": "A JSON Schema describing the shape of the object to return. When present the model performs schema-guided extraction; otherwise it returns prompt-guided free-form JSON blocks.", "examples": [ { "type": "object", "properties": { "plans": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "priceUsd": { "type": "number" } } } } } } ] } } }, "JobAccepted": { "type": "object", "title": "JobAccepted", "description": "Returned with `202 Accepted` when a job has been validated and queued.", "required": [ "id", "status" ], "properties": { "id": { "type": "string", "description": "The job id. Use it with the `GET` endpoints to poll and read results.", "examples": [ "cmg7p4x2a0001l80bq3h9z1kd" ] }, "status": { "type": "string", "const": "queued", "description": "Always `queued` at creation time." } } }, "JobType": { "type": "string", "description": "Which operation created the job.", "enum": [ "scrape", "crawl", "map", "search", "extract", "batch_scrape" ] }, "JobStatusValue": { "type": "string", "description": "Lifecycle state of a job. `queued` → `running` → `succeeded` or `failed`; `canceled` when cancelled via `DELETE /api/v1/crawl/{id}`. `succeeded`, `failed` and `canceled` are terminal.", "enum": [ "queued", "running", "succeeded", "failed", "canceled" ] }, "JobInput": { "description": "The validated input that created the job, with defaults applied. Its shape depends on the job `type`. Scrape options are stored separately and are not included.", "anyOf": [ { "$ref": "#/components/schemas/ScrapeJobInput" }, { "$ref": "#/components/schemas/CrawlJobInput" }, { "$ref": "#/components/schemas/MapJobInput" }, { "$ref": "#/components/schemas/BatchScrapeJobInput" }, { "$ref": "#/components/schemas/SearchJobInput" }, { "$ref": "#/components/schemas/ExtractJobInput" } ] }, "ScrapeJobInput": { "type": "object", "title": "ScrapeJobInput", "description": "Stored input of a `scrape` job.", "required": [ "url" ], "properties": { "url": { "type": "string", "format": "uri" } } }, "CrawlJobInput": { "type": "object", "title": "CrawlJobInput", "description": "Stored input of a `crawl` job (defaults applied; `limit` already capped to the plan).", "required": [ "url", "maxDepth", "limit", "allowExternalLinks", "strategy" ], "properties": { "url": { "type": "string", "format": "uri" }, "maxDepth": { "type": "integer", "minimum": 0, "maximum": 10 }, "limit": { "type": "integer", "minimum": 1 }, "includePaths": { "type": "array", "items": { "type": "string" } }, "excludePaths": { "type": "array", "items": { "type": "string" } }, "allowExternalLinks": { "type": "boolean" }, "strategy": { "type": "string", "enum": [ "bfs", "dfs", "best_first" ] } } }, "MapJobInput": { "type": "object", "title": "MapJobInput", "description": "Stored input of a `map` job (defaults applied).", "required": [ "url", "limit", "includeSubdomains" ], "properties": { "url": { "type": "string", "format": "uri" }, "search": { "type": "string" }, "limit": { "type": "integer", "minimum": 1, "maximum": 100000 }, "includeSubdomains": { "type": "boolean" } } }, "BatchScrapeJobInput": { "type": "object", "title": "BatchScrapeJobInput", "description": "Stored input of a `batch_scrape` job.", "required": [ "urls" ], "properties": { "urls": { "type": "array", "items": { "type": "string", "format": "uri" } } } }, "SearchJobInput": { "type": "object", "title": "SearchJobInput", "description": "Stored input of a `search` job (defaults applied).", "required": [ "query", "limit", "scrape" ], "properties": { "query": { "type": "string" }, "limit": { "type": "integer", "minimum": 1, "maximum": 100 }, "country": { "type": "string" }, "lang": { "type": "string" }, "scrape": { "type": "boolean" } } }, "ExtractJobInput": { "type": "object", "title": "ExtractJobInput", "description": "Stored input of an `extract` job.", "required": [ "urls" ], "properties": { "urls": { "type": "array", "items": { "type": "string", "format": "uri" } }, "prompt": { "type": "string" }, "schema": { "type": "object", "additionalProperties": true } } }, "JobStatus": { "type": "object", "title": "JobStatus", "description": "Status of a job as returned by `GET /api/v1/jobs/{id}` (and embedded in the operation-specific `GET` responses).", "required": [ "id", "type", "status", "creditsUsed", "pageCount", "error", "input", "createdAt", "startedAt", "finishedAt", "hasResult" ], "properties": { "id": { "type": "string", "description": "Job id.", "examples": [ "cmg7p4x2a0001l80bq3h9z1kd" ] }, "type": { "$ref": "#/components/schemas/JobType" }, "status": { "$ref": "#/components/schemas/JobStatusValue" }, "creditsUsed": { "type": "integer", "minimum": 0, "description": "Credits charged for this job. `0` until the job has `succeeded` (failed and cancelled jobs are never charged)." }, "pageCount": { "type": "integer", "minimum": 0, "description": "Number of billable units produced — pages for crawl/batch, results for search, `1` for scrape/map/extract. `0` until the job has `succeeded`." }, "error": { "type": [ "string", "null" ], "description": "Failure message when `status` is `failed`; otherwise `null`." }, "input": { "$ref": "#/components/schemas/JobInput" }, "createdAt": { "type": "string", "format": "date-time", "description": "When the job was created (ISO 8601, UTC)." }, "startedAt": { "type": [ "string", "null" ], "format": "date-time", "description": "When a worker picked the job up; `null` while `queued`." }, "finishedAt": { "type": [ "string", "null" ], "format": "date-time", "description": "When the job reached a terminal state; `null` until then." }, "hasResult": { "type": "boolean", "description": "`true` once a single result is stored and `GET /api/v1/jobs/{id}/result` will succeed. Only scrape, map and extract jobs ever set this; multi-page jobs (crawl, batch, search) keep it `false` and are read through their paged endpoints." } } }, "Document": { "type": "object", "title": "Document", "description": "The result of scraping one URL. Only the fields for the requested `formats` are present; `url` and `metadata` are always present.", "required": [ "url", "metadata" ], "properties": { "url": { "type": "string", "format": "uri", "description": "The URL that was scraped (after redirects, when the engine reports it)." }, "markdown": { "type": "string", "description": "Page content as Markdown. Present when `markdown` was requested." }, "html": { "type": "string", "description": "Cleaned HTML. Present when `html` was requested." }, "rawHtml": { "type": "string", "description": "Original HTML as received. Present when `rawHtml` was requested." }, "links": { "type": "array", "items": { "type": "string" }, "description": "Absolute URLs linked from the page. Present when `links` was requested." }, "summary": { "type": "string", "description": "Short page summary (the page's meta description; empty string when it has none). Present when `summary` was requested." }, "screenshot": { "type": "string", "description": "Path of the PNG screenshot, relative to the API origin (e.g. `/api/v1/jobs/{id}/files/document.png`). Fetch it with your API key. Present when `screenshot` was requested and a screenshot was captured.", "examples": [ "/api/v1/jobs/cmg7p1h3b0000l80b5r8d2x9f/files/document.png" ] }, "metadata": { "type": "object", "additionalProperties": true, "description": "Page metadata as reported by the rendering engine: typically `title`, `description`, `language`, `keywords`, `author` and Open Graph / Twitter card properties (`og:title`, `og:image`, ...). Keys vary by page.", "examples": [ { "title": "Hello, world — Example Blog", "description": "This is the first post on our blog.", "language": "en", "og:title": "Hello, world" } ] }, "statusCode": { "type": "integer", "description": "HTTP status code the target responded with, when known.", "examples": [ 200 ] } } }, "SearchHit": { "type": "object", "title": "SearchHit", "description": "A search result row. When the search was run with `scrape: true` and the page could be scraped, the row is a full Document instead.", "required": [ "url", "title", "description" ], "properties": { "url": { "type": "string", "format": "uri" }, "title": { "type": "string" }, "description": { "type": [ "string", "null" ], "description": "Snippet shown by the search engine; `null` when it provided none." } } }, "FailedPage": { "type": "object", "title": "FailedPage", "description": "A batch-scrape row for a URL that could not be scraped. The job as a whole still succeeds.", "required": [ "url", "error" ], "properties": { "url": { "type": "string", "format": "uri" }, "error": { "type": "string", "description": "Why the scrape failed." } } }, "PageItem": { "description": "One row of a multi-page job: a scraped Document (crawl pages, successful batch URLs, scraped search hits), a SearchHit (search results that were not scraped or whose scrape failed) or a FailedPage (batch URLs that failed).", "anyOf": [ { "$ref": "#/components/schemas/Document" }, { "$ref": "#/components/schemas/SearchHit" }, { "$ref": "#/components/schemas/FailedPage" } ] }, "PagedResult": { "type": "object", "title": "PagedResult", "description": "One page of a multi-page job's rows, with offset-pagination bookkeeping. Rows are ordered by the time they were stored and appear while the job is still running.", "required": [ "status", "total", "completed", "creditsUsed", "data", "next" ], "properties": { "status": { "$ref": "#/components/schemas/JobStatusValue" }, "total": { "type": "integer", "minimum": 0, "description": "Total number of rows stored so far (grows while the job is running)." }, "completed": { "type": "integer", "minimum": 0, "description": "Final billable unit count (same as `pageCount`); `0` until the job has `succeeded`." }, "creditsUsed": { "type": "integer", "minimum": 0, "description": "Credits charged so far (`0` until the job has `succeeded`)." }, "data": { "type": "array", "items": { "$ref": "#/components/schemas/PageItem" }, "description": "The rows in this page, at most `limit` of them." }, "next": { "type": [ "integer", "null" ], "description": "`skip` value to request the next page, or `null` when this page reaches the end of the rows stored so far. For a job that is still running, poll again later with the same `skip` to pick up new rows." } } }, "PagedJob": { "title": "PagedJob", "description": "Response of `GET /api/v1/crawl/{id}` and `GET /api/v1/batch/scrape/{id}`: the job status merged with one page of results.", "allOf": [ { "$ref": "#/components/schemas/JobStatus" }, { "$ref": "#/components/schemas/PagedResult" } ] }, "ExtractJob": { "title": "ExtractJob", "description": "Response of `GET /api/v1/extract/{id}`: the job status plus the extraction output.", "allOf": [ { "$ref": "#/components/schemas/JobStatus" }, { "type": "object", "required": [ "data" ], "properties": { "data": { "oneOf": [ { "$ref": "#/components/schemas/ExtractResult" }, { "type": "null" } ], "description": "The stored ExtractResult once the job has `succeeded`; `null` while it is queued or running, and after a failure." } } } ] }, "ExtractResult": { "type": "object", "title": "ExtractResult", "description": "Output of an extract job.", "required": [ "sources" ], "properties": { "data": { "description": "The extracted data. For a single URL this is the extracted object (or array of blocks for prompt-only extraction); for several URLs it is an array with one entry per URL, in request order. Absent when the model returned nothing parseable.", "examples": [ { "plans": [ { "name": "Free", "priceUsd": 0 }, { "name": "Starter", "priceUsd": 29 } ] } ] }, "sources": { "type": "array", "items": { "type": "string", "format": "uri" }, "description": "The URLs the data was extracted from (the request's `urls`)." } } }, "MapResult": { "type": "object", "title": "MapResult", "description": "Output of a map job.", "required": [ "links" ], "properties": { "links": { "type": "array", "items": { "type": "string", "format": "uri" }, "description": "Discovered URLs, deduplicated, filtered by `search`/`includeSubdomains` and truncated to `limit`.", "examples": [ [ "https://example.com/", "https://example.com/about", "https://example.com/blog" ] ] } } }, "CancelResult": { "type": "object", "title": "CancelResult", "required": [ "status" ], "properties": { "status": { "type": "string", "const": "canceled" } } }, "ValidationIssue": { "type": "object", "title": "ValidationIssue", "description": "One validation problem (a Zod issue). Additional fields such as `expected`, `received`, `minimum`, `maximum` or `validation` are included depending on `code`.", "required": [ "code", "path", "message" ], "additionalProperties": true, "properties": { "code": { "type": "string", "description": "Issue kind, e.g. `invalid_type`, `invalid_string`, `too_small`, `too_big`, `invalid_enum_value`, `invalid_union_discriminator`.", "examples": [ "invalid_type" ] }, "path": { "type": "array", "items": { "type": [ "string", "integer" ] }, "description": "JSON path of the offending value within the request body, e.g. `[\"urls\", 2]`." }, "message": { "type": "string", "examples": [ "Required" ] } } }, "Error": { "type": "object", "title": "Error", "description": "Error envelope returned with every non-2xx response.", "required": [ "error" ], "properties": { "error": { "type": "string", "description": "Machine-readable error code.", "enum": [ "invalid_request", "unauthorized", "js_actions_disabled", "not_found", "quota_exceeded", "rate_limited", "internal_error" ] }, "issues": { "type": "array", "items": { "$ref": "#/components/schemas/ValidationIssue" }, "description": "Validation problems. Only with `invalid_request`." }, "scope": { "type": "string", "description": "The exhausted rate-limit bucket: `org:` or `domain:`. Only with `rate_limited`.", "examples": [ "org:cmg7nz0aa0000l80b1c2d3e4f", "domain:example.com" ] }, "detail": { "type": "string", "description": "Short diagnostic message. Only with `internal_error`." } } } } } }