Sign inStart free

Documentation

Everything in Botnom is an async job: you create one, poll it or wait for a webhook, then read the result. The six operations work the same way over REST, the SDKs and MCP. Every endpoint is described in detail in the API reference.

Quickstart

  1. Create a free account. Your first organization is created with it.
  2. Open API Keys in the dashboard and create a key. Keys start with bn_ and belong to one organization.
  3. Send a request:
curl -X POST https://botnom.com/api/v1/scrape \
  -H "Authorization: Bearer $BOTNOM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown","links"]}'
# → 202 {"id":"cmu…","status":"queued"}

curl https://botnom.com/api/v1/jobs/cmu… -H "Authorization: Bearer $BOTNOM_API_KEY"
# → {"status":"succeeded","pageCount":1,"creditsUsed":1,"hasResult":true,…}

curl https://botnom.com/api/v1/jobs/cmu…/result -H "Authorization: Bearer $BOTNOM_API_KEY"
# → {"url":"https://example.com","title":"Example Domain","markdown":"# Example Domain…","links":[…]}

Instead of polling, you can add a webhook and get a signed POST when the job finishes. See Webhooks.

Jobs and results

Every create endpoint returns 202 with { id, status: "queued" }. A job then moves from queued to running and ends as succeeded, failed or canceled.

EndpointReturns
GET /api/v1/jobs/:idStatus, pageCount, creditsUsed, timestamps, error and hasResult
GET /api/v1/jobs/:id/resultThe single result of a scrape, map or extract job, as JSON
GET /api/v1/crawl/:id?skip=&limit=A page of documents from a crawl. The same endpoint works for search jobs, and /api/v1/batch/scrape/:id for batches
GET /api/v1/jobs/:id/files/:pathStored files, such as document.png for a scrape screenshot or pages/3.png for a crawled page
DELETE /api/v1/crawl/:idCancels a job. A queued job won’t start; a running job isn’t interrupted, but it stays cancelled and isn’t charged

Formats and options

formats can include markdown (the default), html, rawHtml, links, screenshot and summary. Any combination costs the same single credit.

OptionDefaultWhat it does
onlyMainContenttrueRemoves navigation, footers and other boilerplate
includeTags / excludeTags—CSS selectors to keep or drop
waitFor0Milliseconds to wait before capturing the page (up to 60,000)
timeout60,000Time limit per page, in milliseconds
actions—Steps to run first: wait, click, write, press, scroll and executeJavascript

Running raw JavaScript with executeJavascript is turned off on the hosted service.

Scrape

POST /api/v1/scrape
{ "url": "https://example.com/pricing",
  "formats": ["markdown", "screenshot"],
  "actions": [{ "type": "click", "selector": "#annual" }, { "type": "wait", "milliseconds": 500 }] }

Crawl and map

POST /api/v1/crawl
{ "url": "https://docs.example.com",
  "limit": 500, "maxDepth": 3,
  "includePaths": ["/docs/"], "excludePaths": ["/docs/changelog"],
  "strategy": "best_first",
  "scrapeOptions": { "formats": ["markdown"] } }

POST /api/v1/map
{ "url": "https://example.com", "search": "pricing", "limit": 5000, "includeSubdomains": true }

Crawled pages become readable as soon as each one finishes, so you can page through /api/v1/crawl/:id while the crawl is still running, or watch it live in the dashboard. The page limit is capped by your plan.

POST /api/v1/search
{ "query": "site:docs.python.org asyncio gather", "limit": 10, "scrape": true,
  "scrapeOptions": { "formats": ["markdown"] } }

Extract

POST /api/v1/extract
{ "urls": ["https://example.com/pricing"],
  "prompt": "List every plan with its monthly price and included pages",
  "schema": { "type": "object", "properties": { "plans": { "type": "array",
    "items": { "type": "object", "properties": {
      "name": { "type": "string" }, "price": { "type": "number" }, "pages": { "type": "integer" } } } } } } }

Extraction runs on the model your organization’s admin selected in the admin console, and every prompt is recorded in the AI call log with its token count and cost.

SDKs

Official TypeScript and Python SDKs are in progress and haven’t been published yet. Until they are, any HTTP client works: create a job, then poll /api/v1/jobs/:id or use a webhook, as in the quickstart. The OpenAPI spec can also be used to generate a client in most languages.

MCP server

The endpoint is POST https://botnom.com/api/mcp (Streamable HTTP), authenticated with your API key on each request. It provides botnom_scrape, botnom_crawl, botnom_map, botnom_search, botnom_extract, botnom_batch_scrape, botnom_get_job and botnom_get_result.

{
  "mcpServers": {
    "botnom": {
      "url": "https://botnom.com/api/mcp",
      "headers": { "Authorization": "Bearer bn_your_key" }
    }
  }
}

The MCP page in your dashboard has ready-to-copy settings for Claude Desktop, Cursor and other clients.

Webhooks

Add an endpoint under Webhooks. Job events are named after the operation and the event, for example crawl.completed, extract.failed, scrape.started or crawl.page. Subscribe to completed or failed (or job.completed and job.failed) to match every operation, and to monitor.changed for monitors. Leave the list empty to receive everything.

Each delivery is signed with an HMAC of the body in the X-Botnom-Signature header, using the endpoint’s secret. Check the signature before you trust the payload.

Schedules and monitoring

Under Schedules you can run any operation on a cron schedule with five or six fields. Turn on Track changes to make it a monitor: Botnom stores a snapshot of the result, compares each run with the previous one, keeps a history of changes, and sends monitor.changed with the number of added and removed lines and a preview.

Credits and limits

Scrape and map cost 1 credit, crawls 1 credit per page crawled, batch jobs 1 credit per URL, search 2 credits per 10 results and extract 5 credits per request. Each plan’s monthly credits, requests per minute and maximum crawl size are on the pricing page. Going over a limit returns 429 with rate_limited or quota_exceeded and a Retry-After header. Requests to any single domain are also rate-limited across all customers.

Documentation · Botnom