API Reference

Complete REST API documentation for WebPeel. All endpoints return JSON and use standard HTTP status codes.

Authentication

Pass your API key via the Authorization header or the X-API-Key header:

# Option 1: Bearer token (recommended)
curl https://api.webpeel.dev/v1/fetch?url=https://example.com \
  -H "Authorization: Bearer wp_live_abc123..."

# Option 2: X-API-Key header
curl https://api.webpeel.dev/v1/fetch?url=https://example.com \
  -H "X-API-Key: wp_live_abc123..."

GET /v1/fetch

Fetch and extract clean, AI-ready content from any URL. WebPeel automatically escalates from fast HTTP to browser rendering to stealth mode as needed.

Query Parameters

Response

{
  "url": "https://example.com",
  "title": "Example Domain",
  "content": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "metadata": {
    "description": "Example Domain",
    "author": null,
    "published": null,
    "image": null,
    "canonical": "https://example.com"
  },
  "links": ["https://www.iana.org/domains/example"],
  "tokens": 125,
  "method": "simple",
  "elapsed": 118,
  "contentType": "html",
  "quality": 0.95,
  "fingerprint": "a1b2c3d4e5f6g7h8"
}

Examples

# Basic fetch
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com"

# With browser rendering
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com&render=true"

# Extract only main content, exclude nav/footer
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com&onlyMainContent=true&excludeTags=nav,footer"

# With authentication
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com" \
  -H "Authorization: Bearer YOUR_API_KEY"

const response = await fetch(
  'https://api.webpeel.dev/v1/fetch?url=https://example.com&format=markdown',
  {
    headers: { 'Authorization': 'Bearer YOUR_API_KEY' }
  }
);
const data = await response.json();

console.log(data.title);    // "Example Domain"
console.log(data.content);  // Markdown content
console.log(data.method);   // "simple" | "browser" | "stealth"

import urllib.request, json

url = "https://api.webpeel.dev/v1/fetch?url=https://example.com"
req = urllib.request.Request(url)
req.add_header("Authorization", "Bearer YOUR_API_KEY")

with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
    print(data["title"])    # "Example Domain"
    print(data["content"])  # Markdown content

Error Responses

POST /v1/fetch

Same as GET /v1/fetch but accepts a JSON body. Adds support for page actions, inline LLM extraction (BYOK), and the Firecrawl-compatible formats array.

Request Body (JSON)

Response

Same as GET /v1/fetch, plus these additional fields when inline extraction is used:

{
  "url": "https://example.com",
  "title": "Example Domain",
  "content": "# Example Domain\n...",
  "json": {
    "company": "Example Corp",
    "founded": 2006
  },
  "extractTokensUsed": { "input": 1234, "output": 56 },
  ...
}

Examples

# Basic POST fetch
curl -X POST https://api.webpeel.dev/v1/fetch \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://example.com",
    "format": "markdown",
    "onlyMainContent": true
  }'

# With inline LLM extraction
curl -X POST https://api.webpeel.dev/v1/fetch \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/about",
    "extract": {
      "schema": {
        "type": "object",
        "properties": {
          "company": { "type": "string" },
          "founded": { "type": "number" }
        }
      }
    },
    "llmProvider": "openai",
    "llmApiKey": "sk-..."
  }'

const response = await fetch('https://api.webpeel.dev/v1/fetch', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
  },
  body: JSON.stringify({
    url: 'https://example.com/about',
    extract: {
      schema: {
        type: 'object',
        properties: {
          company: { type: 'string' },
          founded: { type: 'number' }
        }
      }
    },
    llmProvider: 'openai',
    llmApiKey: 'sk-...'
  })
});

const data = await response.json();
console.log(data.json);  // { company: "Example Corp", founded: 2006 }

import urllib.request, json

body = json.dumps({
    "url": "https://example.com/about",
    "extract": {
        "schema": {
            "type": "object",
            "properties": {
                "company": {"type": "string"},
                "founded": {"type": "number"}
            }
        }
    },
    "llmProvider": "openai",
    "llmApiKey": "sk-..."
}).encode()

req = urllib.request.Request(
    "https://api.webpeel.dev/v1/fetch",
    data=body, method="POST"
)
req.add_header("Content-Type", "application/json")
req.add_header("Authorization", "Bearer YOUR_API_KEY")

with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
    print(data["json"])  # {"company": "Example Corp", "founded": 2006}

Page Actions

Page actions let you interact with a page (click, type, scroll, wait) before content is extracted. Actions auto-enable browser rendering. Compatible with Firecrawl's action format.

{
  "url": "https://example.com/search",
  "actions": [
    { "type": "fill", "selector": "#search-input", "value": "AI agents" },
    { "type": "click", "selector": "#search-btn" },
    { "type": "wait", "ms": 2000 },
    { "type": "scroll", "direction": "down", "amount": 500 }
  ]
}

GET /v1/search

Search the web using DuckDuckGo (free, default) or Brave Search (BYOK for higher-quality results). Optionally scrape each result URL for full content.

Query Parameters

Response

{
  "success": true,
  "data": {
    "web": [
      {
        "title": "Example Domain",
        "url": "https://example.com",
        "snippet": "This domain is for use in illustrative examples...",
        "content": "# Example Domain\n..."  // Only if scrapeResults=true
      }
    ],
    "news": [ ... ],   // Only if sources includes "news"
    "images": [ ... ]  // Only if sources includes "images"
  }
}

Examples

# Basic search
curl "https://api.webpeel.dev/v1/search?q=latest+AI+news&count=5"

# With Brave Search (BYOK)
curl "https://api.webpeel.dev/v1/search?q=latest+AI+news&provider=brave" \
  -H "x-search-api-key: YOUR_BRAVE_KEY"

# Search + scrape results
curl "https://api.webpeel.dev/v1/search?q=MCP+protocol&scrapeResults=true&count=3"

const resp = await fetch(
  'https://api.webpeel.dev/v1/search?q=latest+AI+news&count=5',
  { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } }
);
const { data } = await resp.json();

for (const result of data.web) {
  console.log(`${result.title}: ${result.url}`);
}

import urllib.request, json

url = "https://api.webpeel.dev/v1/search?q=latest+AI+news&count=5"
req = urllib.request.Request(url)
req.add_header("Authorization", "Bearer YOUR_API_KEY")

with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
    for r in data["data"]["web"]:
        print(f"{r['title']}: {r['url']}")

POST /v1/screenshot

Take a screenshot of any URL and return a base64-encoded image. Supports full-page capture, custom viewport sizes, page actions, and stealth mode.

Request Body (JSON)

Response

{
  "success": true,
  "data": {
    "url": "https://example.com",
    "screenshot": "data:image/png;base64,iVBORw0KGgo...",
    "metadata": {
      "sourceURL": "https://example.com",
      "format": "png",
      "width": 1280,
      "height": 720,
      "fullPage": false
    }
  }
}

Examples

curl -X POST https://api.webpeel.dev/v1/screenshot \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://example.com",
    "fullPage": true,
    "width": 1440,
    "format": "png"
  }'

const resp = await fetch('https://api.webpeel.dev/v1/screenshot', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
  },
  body: JSON.stringify({
    url: 'https://example.com',
    fullPage: true,
    width: 1440
  })
});

const { data } = await resp.json();
// data.screenshot is a data: URL — use directly in <img src="...">

import urllib.request, json, base64

body = json.dumps({
    "url": "https://example.com",
    "fullPage": True,
    "width": 1440
}).encode()

req = urllib.request.Request(
    "https://api.webpeel.dev/v1/screenshot",
    data=body, method="POST"
)
req.add_header("Content-Type", "application/json")
req.add_header("Authorization", "Bearer YOUR_API_KEY")

with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
    # Save screenshot to file
    img_data = data["data"]["screenshot"].split(",")[1]
    with open("screenshot.png", "wb") as f:
        f.write(base64.b64decode(img_data))

POST /v1/answer

Ask a question, search the web, fetch top sources, and generate an LLM answer with inline citations ([1], [2]). BYOK — you provide your own LLM API key. Supports SSE streaming.

Request Body (JSON)

Response (non-streaming)

{
  "answer": "MCP (Model Context Protocol) is an open standard... [1] ... [2] ...",
  "citations": [
    { "title": "Model Context Protocol", "url": "https://...", "snippet": "..." },
    { "title": "MCP Documentation", "url": "https://...", "snippet": "..." }
  ],
  "searchProvider": "duckduckgo",
  "llmProvider": "openai",
  "llmModel": "gpt-4o-mini",
  "tokensUsed": { "input": 3456, "output": 789 }
}

Response (streaming)

When stream=true, the endpoint returns text/event-stream with JSON-encoded events:

data: {"type":"chunk","text":"MCP (Model Context Protocol) is"}
data: {"type":"chunk","text":" an open standard for AI tool..."}
data: {"type":"done","citations":[...],"searchProvider":"duckduckgo","llmProvider":"openai","llmModel":"gpt-4o-mini","tokensUsed":{"input":3456,"output":789}}

Examples

curl -X POST https://api.webpeel.dev/v1/answer \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is MCP and why does it matter for AI agents?",
    "llmProvider": "openai",
    "llmApiKey": "sk-...",
    "maxSources": 5,
    "stream": false
  }'

const resp = await fetch('https://api.webpeel.dev/v1/answer', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    question: 'What is MCP and why does it matter for AI agents?',
    llmProvider: 'openai',
    llmApiKey: 'sk-...',
    maxSources: 5
  })
});

const data = await resp.json();
console.log(data.answer);
console.log(data.citations);

import urllib.request, json

body = json.dumps({
    "question": "What is MCP and why does it matter for AI agents?",
    "llmProvider": "openai",
    "llmApiKey": "sk-...",
    "maxSources": 5
}).encode()

req = urllib.request.Request(
    "https://api.webpeel.dev/v1/answer",
    data=body, method="POST"
)
req.add_header("Content-Type", "application/json")

with urllib.request.urlopen(req) as resp:
    data = json.loads(resp.read())
    print(data["answer"])

POST /v1/agent

Run an autonomous AI research agent. It plans search queries, fetches pages, and synthesizes a structured answer using your LLM (BYOK). Supports basic and thorough research depth, topic tuning, JSON Schema output, and SSE streaming.

Request Body (JSON)

Response (synchronous)

{
  "success": true,
  "data": {
    "answer": "## AI Coding Assistants Comparison\n...",
    "keyFindings": ["Finding 1", "Finding 2"]
  },
  "answer": "## AI Coding Assistants Comparison\n...",
  "sources": ["https://...", "https://..."],
  "sourcesDetailed": [
    { "url": "https://...", "title": "Source Title" }
  ],
  "pagesVisited": 5,
  "creditsUsed": 6,
  "tokensUsed": { "input": 12000, "output": 2500 }
}

SSE Streaming Events

When stream=true, the endpoint returns text/event-stream:

data: {"type":"step","action":"searching","query":"AI coding assistants 2026"}
data: {"type":"step","action":"fetching","url":"https://..."}
data: {"type":"step","action":"analyzing","summary":"Synthesizing from 5 sources..."}
data: {"type":"chunk","text":"## AI Coding Assistants\n"}
data: {"type":"chunk","text":"Here are the top 5..."}
data: {"type":"done","answer":"...","sources":[...],"tokensUsed":{...}}

Async Mode

For long-running research, use the async endpoint:

Same request body as above, plus optional webhook (WebhookConfig object). Returns a job ID:

{ "success": true, "id": "job-uuid", "url": "/v1/agent/job-uuid" }

Poll job status and results.

Cancel a running agent job.

Example

curl -X POST https://api.webpeel.dev/v1/agent \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "prompt": "Compare the top 5 AI coding assistants by features and pricing",
    "llmApiKey": "sk-...",
    "depth": "thorough",
    "topic": "technical",
    "maxSources": 10,
    "outputSchema": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "features": { "type": "array", "items": { "type": "string" } },
          "pricing": { "type": "string" }
        },
        "required": ["name", "features", "pricing"]
      }
    }
  }'

POST /v1/crawl

Start an async crawl job. Follows links from a starting URL and extracts content from each page. Returns a job ID for polling. Supports webhooks and SSE for real-time progress.

Request Body (JSON)

Response (202 Accepted)

{
  "success": true,
  "id": "crawl_abc123",
  "url": "/v1/crawl/crawl_abc123"
}

GET /v1/crawl/:id — Poll Results

Returns job status and results. Supports SSE by sending Accept: text/event-stream for real-time progress updates.

{
  "success": true,
  "status": "completed",
  "progress": 100,
  "total": 50,
  "completed": 50,
  "creditsUsed": 50,
  "expiresAt": "2026-02-16T00:00:00.000Z",
  "data": [
    {
      "url": "https://example.com",
      "title": "Home",
      "markdown": "# Home\n...",
      "links": ["https://example.com/about"]
    }
  ]
}

DELETE /v1/crawl/:id — Cancel Job

Cancel a queued or in-progress crawl job.

Example

# Start crawl
curl -X POST https://api.webpeel.dev/v1/crawl \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{ "url": "https://docs.example.com", "limit": 50, "maxDepth": 3 }'

# Poll results
curl "https://api.webpeel.dev/v1/crawl/crawl_abc123" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Cancel
curl -X DELETE "https://api.webpeel.dev/v1/crawl/crawl_abc123" \
  -H "Authorization: Bearer YOUR_API_KEY"

// Start crawl
const { id } = await (await fetch('https://api.webpeel.dev/v1/crawl', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
  },
  body: JSON.stringify({ url: 'https://docs.example.com', limit: 50 })
})).json();

// Poll until complete
let job;
do {
  await new Promise(r => setTimeout(r, 2000));
  job = await (await fetch(`https://api.webpeel.dev/v1/crawl/${id}`, {
    headers: { 'Authorization': 'Bearer YOUR_API_KEY' }
  })).json();
  console.log(`Progress: ${job.completed}/${job.total}`);
} while (job.status === 'processing' || job.status === 'queued');

console.log(`Done! ${job.data.length} pages crawled.`);

import urllib.request, json, time

# Start crawl
body = json.dumps({"url": "https://docs.example.com", "limit": 50}).encode()
req = urllib.request.Request(
    "https://api.webpeel.dev/v1/crawl", data=body, method="POST"
)
req.add_header("Content-Type", "application/json")
req.add_header("Authorization", "Bearer YOUR_API_KEY")

with urllib.request.urlopen(req) as resp:
    job_id = json.loads(resp.read())["id"]

# Poll until complete
while True:
    time.sleep(2)
    req = urllib.request.Request(f"https://api.webpeel.dev/v1/crawl/{job_id}")
    req.add_header("Authorization", "Bearer YOUR_API_KEY")
    with urllib.request.urlopen(req) as resp:
        job = json.loads(resp.read())
    if job["status"] in ("completed", "failed"):
        break
    print(f"Progress: {job['completed']}/{job['total']}")

print(f"Done! {len(job['data'])} pages.")

POST /v1/batch/scrape

Batch scrape multiple URLs concurrently. Creates an async job with up to 5 parallel fetches. Poll for results or use webhooks.

Request Body (JSON)

Response (202 Accepted)

{
  "success": true,
  "id": "batch_xyz789",
  "url": "/v1/batch/scrape/batch_xyz789"
}

GET /v1/batch/scrape/:id — Poll Results

Same response format as crawl jobs:

{
  "success": true,
  "status": "completed",
  "progress": 100,
  "total": 10,
  "completed": 10,
  "creditsUsed": 10,
  "data": [ { "url": "...", "content": "...", ... }, ... ]
}

DELETE /v1/batch/scrape/:id — Cancel Job

Cancel a running batch job.

Example

curl -X POST https://api.webpeel.dev/v1/batch/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "urls": [
      "https://example.com/page1",
      "https://example.com/page2",
      "https://example.com/page3"
    ],
    "formats": ["markdown"]
  }'

POST /v1/map

Discover all URLs on a domain using sitemap.xml and link crawling. Returns a flat list of URLs without fetching page content. Much faster and cheaper than a full crawl.

Request Body (JSON)

Response

{
  "success": true,
  "links": [
    "https://example.com/",
    "https://example.com/about",
    "https://example.com/blog",
    "https://example.com/blog/post-1",
    ...
  ]
}

Example

curl -X POST https://api.webpeel.dev/v1/map \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "limit": 100, "search": "/blog" }'

GET /v1/jobs

List all async jobs (crawl, batch, agent). Useful for dashboards and monitoring.

Query Parameters

Response

{
  "success": true,
  "count": 3,
  "jobs": [
    {
      "id": "crawl_abc123",
      "type": "crawl",
      "status": "completed",
      "progress": 100,
      "total": 50,
      "completed": 50,
      "creditsUsed": 50,
      "createdAt": "2026-02-15T10:00:00Z",
      "expiresAt": "2026-02-16T10:00:00Z"
    }
  ]
}

POST /v1/scrape (Firecrawl-compatible)

Drop-in replacement for Firecrawl's /v1/scrape endpoint. Change your base URL from api.firecrawl.dev to api.webpeel.dev and your existing code works — zero changes needed.

Request Body (Firecrawl format)

Response

{
  "success": true,
  "data": {
    "markdown": "# Page Title\n\nContent...",
    "metadata": {
      "title": "Page Title",
      "description": "...",
      "sourceURL": "https://example.com",
      "statusCode": 200
    },
    "links": ["https://..."],
    "screenshot": "data:image/png;base64,...",
    "json": { ... }
  }
}

POST /v2/scrape (Firecrawl v2)

Firecrawl v2 compatible endpoint with first-class screenshot support. When formats is ["screenshot"] only, it uses a dedicated screenshot pipeline for faster, higher-quality captures.

Accepts the same body as POST /v1/scrape plus these additional fields:

POST /mcp

Hosted MCP endpoint using Streamable HTTP transport. Connect any MCP client (Claude, Cursor, VS Code) with a single URL — no local server required.

Connection

{
  "url": "https://api.webpeel.dev/mcp",
  "headers": {
    "Authorization": "Bearer YOUR_API_KEY"
  }
}

Available Tools

Stateless: No session management required. Each request is authenticated and handled independently.

GET /mcp returns 405. DELETE /mcp returns 200 (no-op for stateless mode).

GET /health

Health check endpoint. Returns server status, version, and uptime. Not rate-limited — safe for monitoring services to poll frequently.

Response

{
  "status": "healthy",
  "version": "0.8.1",
  "uptime": 86400,
  "timestamp": "2026-02-15T21:00:00.000Z"
}

GET /v1/stats

Get usage statistics for the authenticated API key's account. Shows total requests, success rate, and average response time.

Response

{
  "totalRequests": 1523,
  "successRate": 98.2,
  "avgResponseTime": 342
}

GET /v1/cli/usage

Get detailed usage information for the CLI's webpeel usage command. Shows plan info, weekly quota usage, burst limits, and reset times.

Response

{
  "plan": {
    "tier": "pro",
    "weeklyLimit": 1250,
    "burstLimit": 100
  },
  "weekly": {
    "used": 342,
    "limit": 1250,
    "remaining": 908,
    "resetsAt": "2026-02-17T00:00:00.000Z",
    "percentUsed": 27
  },
  "burst": {
    "used": 12,
    "limit": 100,
    "resetsIn": "47m"
  },
  "canFetch": true,
  "upgradeUrl": "https://webpeel.dev/#pricing"
}

Webhooks

Async jobs (crawl, batch, agent) support webhook notifications. Webhooks are signed with HMAC-SHA256 if you provide a secret, and retried up to 3 times with exponential backoff.

Webhook Config Object

Webhook Payload

{
  "event": "completed",
  "timestamp": "2026-02-15T21:00:00.000Z",
  "data": {
    "jobId": "crawl_abc123",
    "total": 50,
    "completed": 50
  }
}

Rate Limits

WebPeel uses a two-tier rate limiting system: weekly quotas (soft limit) and hourly burst limits (hard limit).

Soft vs Hard Limits

• Weekly quota exceeded (soft limit): Requests are not blocked. Instead, they are degraded to HTTP-only mode (no browser rendering or stealth). The X-Soft-Limited header is set.
• Burst limit exceeded (hard limit): Returns 429 Too Many Requests with a Retry-After header.
• Extra usage: If enabled on your account, you can continue using full features beyond your weekly quota at per-request pricing.

Response Headers

WebPeel includes useful metadata in response headers:

Error Responses

All errors return a consistent JSON format:

{
  "error": "error_code",
  "message": "Human-readable error message"
}