Documentation / Error Reference

Error Reference

Every error the WebPeel API can return — with HTTP status codes, causes, and step-by-step fixes.

Error Format

All errors follow a consistent JSON envelope regardless of the endpoint. Check the HTTP status code first, then branch on error.type in your code.

{
  "success": false,
  "error": {
    "type": "rate_limited",
    "message": "Rate limit exceeded. You've made 26 requests in the current window.",
    "hint": "Wait 45 seconds or upgrade to Pro for 100/hr burst limit.",
    "docs": "https://webpeel.dev/docs/errors#rate-limited"
  },
  "metadata": {
    "requestId": "req_abc123"
  }
}

Field	Type	Description
`success`	`false`	Always `false` on error responses
`error.type`	string	Machine-readable error code — branch on this in your code
`error.message`	string	Human-readable explanation of what went wrong
`error.hint`	string?	Actionable suggestion for fixing the error (present on most errors)
`error.docs`	string?	Direct link to the relevant section of this page
`metadata.requestId`	string?	Unique request ID — include this in bug reports and support tickets

Retry-After Header

On 429 responses, the server also sends a Retry-After header with the number of seconds to wait before retrying. Always respect this header — ignoring it will extend your lockout window.

Quick Reference

Scan this table to quickly identify an error. Click any type to jump to the full description.

Error type	HTTP status	Retryable?	Description
`invalid_request`	400	❌ Fix request	Missing or invalid parameters
`invalid_url`	400	❌ Fix request	Malformed URL or URL exceeds 2048 chars
`unauthorized`	401	❌ Fix auth	Missing or invalid API key
`invalid_token`	401	❌ Fix auth	API token expired or revoked
`forbidden_url`	403	❌ Not allowed	SSRF attempt — localhost or private IP
`blocked`	403	⚡ Use stealth	Target site blocked automated access
`rate_limited`	429	⏳ After reset	Hourly burst limit exceeded
`network_error`	502	✅ Yes	Could not reach target URL
`timeout`	504	✅ Yes	Page load timed out
`internal_error`	500	✅ Yes	Unexpected server-side error

400 — Bad Request

`invalid_request`

400 Bad Request ❌ Fix your request — not retryable

What causes it: The request is missing a required parameter, or a parameter has an invalid value or type. This is always a client-side mistake.

Common triggers:

Missing the required url parameter on GET /v1/fetch
Invalid format value (must be markdown, text, or html)
wait or maxDepth is not a valid number, or out of range
Sending a malformed JSON body on a POST endpoint
actions array contains an unsupported action type

How to fix: Check the API Reference for required fields and accepted values. The error.message field will name the specific parameter that failed validation.

{
  "success": false,
  "error": {
    "type": "invalid_request",
    "message": "Missing required parameter: url",
    "hint": "Pass ?url=https://example.com as a query parameter or include 'url' in the request body.",
    "docs": "https://webpeel.dev/docs/errors#invalid-request"
  },
  "metadata": { "requestId": "req_abc123" }
}

`invalid_url`

400 Bad Request ❌ Fix your request — not retryable

What causes it: A url parameter was provided, but it cannot be parsed as a valid URL or exceeds the maximum allowed length of 2048 characters.

Common triggers:

URL is missing the scheme — use https://example.com, not example.com
URL length exceeds 2048 characters (common with data URIs or very long query strings)
Special characters in the URL were not percent-encoded (e.g., spaces, [, ])
URL contains a fragment identifier (#) — fragments are client-side only and cannot be fetched

How to fix: Ensure the URL starts with https:// or http:// and is properly encoded. Use encodeURIComponent() when passing a URL as a query parameter value.

{
  "success": false,
  "error": {
    "type": "invalid_url",
    "message": "URL exceeds the maximum allowed length of 2048 characters.",
    "hint": "Shorten the URL or remove unnecessary query parameters. Max length: 2048 chars.",
    "docs": "https://webpeel.dev/docs/errors#invalid-url"
  },
  "metadata": { "requestId": "req_abc124" }
}

401 — Unauthorized

`unauthorized`

401 Unauthorized ❌ Fix authentication

What causes it: The request reached a protected endpoint (such as GET /v1/stats or POST /mcp) without a valid API key, or the API key is malformed.

Common triggers:

No Authorization header or X-API-Key header was sent
API key was passed in the wrong format — must be Bearer wp_live_…
Typo or extra whitespace in the key value
Using a test-mode key (wp_test_…) against a production endpoint

How to fix: Add your API key via the Authorization: Bearer <key> header. You can find your key in the Dashboard. Most endpoints are also accessible without authentication — check the API Reference for which ones require auth.

# Correct usage
curl https://api.webpeel.dev/v1/stats \
  -H "Authorization: Bearer wp_live_abc123..."

{
  "success": false,
  "error": {
    "type": "unauthorized",
    "message": "This endpoint requires a valid API key.",
    "hint": "Pass your API key via 'Authorization: Bearer <key>' or 'X-API-Key: <key>'.",
    "docs": "https://webpeel.dev/docs/errors#unauthorized"
  },
  "metadata": { "requestId": "req_abc125" }
}

`invalid_token`

401 Unauthorized ❌ Refresh or replace token

What causes it: An API key was provided and is correctly formatted, but the token has expired, been revoked, or does not exist in the database.

Common triggers:

Key was manually revoked in the Dashboard
Account was deleted or suspended
Key was rotated and you're still using the old one
Hardcoded key in a deployment that is out-of-date

How to fix: Go to the Dashboard to generate a new API key. Update all deployments and environment variables that reference the old key. Avoid committing API keys to source control — use environment variables or a secrets manager.

{
  "success": false,
  "error": {
    "type": "invalid_token",
    "message": "The provided API key has been revoked or does not exist.",
    "hint": "Generate a new API key at https://app.webpeel.dev/settings/api-keys.",
    "docs": "https://webpeel.dev/docs/errors#invalid-token"
  },
  "metadata": { "requestId": "req_abc126" }
}

403 — Forbidden

`forbidden_url`

403 Forbidden ❌ Not allowed — SSRF protection

What causes it: The URL is syntactically valid but targets a resource that WebPeel is not allowed to fetch. This is SSRF (Server-Side Request Forgery) protection — it prevents the API from being used to probe internal infrastructure.

Blocked URL patterns:

Localhost and loopback addresses: localhost, 127.0.0.1, ::1
Private IP ranges: 10.x.x.x, 172.16–31.x.x, 192.168.x.x
Link-local addresses: 169.254.x.x, fe80::/10
Non-HTTP/HTTPS schemes: ftp://, file://, gopher://, etc.
Hostnames that resolve to private IPs after DNS lookup

How to fix: Only pass publicly reachable URLs starting with https:// or http://. If you are self-hosting WebPeel and need to fetch internal services, you can disable SSRF protection via the DISABLE_SSRF_PROTECTION=true environment variable — never do this on a publicly accessible instance.

{
  "success": false,
  "error": {
    "type": "forbidden_url",
    "message": "Fetching private or loopback addresses is not permitted.",
    "hint": "Only publicly routable URLs (http:// or https://) are allowed.",
    "docs": "https://webpeel.dev/docs/errors#forbidden-url"
  },
  "metadata": { "requestId": "req_abc127" }
}

`blocked`

403 Forbidden ⚡ Retry with stealth mode

What causes it: WebPeel successfully connected to the target server, but the server returned a bot-detection challenge or access-denied page (Cloudflare, PerimeterX, DataDome, Akamai, etc.) that could not be bypassed automatically. This corresponds to the BLOCKED internal error code.

Common triggers:

Site aggressively blocks datacenter IP ranges
Request requires JavaScript execution to pass a challenge page
Site detects headless browser fingerprints and returns a 403
Cloudflare "Under Attack Mode" or similar high-security configurations

How to fix:

Add render=true to the request — browser rendering solves most JS challenges
Add stealth=true for sites with advanced bot detection (uses a full stealth browser profile)
If both fail, the site may require session cookies or human interaction — consider if automated access is appropriate
Some sites explicitly prohibit crawling; check the site's robots.txt and Terms of Service

# Retry with browser rendering
curl "https://api.webpeel.dev/v1/fetch?url=https://example.com&render=true" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": false,
  "error": {
    "type": "blocked",
    "message": "The target site blocked this request. Bot detection was triggered.",
    "hint": "Retry with render=true or stealth=true to bypass JavaScript challenges.",
    "docs": "https://webpeel.dev/docs/errors#blocked"
  },
  "metadata": { "requestId": "req_abc128" }
}

429 — Too Many Requests

`rate_limited`

429 Too Many Requests ⏳ Retry after Retry-After header

What causes it: You've exceeded your plan's hourly burst limit. WebPeel enforces a sliding-window rate limit to protect the service and ensure fair usage for all users.

Check the Retry-After Header

Every 429 response includes a Retry-After header with the exact number of seconds until your burst window resets. Build this into your retry logic to avoid extended lockouts.

Rate limits by plan

Plan	Burst limit	Weekly quota	Price
Free	25 / hour	125 / week	$0
Starter	50 / hour	500 / week	$5/mo
Pro	100 / hour	1,250 / week	$9/mo
Max	250 / hour	6,250 / week	$29/mo

How to fix:

Read the Retry-After response header and wait that many seconds before retrying
Implement exponential backoff with jitter in your client code
Monitor X-Burst-Remaining on each response and throttle proactively before hitting the limit
Cache responses locally for frequently fetched URLs — use maxAge to control freshness
Upgrade your plan for higher burst limits

{
  "success": false,
  "error": {
    "type": "rate_limited",
    "message": "Rate limit exceeded. You've made 26 requests in the current window.",
    "hint": "Wait 45 seconds or upgrade to Pro for 100/hr burst limit.",
    "docs": "https://webpeel.dev/docs/errors#rate-limited"
  },
  "metadata": {
    "requestId": "req_abc129"
  }
}

Relevant response headers:

Header	Description
`Retry-After`	Seconds until the burst window resets
`X-Burst-Limit`	Your plan's hourly burst limit
`X-Burst-Used`	Requests made in the current window
`X-Burst-Remaining`	Requests remaining in the current window
`X-RateLimit-Reset`	Unix timestamp when the window resets

502 — Bad Gateway

`network_error`

502 Bad Gateway ✅ Retryable with backoff

What causes it: A network-level error occurred while WebPeel attempted to reach the target URL. The request never completed. This corresponds to the NETWORK internal error code.

Common triggers:

DNS resolution failure — domain does not exist or has no A/AAAA record
Connection refused — server is not listening on port 80 or 443
TLS/SSL handshake failure — invalid, expired, or self-signed certificate
Connection reset mid-response — server closed the socket unexpectedly
Target server is offline, overloaded, or temporarily unreachable
Firewall or network policy blocking outbound connections from WebPeel's IP range

How to fix:

Verify the URL is correct and the site is currently online — open it in a browser or run curl -I <url>
For transient failures, retry with exponential backoff (start at 1 s, double each attempt, cap at 30 s)
For SSL certificate errors on self-hosted instances, ensure your cert is valid and not expired
If the error is systematic for a specific domain, the site may have blocked WebPeel's IP range — try render=true to route through a different network path

{
  "success": false,
  "error": {
    "type": "network_error",
    "message": "DNS resolution failed for 'example-nonexistent.xyz'.",
    "hint": "Verify the domain exists and is publicly accessible. Check for typos in the URL.",
    "docs": "https://webpeel.dev/docs/errors#network-error"
  },
  "metadata": { "requestId": "req_abc130" }
}

504 — Gateway Timeout

`timeout`

504 Gateway Timeout ✅ Retryable

What causes it: The target page did not fully load within WebPeel's fetch timeout. This corresponds to the TIMEOUT internal error code. Timeouts vary by fetch mode: 30 s for simple HTTP, 60 s for browser rendering.

Common triggers:

Target server is slow or temporarily overloaded
JavaScript-heavy SPA that never fires a load or networkidle event
Page contains long-running scripts or infinite polling loops
Server-side rendering that takes too long (e.g., complex database queries)
Very large pages with many assets and no streaming

How to fix:

Retry the request — the timeout may be transient due to a momentary spike in server load
For JS-heavy SPAs, pass render=true and tune the wait parameter (milliseconds after page load event) to give dynamic content time to settle
If the site consistently times out, try fetching during off-peak hours
For very slow pages, consider whether a lighter endpoint (e.g., the site's own API or RSS feed) is available

# Allow extra wait time for slow SPAs
curl "https://api.webpeel.dev/v1/fetch?url=https://slow-spa.com&render=true&wait=5000" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": false,
  "error": {
    "type": "timeout",
    "message": "Page load timed out after 30000ms.",
    "hint": "Try render=true with a higher wait value (e.g., wait=5000) for JS-heavy pages.",
    "docs": "https://webpeel.dev/docs/errors#timeout"
  },
  "metadata": { "requestId": "req_abc131" }
}

500 — Internal Server Error

`internal_error`

500 Internal Server Error ✅ Retryable — likely transient

What causes it: An unexpected error occurred inside WebPeel's infrastructure. This is not caused by your request and indicates a bug or transient infrastructure failure.

Common triggers:

A bug in WebPeel's content extraction pipeline triggered by an unusual page structure
Out-of-memory condition during browser rendering of an unusually large page
Transient database or cache connection failure
Brief infrastructure blip during a deployment

How to fix:

Retry the request — most 500 errors are transient and resolve on the next attempt
Implement retry logic with exponential backoff (e.g., 3 retries at 1 s, 2 s, 4 s)
Check webpeel.dev/status for any active incidents
If the error persists for the same URL, open a bug report and include the metadata.requestId from the response

{
  "success": false,
  "error": {
    "type": "internal_error",
    "message": "An unexpected error occurred. Our team has been notified.",
    "hint": "Retry the request. If this persists, check https://webpeel.dev/status.",
    "docs": "https://webpeel.dev/docs/errors#internal-error"
  },
  "metadata": { "requestId": "req_abc132" }
}

Handling Errors in Code

Use the pattern below to handle all error types robustly. Branch on error.type and implement retry logic for transient errors.

async function fetchWithRetry(url, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const resp = await fetch(
      `https://api.webpeel.dev/v1/fetch?url=${encodeURIComponent(url)}`,
      { headers: { 'Authorization': 'Bearer YOUR_API_KEY' } }
    );

    if (resp.ok) return resp.json();

    const body = await resp.json();
    const { type, hint } = body.error;

    switch (type) {
      // Client errors — don't retry, fix the request
      case 'invalid_request':
      case 'invalid_url':
      case 'unauthorized':
      case 'invalid_token':
      case 'forbidden_url':
        throw new Error(`[${type}] ${body.error.message}\nHint: ${hint}`);

      // Blocked — retry with stealth mode
      case 'blocked':
        if (attempt === 0) {
          url = url + (url.includes('?') ? '&' : '?') + 'render=true&stealth=true';
          break;
        }
        throw new Error(`[blocked] Site blocked all fetch strategies.`);

      // Rate limited — wait for the reset window
      case 'rate_limited': {
        const retryAfter = parseInt(resp.headers.get('Retry-After') ?? '60', 10);
        console.warn(`Rate limited. Waiting ${retryAfter}s...`);
        await new Promise(r => setTimeout(r, retryAfter * 1000));
        break;
      }

      // Transient — exponential backoff
      case 'network_error':
      case 'timeout':
      case 'internal_error': {
        if (attempt === maxRetries - 1) throw new Error(`[${type}] ${body.error.message}`);
        const wait = Math.pow(2, attempt) * 1000;
        console.warn(`Transient error (${type}). Retrying in ${wait}ms...`);
        await new Promise(r => setTimeout(r, wait));
        break;
      }

      default:
        throw new Error(`[${type}] ${body.error.message}`);
    }
  }
}

import urllib.request, urllib.parse, json, time

def fetch_with_retry(url, max_retries=3, api_key="YOUR_API_KEY"):
    for attempt in range(max_retries):
        encoded = urllib.parse.quote(url, safe="")
        req = urllib.request.Request(
            f"https://api.webpeel.dev/v1/fetch?url={encoded}"
        )
        req.add_header("Authorization", f"Bearer {api_key}")

        try:
            with urllib.request.urlopen(req) as resp:
                return json.loads(resp.read())
        except urllib.error.HTTPError as e:
            body = json.loads(e.read())
            error_type = body["error"]["type"]
            message = body["error"]["message"]

            if error_type in ("invalid_request", "invalid_url",
                              "unauthorized", "invalid_token", "forbidden_url"):
                # Client errors — don't retry
                raise ValueError(f"[{error_type}] {message}") from e

            elif error_type == "blocked":
                if attempt == 0:
                    url += ("&" if "?" in url else "?") + "render=true"
                    continue
                raise RuntimeError("Site blocked all fetch strategies.") from e

            elif error_type == "rate_limited":
                retry_after = int(e.headers.get("Retry-After", 60))
                print(f"Rate limited. Waiting {retry_after}s...")
                time.sleep(retry_after)

            elif error_type in ("network_error", "timeout", "internal_error"):
                if attempt == max_retries - 1:
                    raise RuntimeError(f"[{error_type}] {message}") from e
                wait = 2 ** attempt
                print(f"Transient error ({error_type}). Retrying in {wait}s...")
                time.sleep(wait)

            else:
                raise RuntimeError(f"[{error_type}] {message}") from e

    return None

#!/usr/bin/env bash

fetch_url() {
  local URL="$1"
  local MAX_RETRIES=3
  local ATTEMPT=0

  while [ $ATTEMPT -lt $MAX_RETRIES ]; do
    RESPONSE=$(curl -s -w "\n%{http_code}" \
      "https://api.webpeel.dev/v1/fetch?url=$(python3 -c "import urllib.parse,sys; print(urllib.parse.quote(sys.argv[1]))" "$URL")" \
      -H "Authorization: Bearer YOUR_API_KEY")

    HTTP_CODE=$(echo "$RESPONSE" | tail -1)
    BODY=$(echo "$RESPONSE" | sed '$d')

    if [ "$HTTP_CODE" -eq 200 ]; then
      echo "$BODY"
      return 0
    fi

    ERROR_TYPE=$(echo "$BODY" | python3 -c "import sys,json; print(json.load(sys.stdin)['error']['type'])")

    case "$ERROR_TYPE" in
      invalid_request|invalid_url|unauthorized|invalid_token|forbidden_url)
        echo "Non-retryable error: $ERROR_TYPE" >&2
        return 1
        ;;
      blocked)
        URL="${URL}$(echo $URL | grep -q '?' && echo '&' || echo '?')render=true"
        ;;
      rate_limited)
        RETRY_AFTER=$(echo "$BODY" | python3 -c "import sys,json; d=json.load(sys.stdin); print(60)")
        echo "Rate limited. Waiting ${RETRY_AFTER}s..." >&2
        sleep "$RETRY_AFTER"
        ;;
      network_error|timeout|internal_error)
        WAIT=$((2 ** ATTEMPT))
        echo "Transient error ($ERROR_TYPE). Retrying in ${WAIT}s..." >&2
        sleep "$WAIT"
        ;;
    esac

    ATTEMPT=$((ATTEMPT + 1))
  done

  echo "Max retries exceeded." >&2
  return 1
}

fetch_url "https://example.com"