Documentation / MCP Server

MCP Server

Connect WebPeel to Claude Desktop, Cursor, Windsurf, OpenClaw, and other MCP-compatible tools. Give your AI assistant powerful web scraping capabilities.

What is MCP?

The Model Context Protocol (MCP) is an open standard developed by Anthropic that lets AI assistants access external tools and data sources. With WebPeel's MCP server, your AI can:

Quick Start

Connect to WebPeel via MCP in two ways:

Hosted MCP endpoint (recommended)

Use the hosted endpoint at https://api.webpeel.dev/mcp. Any MCP client that supports Streamable HTTP can connect with just:

{
  "url": "https://api.webpeel.dev/mcp",
  "headers": {
    "Authorization": "Bearer <key>"
  }
}

Stateless: no session management required.

Local MCP server

Run WebPeel's MCP server locally via npx — no installation required.

Start the Server

npx webpeel mcp

The local server runs on stdio and is automatically managed by your MCP client.

Configuration

Add WebPeel to your MCP client's configuration file:

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%/Claude/claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"]
    }
  }
}

Restart Claude Desktop to load the server.

Edit ~/.cursor/mcp.json:

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"]
    }
  }
}

Restart Cursor to apply changes.

Open VS Code Settings (JSON) and add to your cline.mcpServers config, or edit ~/.vscode/cline_mcp_settings.json:

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"]
    }
  }
}

Restart Cline or reload the VS Code window to load the server.

Edit ~/.windsurf/mcp_server_config.json:

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"]
    }
  }
}

Restart Windsurf to load the server.

Edit ~/.openclaw/mcp.json:

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"]
    }
  }
}

Restart OpenClaw or reload the MCP configuration.

Claude Code (CLI)

For Claude Code, use the one-liner:

claude mcp add webpeel -- npx -y webpeel mcp

Using an API Key

To use WebPeel's hosted API instead of local scraping, set the WEBPEEL_API_KEY environment variable:

{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"],
      "env": {
        "WEBPEEL_API_KEY": "your-api-key-here"
      }
    }
  }
}
{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"],
      "env": {
        "WEBPEEL_API_KEY": "your-api-key-here"
      }
    }
  }
}
{
  "mcpServers": {
    "webpeel": {
      "command": "npx",
      "args": ["webpeel", "mcp"],
      "env": {
        "WEBPEEL_API_KEY": "your-api-key-here"
      }
    }
  }
}

Smithery Installation

Install via Smithery registry (one-command setup):

npx @smithery/cli install @webpeel/mcp

Available Tools

WebPeel provides 11 core MCP tools (supported by the hosted MCP endpoint): webpeel_fetch, webpeel_search, webpeel_crawl, webpeel_map, webpeel_extract, webpeel_batch, webpeel_agent, webpeel_screenshot, webpeel_brand, webpeel_summarize, webpeel_answer.

The local MCP server may expose additional tools depending on your WebPeel version.

webpeel_fetch

Fetch a URL and return clean, AI-ready markdown content.

Parameters

Example

// Ask your AI assistant:
"Fetch https://example.com and extract the main article"

// With stealth mode:
"Fetch https://protected-site.com with stealth mode enabled"

// Extract specific content:
"Fetch https://news.com and only show me the article content, excluding nav and ads"

Search the web (DuckDuckGo by default, or Brave Search with a BYOK API key) and return results with titles, URLs, and snippets.

Parameters

Example

// Ask your AI assistant:
"Search for 'docker containers best practices'"

// Then fetch the top result:
"Fetch the first result and summarize it"

webpeel_crawl

Crawl a website starting from a URL, following links and extracting content. Perfect for documentation sites or building knowledge bases.

Parameters

Example

// Ask your AI assistant:
"Crawl https://docs.example.com up to 50 pages and build a summary"

webpeel_map

Discover all URLs on a domain using sitemap.xml and link crawling. Returns URLs without fetching content.

Parameters

Example

// Ask your AI assistant:
"Map all URLs on https://example.com/docs/"

webpeel_extract

Extract structured data from a webpage using CSS selectors, JSON schema, or AI-powered extraction.

Parameters

Example

// Ask your AI assistant:
"Extract the product title, price, and rating from https://shop.com/product/123"

// With AI extraction:
"Extract all the features mentioned on https://example.com/product as a bullet list"

webpeel_batch

Fetch multiple URLs in batch with concurrency control.

Parameters

Example

// Ask your AI assistant:
"Fetch these 5 URLs and compare their main content: [url1, url2, url3, url4, url5]"

webpeel_agent

Run WebPeel’s research agent: search → fetch → analyze → answer. Supports streaming and structured output via JSON Schema.

Parameters

Example

// Ask your AI assistant:
"Use webpeel_agent to research the current best practices for SSE in Node.js.\nDepth: thorough. Topic: technical. Return JSON with fields: summary, pitfalls, and examples."

Additional tools (local MCP server only, if enabled):

webpeel_brand

Extract branding and design system from a URL. Returns colors, fonts, and visual identity.

Parameters

Example

// Ask your AI assistant:
"What colors and fonts does https://stripe.com use?"

webpeel_change_track

Track changes on a URL by generating a content fingerprint.

Parameters

Example

// Ask your AI assistant:
"Generate a fingerprint for https://example.com so I can detect changes later"

webpeel_summarize

Generate an AI-powered summary of a webpage using an LLM.

Parameters

Example

// Ask your AI assistant:
"Summarize https://longform-article.com in a few sentences"

Usage Tips

When to Use Render Mode

When to Use Stealth Mode

Best Practices

  1. Start simple — Use basic fetch first, escalate to render/stealth only if needed
  2. Use selectors — Extract only what you need with selector and exclude
  3. Set maxTokens — Prevent large pages from overwhelming your context window
  4. Batch similar requests — Use webpeel_batch for multiple URLs from the same site
  5. Respect rate limits — Add delays when crawling large sites

Troubleshooting

Server not starting

Check your MCP client's logs. Common issues:

Tools not appearing

  1. Restart your MCP client
  2. Check the config file syntax
  3. Verify the server starts: npx webpeel mcp

Fetch timing out

Next Steps

💻

CLI

Use WebPeel from the command line with 18+ commands
📖

API Reference

Complete REST API documentation for programmatic access
🐳

Self-Hosting

Run your own WebPeel API server with Docker
🐙

GitHub

View source code, report issues, and contribute