WebPeel Documentation

Fast web fetching for AI agents. Smart escalation from HTTP to headless browser with stealth mode.

Quick Start

Get up and running with WebPeel in 30 seconds.

Installation

npm install -g webpeel

pnpm add -g webpeel

yarn global add webpeel

pip install webpeel

First Scrape

# Fetch any URL
npx webpeel https://example.com

# Output as JSON
npx webpeel https://example.com --json

# Use browser rendering for JS-heavy sites
npx webpeel https://example.com --render

import { peel } from 'webpeel';

const result = await peel('https://example.com');
console.log(result.title);
console.log(result.content); // Markdown

from webpeel import WebPeel

client = WebPeel()
result = client.scrape("https://example.com")
print(result.title)
print(result.content)

Core Concepts

Smart Escalation

WebPeel automatically chooses the fastest method:

1. HTTP Fetch — Fast HTTP request with smart headers (~118ms)
2. Browser Mode — Headless browser for JavaScript-heavy sites (~2s)
3. Stealth Mode — Full anti-detection for protected sites (~5s)

// Simple fetch (HTTP)
const result1 = await peel('https://example.com');

// Force browser rendering
const result2 = await peel('https://example.com', { render: true });

// Use stealth mode for Cloudflare-protected sites
const result3 = await peel('https://example.com', { stealth: true });

Output Formats

Choose the format that works best for your use case:

const result = await peel('https://example.com', {
  format: 'markdown' // Default
});
// Perfect for LLMs — clean, structured content

const result = await peel('https://example.com', {
  format: 'text'
});
// Plain text — no formatting, just content

const result = await peel('https://example.com', {
  format: 'html'
});
// Raw HTML — full page source

Content Filtering

Extract only what you need:

// Extract specific content with CSS selector
const result = await peel('https://example.com', {
  selector: 'article',
  exclude: ['.sidebar', '.ads', 'nav']
});

// Only include main content tags
const result2 = await peel('https://example.com', {
  includeTags: ['article', 'main'],
  excludeTags: ['nav', 'footer', 'aside']
});

Advanced Features

Page Actions

Interact with pages before scraping:

const result = await peel('https://example.com', {
  actions: [
    { type: 'click', selector: '.load-more' },
    { type: 'wait', ms: 2000 },
    { type: 'scroll', to: 'bottom' },
    { type: 'type', selector: '#search', value: 'query' }
  ]
});

Structured Extraction

Extract data with CSS selectors or AI:

// CSS-based extraction
const result = await peel('https://example.com/product', {
  extract: {
    selectors: {
      title: 'h1',
      price: '.price',
      rating: '.rating'
    }
  }
});

// AI-powered extraction (BYOK)
const result2 = await peel('https://example.com', {
  extract: {
    prompt: 'Extract the main features as a list',
    llmApiKey: process.env.OPENAI_API_KEY
  }
});

Crawling

Crawl entire websites:

import { crawl } from 'webpeel';

const results = await crawl('https://example.com', {
  maxPages: 50,
  maxDepth: 2,
  excludePatterns: ['/admin/', '/login']
});

results.forEach(page => {
  console.log(page.title, page.url);
});

Change Tracking

Monitor content changes over time:

import { trackChange } from 'webpeel';

const change = await trackChange('https://example.com');
if (change.changed) {
  console.log('Content changed!');
  console.log('Added:', change.added);
  console.log('Removed:', change.removed);
}

Web Search (DuckDuckGo + Brave BYOK)

WebPeel includes web search out of the box:

• DuckDuckGo — default, free, no key required
• Brave Search — bring your own API key for higher-quality results

# DuckDuckGo (default)
npx webpeel search "latest AI news"

# Brave Search (BYOK)
npx webpeel config set braveApiKey $BRAVE_KEY
npx webpeel search "latest AI news" --provider brave

Cited Answers (/v1/answer)

Need a single, cited answer instead of raw pages? Use the Answer endpoint (search + fetch + LLM → response with [1], [2] citations). BYOK — use your own LLM key.

npx webpeel answer "What is MCP?" \
  --llm openai \
  --llm-api-key $OPENAI_API_KEY \
  --max-sources 5

Next Steps