# WebPeel Documentation

WebPeel is an open-source web intelligence tool for AI agents. It converts any webpage into clean, AI-ready Markdown with smart token budgeting (use budget=4000 to get ~4K tokens from any page), automatic content pruning (83% size reduction), and browser rendering for JavaScript-heavy sites. Includes web search, crawling, batch fetching, deep research, YouTube transcripts, domain-aware extractors, LLM-free Q&A, and an MCP server with 18 tools for AI assistants.

## Key Features
- **Smart token budget** — `budget=4000` distills any page to ~4K tokens (strips boilerplate, compresses tables)
- Web scraping with smart escalation (HTTP → browser → stealth)
- Content density pruning (83% size reduction on typical pages)
- BM25 query-focused filtering
- Deep research agent (multi-source, synthesized reports)
- Page actions (click, scroll, type, fill, wait)
- JSON Schema extraction (LLM-powered)
- Browser profiles (persistent login sessions)
- Domain-aware extractors (Twitter/X, Reddit, GitHub, Hacker News)
- YouTube transcript extraction (no API key needed)
- LLM-free quick answer with confidence scoring
- Proxy support (HTTP/HTTPS/SOCKS5)

## Getting Started
https://webpeel.dev/docs/getting-started

## CLI Reference
https://webpeel.dev/docs/cli

## API Reference
https://webpeel.dev/docs/api-reference

## MCP Server (18 tools)
https://webpeel.dev/docs/mcp

## Deep Research
https://webpeel.dev/docs/research

## Token Efficiency
https://webpeel.dev/docs/token-efficiency

## SDKs & Integrations
- Node.js: `npm install webpeel`
- Python: `pip install webpeel`
- MCP: stdio or HTTP Streamable transport
- Firecrawl-compatible API at `/v1/scrape` and `/v1/crawl` — switch by changing the base URL from `https://api.firecrawl.dev` to `https://api.webpeel.dev`

## Migrating from Firecrawl
WebPeel is a drop-in replacement for Firecrawl's `/v1/scrape` and `/v1/crawl` endpoints. No code changes needed beyond swapping the base URL and API key. Migration guide: https://webpeel.dev/docs/migrate-from-firecrawl

## Self-Hosting
https://webpeel.dev/docs/self-hosting

## Changelog
https://webpeel.dev/changelog