Documentation / CLI

CLI Reference

Command-line interface for WebPeel. 18+ commands for web scraping, searching, and crawling.

Installation

npm install -g webpeel

Quick Start

# Fetch a URL
npx webpeel https://example.com

# Output as JSON
npx webpeel https://example.com --json

# Use browser rendering
npx webpeel https://example.com --render

# Search the web (DuckDuckGo default)
npx webpeel search "AI news"

# Search with Brave (BYOK)
npx webpeel search "AI news" --provider brave --search-api-key $BRAVE_KEY

# Get a cited answer (BYOK)
npx webpeel answer "What is MCP?" --llm openai --llm-api-key $OPENAI_API_KEY

Global Options

Option	Description
`-s, --silent`	Silent mode (no spinner, no progress)
`--json`	Output as JSON
`-h, --help`	Show help
`-V, --version`	Show version number

Commands

`webpeel <url>`

Fetch and extract content from a URL.

# Basic usage
npx webpeel https://example.com

# Output formats
npx webpeel https://example.com --html
npx webpeel https://example.com --text
npx webpeel https://example.com --json

# Browser rendering
npx webpeel https://example.com --render
npx webpeel https://example.com --stealth
npx webpeel https://example.com --wait 5000

# Content filtering
npx webpeel https://example.com --selector "article"
npx webpeel https://example.com --exclude ".sidebar" ".ads"
npx webpeel https://example.com --only-main-content

# Advanced options
npx webpeel https://example.com --screenshot
npx webpeel https://example.com --screenshot screenshot.png
npx webpeel https://example.com --full-page
npx webpeel https://example.com --max-tokens 5000
npx webpeel https://example.com --cache 1h

Options

Option	Type	Description
`--html`	boolean	Output raw HTML
`--text`	boolean	Output plain text
`-r, --render`	boolean	Use headless browser
`--stealth`	boolean	Use stealth mode
`-w, --wait <ms>`	number	Wait time after page load (ms)
`--selector <css>`	string	CSS selector to extract
`--exclude <selectors...>`	string[]	CSS selectors to exclude
`--include-tags <tags>`	string	Comma-separated tags to include
`--exclude-tags <tags>`	string	Comma-separated tags to exclude
`--only-main-content`	boolean	Extract only main/article content
`--screenshot [path]`	string	Capture screenshot (optional path)
`--full-page`	boolean	Full-page screenshot
`--links`	boolean	Output only links
`--images`	boolean	Output image URLs
`--meta`	boolean	Output only metadata
`--max-tokens <n>`	number	Max token count (truncate)
`--cache <ttl>`	string	Cache TTL (e.g., "5m", "1h", "1d")
`--location <country>`	string	ISO country code
`--language <lang>`	string	Language preference

`webpeel search <query>`

Search the web using DuckDuckGo (default, free) or Brave Search (BYOK).

# Basic search (DuckDuckGo)
npx webpeel search "AI news"

# Use Brave Search (BYOK)
npx webpeel search "AI news" --provider brave --search-api-key $BRAVE_KEY

# Or set once in config
npx webpeel config set braveApiKey $BRAVE_KEY
npx webpeel search "AI news" --provider brave

# Limit results
npx webpeel search "python tutorials" -n 10

# JSON output
npx webpeel search "web scraping" --json

Options

Option	Description
`-n, --count <n>`	Number of results (1-10, default: 5)
`--provider <provider>`	Search provider: `duckduckgo` (default) or `brave`
`--search-api-key <key>`	Brave Search API key (or env `WEBPEEL_BRAVE_API_KEY`)
`--json`	Output as JSON
`-s, --silent`	Silent mode

`webpeel answer <question>`

Ask a question, search the web, fetch sources, and get an AI-generated answer with citations. BYOK — bring your own LLM API key.

# Answer with citations (DuckDuckGo search by default)
npx webpeel answer "What is MCP?" --llm openai --llm-api-key $OPENAI_API_KEY

# Use Brave Search for higher-quality results (BYOK)
npx webpeel answer "What is MCP?" \
  --provider brave \
  --search-api-key $BRAVE_KEY \
  --llm anthropic \
  --llm-api-key $ANTHROPIC_API_KEY \
  --max-sources 5

# JSON output
npx webpeel answer "Compare DuckDuckGo vs Brave Search" \
  --llm openai --llm-api-key $OPENAI_API_KEY \
  --json

Options

Option	Description
`--provider <provider>`	Search provider: `duckduckgo` (default) or `brave`
`--search-api-key <key>`	Brave Search API key (or env `WEBPEEL_BRAVE_API_KEY`)
`--llm <provider>`	LLM provider: `openai`, `anthropic`, or `google`
`--llm-api-key <key>`	LLM API key (or env `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GOOGLE_API_KEY`)
`--llm-model <model>`	Optional LLM model name (provider-specific)
`--max-sources <n>`	Maximum sources to fetch (1-10, default: 5)
`--json`	Output as JSON
`-s, --silent`	Silent mode

`webpeel batch [file]`

Fetch multiple URLs from file or stdin.

# From file
npx webpeel batch urls.txt

# From stdin
cat urls.txt | npx webpeel batch

# With concurrency
npx webpeel batch urls.txt -c 5

# Save to directory
npx webpeel batch urls.txt -o output/

# JSON output
npx webpeel batch urls.txt --json

Options

Option	Description
`-c, --concurrency <n>`	Max concurrent fetches (default: 3)
`-o, --output <dir>`	Output directory (one file per URL)
`--json`	Output as JSON array
`-r, --render`	Use browser rendering for all
`--selector <css>`	CSS selector to extract

`webpeel crawl <url>`

Crawl a website recursively.

# Basic crawl
npx webpeel crawl https://example.com

# Limit pages and depth
npx webpeel crawl https://example.com --max-pages 100 --max-depth 3

# Exclude patterns
npx webpeel crawl https://example.com --exclude "/admin/" "/login"

# Domain restrictions
npx webpeel crawl https://example.com --allowed-domains "example.com" "docs.example.com"

# Ignore robots.txt
npx webpeel crawl https://example.com --ignore-robots

# With browser rendering
npx webpeel crawl https://example.com --render --stealth

Options

Option	Description
`--max-pages <n>`	Max pages to crawl (default: 10, max: 100)
`--max-depth <n>`	Max depth (default: 2, max: 5)
`--allowed-domains <domains...>`	Only crawl these domains
`--exclude <patterns...>`	Exclude URL patterns (regex)
`--ignore-robots`	Ignore robots.txt
`--rate-limit <ms>`	Rate limit (default: 1000ms)
`-r, --render`	Use browser rendering
`--stealth`	Use stealth mode

`webpeel map <url>`

Discover all URLs on a domain.

# Map a domain
npx webpeel map https://example.com

# Skip sitemap
npx webpeel map https://example.com --no-sitemap

# Skip crawl
npx webpeel map https://example.com --no-crawl

# Limit results
npx webpeel map https://example.com --max 1000

# Filter patterns
npx webpeel map https://example.com --include "/docs/" --exclude "/api/"

Options

Option	Description
`--no-sitemap`	Skip sitemap.xml discovery
`--no-crawl`	Skip homepage crawl
`--max <n>`	Max URLs (default: 5000)
`--include <patterns...>`	Include URL patterns (regex)
`--exclude <patterns...>`	Exclude URL patterns (regex)

`webpeel login`

Authenticate CLI with API key.

npx webpeel login
# Prompts for API key, saves to ~/.webpeel/config.json

`webpeel logout`

Clear saved credentials.

npx webpeel logout

`webpeel whoami`

Show authentication status.

npx webpeel whoami
# Logged in with API key: sk-1234...abcd
# Plan: Pro
# Config: ~/.webpeel/config.json

`webpeel usage`

Show usage and quota.

npx webpeel usage
# Free: 45/125 fetches this week (36%)
# Resets: Mon Feb 17 2026 00:00:00 EST

`webpeel config [action] [key] [value]`

View or update CLI configuration (stored in ~/.webpeel/config.json).

# Show overview
npx webpeel config

# Get specific key
npx webpeel config get braveApiKey

# Set Brave Search API key (BYOK)
npx webpeel config set braveApiKey $BRAVE_KEY

`webpeel cache <action>`

Manage local response cache.

# Show cache stats
npx webpeel cache stats

# Clear expired entries
npx webpeel cache clear

# Purge all entries
npx webpeel cache purge

`webpeel serve`

Start API server.

# Default port 3000
npx webpeel serve

# Custom port
npx webpeel serve -p 8080

`webpeel mcp`

Start MCP server for Claude/Cursor.

npx webpeel mcp
# Runs on stdio for Claude Desktop / Cursor integration

`webpeel agent <prompt>`

Run autonomous web research agent.

# Basic agent
npx webpeel agent "Find the top 5 AI coding tools" --llm-key $OPENAI_API_KEY

# With schema
npx webpeel agent "Compare AI tools" \
  --llm-key $OPENAI_API_KEY \
  --schema '{"type":"array","items":{"properties":{"name":{"type":"string"}}}}'

# With starting URLs
npx webpeel agent "Extract pricing" \
  --urls https://example.com,https://example.org \
  --llm-key $OPENAI_API_KEY

`webpeel track <url>`

Track content changes.

npx webpeel track https://example.com
# Returns fingerprint for change detection

`webpeel summarize <url>`

Generate AI summary.

npx webpeel summarize https://example.com --llm-key $OPENAI_API_KEY

# Custom prompt
npx webpeel summarize https://example.com \
  --llm-key $OPENAI_API_KEY \
  --prompt "Summarize in 3 bullet points"

`webpeel brand <url>`

Extract branding and design system.

npx webpeel brand https://example.com --json

`webpeel jobs`

List active async jobs.

npx webpeel jobs --json

`webpeel job <id>`

Get job status.

npx webpeel job crawl_abc123 --json

Environment Variables

Variable	Description
`WEBPEEL_API_KEY`	API key for authentication
`OPENAI_API_KEY`	OpenAI API key for LLM features
`ANTHROPIC_API_KEY`	Anthropic API key for LLM features
`GOOGLE_API_KEY`	Google API key for Gemini / LLM features
`WEBPEEL_BRAVE_API_KEY`	Brave Search API key for `webpeel search` / `webpeel answer`
`WEBPEEL_LLM_MODEL`	LLM model (default: gpt-4o-mini)
`WEBPEEL_LLM_BASE_URL`	LLM API base URL
`WEBPEEL_API_URL`	WebPeel API URL (self-hosted)

Examples

Extract Product Data

npx webpeel https://example.com/product \
  --extract '{"title": "h1", "price": ".price", "rating": ".stars"}' \
  --json

Monitor Price Changes

#!/bin/bash
URL="https://example.com/product"

# First run
FINGERPRINT=$(npx webpeel $URL --json | jq -r '.fingerprint')
echo $FINGERPRINT > fingerprint.txt

# Later runs
NEW_FP=$(npx webpeel $URL --json | jq -r '.fingerprint')
OLD_FP=$(cat fingerprint.txt)

if [ "$NEW_FP" != "$OLD_FP" ]; then
  echo "Price changed!"
fi

Batch Download Documentation

# Discover all docs URLs
npx webpeel map https://docs.example.com \
  --include "/docs/" \
  --json > urls.json

# Extract URLs
cat urls.json | jq -r '.urls[]' > urls.txt

# Batch download
npx webpeel batch urls.txt -o docs/ --selector "article"