Self-Hosting WebPeel

Run your own instance of the WebPeel API server using Docker. Full control, no rate limits, complete privacy.

Prerequisites

Before you begin, make sure you have:

• Docker (v20.10+) — Install Docker
• Docker Compose (v2.0+) — Usually included with Docker Desktop
• Git — To clone the repository

Verify your setup:

docker --version
docker compose version

Quick Start

Get WebPeel running in under 5 minutes.

1. Clone the Repository

git clone https://github.com/webpeel/webpeel.git
cd webpeel

2. Configure Environment Variables

Create a .env file in the project root:

cp .env.example .env

Edit .env with your settings:

# Required: Change this to a secure random string!
JWT_SECRET=your-secure-random-secret-here

# Optional: Add custom CORS origins
CORS_ORIGINS=https://your-domain.com,https://app.your-domain.com

# Optional: Stripe integration (for paid tiers)
STRIPE_SECRET_KEY=sk_live_...
STRIPE_WEBHOOK_SECRET=whsec_...

⚠️ Security: Generate a strong JWT_SECRET using:

openssl rand -base64 32

3. Start the Server

docker compose up -d

That's it! Your WebPeel instance is now running at http://localhost:3000.

4. Verify It's Working

Check the health endpoint:

curl http://localhost:3000/health

Expected response:

{
  "status": "healthy",
  "timestamp": "2024-...",
  "uptime": 123
}

Environment Variables Reference

Default CORS Origins:

• https://app.webpeel.dev
• https://webpeel.dev
• http://localhost:3000
• http://localhost:3001

*Note: DATABASE_URL is automatically set by docker-compose.yaml. Only change it if you're using an external database.

Docker Compose Configuration

The docker-compose.yaml file defines two services:

services:
  webpeel:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - PORT=3000
      - DATABASE_URL=postgresql://webpeel:webpeel@db:5432/webpeel
      - JWT_SECRET=change-me-in-production
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=webpeel
      - POSTGRES_PASSWORD=webpeel
      - POSTGRES_DB=webpeel
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U webpeel"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: unless-stopped

volumes:
  pgdata:

Usage

Register a User

curl -X POST http://localhost:3000/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "you@example.com",
    "password": "your-secure-password"
  }'

Response:

{
  "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "user": {
    "id": "user_...",
    "email": "you@example.com",
    "tier": "free"
  }
}

Save the token — you'll need it for API requests.

Fetch a Page

curl "http://localhost:3000/v1/fetch?url=https://example.com" \
  -H "Authorization: Bearer YOUR_TOKEN_HERE"

Search the Web

curl "http://localhost:3000/v1/search?q=docker+containers" \
  -H "Authorization: Bearer YOUR_TOKEN_HERE"

PostgreSQL Setup

Use External PostgreSQL

Edit docker-compose.yaml and remove the db service:

services:
  webpeel:
    environment:
      - DATABASE_URL=postgresql://user:pass@your-db-host:5432/webpeel
    # Remove the depends_on section

Backup PostgreSQL

# Backup
docker compose exec db pg_dump -U webpeel webpeel > backup.sql

# Restore
docker compose exec -T db psql -U webpeel webpeel < backup.sql

Access PostgreSQL Shell

docker compose exec db psql -U webpeel

Reverse Proxy (Nginx)

For production deployments, put WebPeel behind a reverse proxy for HTTPS and load balancing.

Nginx Configuration

server {
    listen 443 ssl http2;
    server_name api.your-domain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://localhost:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        # Increase timeouts for large scraping jobs
        proxy_read_timeout 300s;
        proxy_connect_timeout 75s;
    }
}

SSL/TLS with Let's Encrypt

Use Certbot to get a free SSL certificate:

# Install Certbot
sudo apt-get install certbot python3-certbot-nginx

# Get certificate
sudo certbot --nginx -d api.your-domain.com

# Auto-renewal (runs twice daily)
sudo systemctl enable certbot.timer

Custom Configuration

Change Port

Edit docker-compose.yaml:

services:
  webpeel:
    ports:
      - "8080:3000"  # Host:Container

Increase Shared Memory

If Playwright browser crashes, increase shared memory:

services:
  webpeel:
    shm_size: '2gb'  # Add this line

Custom Database Password

Edit docker-compose.yaml:

services:
  db:
    environment:
      - POSTGRES_PASSWORD=your-secure-password
  webpeel:
    environment:
      - DATABASE_URL=postgresql://webpeel:your-secure-password@db:5432/webpeel

Scaling Considerations

Horizontal Scaling

Run multiple WebPeel instances behind a load balancer:

# Scale to 3 instances
docker compose up -d --scale webpeel=3

Use a load balancer like Nginx or HAProxy to distribute traffic.

Resource Requirements

Performance Tuning

• Browser pooling: WebPeel reuses browser instances to reduce overhead
• Connection pooling: PostgreSQL connections are pooled automatically
• Rate limiting: Configured per user tier (modify in src/server/middleware/rateLimit.ts)

Updating

Pull the latest changes and rebuild:

cd webpeel
git pull origin main
docker compose down
docker compose up -d --build

Note: Your database data persists in the pgdata Docker volume.

Zero-Downtime Updates

1. Start new containers with --scale
2. Wait for health checks to pass
3. Stop old containers

# Start 3 new instances
docker compose up -d --scale webpeel=3 --no-recreate

# Wait 60 seconds for health checks
sleep 60

# Stop old containers
docker compose stop webpeel

Production Security Checklist

• Change JWT_SECRET to a strong random value
• Change default database password in docker-compose.yaml
• Use HTTPS (put WebPeel behind a reverse proxy)
• Set NODE_ENV=production
• Restrict CORS origins to your domains only
• Enable firewall rules (only expose necessary ports)
• Set up automated PostgreSQL backups
• Monitor logs for suspicious activity
• Keep Docker and dependencies updated
• Use strong passwords for user accounts

Troubleshooting

Server won't start

Check logs:

docker compose logs webpeel

Common issues:

• JWT_SECRET not set → Edit .env and add a secret
• Database connection failed → Wait 30s for Postgres to initialize
• Port 3000 already in use → Change the port mapping in docker-compose.yaml

Health check fails

The server needs ~30-40 seconds to start (Playwright browser installation). Check:

docker compose ps

If webpeel shows as unhealthy, wait a bit longer or check logs.

Playwright browser crashes

Increase shared memory in docker-compose.yaml:

services:
  webpeel:
    shm_size: '2gb'

Reset everything

Stop containers and delete all data:

docker compose down -v

⚠️ Warning: This deletes your database!

Monitoring

View Logs

# All logs
docker compose logs -f

# WebPeel only
docker compose logs -f webpeel

# Last 100 lines
docker compose logs --tail=100 webpeel

Check Resource Usage

docker stats

Health Check

curl http://localhost:3000/health

Next Steps