Video Downloader (yt-dlp-web-ui)

Production-grade video downloading service using yt-dlp-web-ui with VPN geo-bypass, category page scraping, JavaScript runtime support, and comprehensive fallback options.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Gluetun VPN Container                               │
│                       (ExpressVPN → Canada)                                 │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                   Shared Network Namespace                            │   │
│  │  ┌─────────────────────────┐    ┌─────────────────────────┐          │   │
│  │  │   video-downloader      │    │   category-scraper      │          │   │
│  │  │   (yt-dlp-webui)        │◄───│   (Playwright)          │          │   │
│  │  │   Port 3033             │    │   Port 3034             │          │   │
│  │  └─────────────────────────┘    └─────────────────────────┘          │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│  Exposed Ports: 3033 (downloader), 3034 (scraper)                           │
└─────────────────────────────────────────────────────────────────────────────┘
                                      │
                  ┌───────────────────┴───────────────────┐
                  ▼                                       ▼
┌─────────────────────────────────┐    ┌─────────────────────────────────┐
│  Prometheus Exporter (9105)     │    │  Downloads → NAS Storage        │
│  video-downloader-exporter      │    │  /mnt/nas1/media/videos/ytdlp   │
└─────────────────────────────────┘    └─────────────────────────────────┘

Access

Documentation

• USER_GUIDE.md - Complete user guide with examples, troubleshooting, and best practices
• ROADMAP.md - Planned enhancements and feature roadmap
• Implementation Plan - Deployment details

Quick Start

cd /mnt/apps/docker/utils/video-downloader
cp .env.example .env
# Edit .env with your settings (optional)
docker compose up -d

Storage

Features

• 1000+ Sites: YouTube, Vimeo, Twitter/X, TikTok, Twitch, and more
• Queue Management: Concurrent downloads (configurable, default: 4)
• Format Selection: Best quality auto-selection or manual format choice
• Audio Extraction: MP3, FLAC, M4A, OPUS formats
• Subtitle Support: Auto-download and embed subtitles
• Cookie Authentication: Download age-restricted and private content
• Archive Tracking: Prevent re-downloading with --download-archive
• Automatic Retries: 10 retries with configurable backoff for reliability
• Metadata Embedding: Track info and thumbnails embedded in files
• Geo-Bypass: Access region-restricted content
• REST API: Full OpenAPI documentation for automation

Default Arguments

All downloads automatically include these optimizations (configured via YTDLP_ARGS environment variable):

--no-check-certificate       # Bypass SSL verification for compatibility
--retries 10                 # Retry failed downloads 10 times
--fragment-retries 10        # Retry failed fragments 10 times
--retry-sleep 5              # Wait 5 seconds between retries
--file-access-retries 10     # Retry file access 10 times
--extractor-retries 5        # Retry extractor detection 5 times
--embed-metadata             # Embed video metadata (title, artist, etc.)
--embed-thumbnail            # Embed thumbnail as cover art
--restrict-filenames         # Use ASCII-only safe filenames
--geo-bypass                 # Bypass geographic restrictions
--force-generic-extractor    # Try generic extraction as fallback
--js-runtimes node           # Enable JavaScript decryption (Node.js)
--yes-playlist               # Download full playlists/categories

These defaults ensure maximum compatibility and reliability across all sites.

VPN Geo-Bypass (Gluetun)

All traffic routes through ExpressVPN (Canada) to bypass geographic restrictions:

Bypassed restrictions:
- PornHub (blocked in Utah, Virginia, Texas, Montana, North Carolina, etc.)
- Region-locked content
- Rate-limited sites

VPN Health Check:

# Check VPN IP
docker exec downloader-vpn curl -s https://ipinfo.io/ip

# Check VPN status
docker logs downloader-vpn | grep -i "vpn is ready"

Category Scraper

For pages that yt-dlp doesn't recognize as playlists (category pages, model pages, search results), use the Category Scraper:

Access: https://scraper.haiven.local

How it works:
1. Playwright renders JavaScript-heavy pages in headless Chromium
2. Auto-scrolls to load infinite scroll content
3. Extracts all video URLs using regex patterns
4. Queues each video to the downloader via API

Supported page types:

API Usage:

# Preview URLs (don't download)
curl -X POST http://localhost:3034/preview \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.pornhub.com/categories/ebony"}'

# Scrape and queue all
curl -X POST http://localhost:3034/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.pornhub.com/model/perfectdime"}'

Fallback Chain

When downloading a video, yt-dlp follows this fallback chain:

1. Site-specific extractor (YouTube, Vimeo, PornHub, etc.)
2. Extractor retry (up to 5 attempts via --extractor-retries)
3. Generic extractor (via --force-generic-extractor)
   - HTML5 <video> tags
   - HLS/m3u8 streams
   - DASH manifests
   - Direct .mp4/.webm links
4. JavaScript decryption (via --js-runtimes node)

API Usage

Download a Video

curl -X POST "https://downloader.haiven.local/api/v1/exec" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=VIDEO_ID",
    "path": "/downloads",
    "rename": "",
    "args": ""
  }'

Extract Audio Only

curl -X POST "https://downloader.haiven.local/api/v1/exec" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/watch?v=VIDEO_ID",
    "path": "/audio",
    "rename": "",
    "args": "-x --audio-format mp3 --audio-quality 0"
  }'

Check Running Downloads

curl https://downloader.haiven.local/api/v1/running | jq

Cookie Management

For authenticated sites (age-restricted, private content):

1. Export cookies from browser using "Get cookies.txt LOCALLY" extension
2. Save to ./cookies/youtube.txt, ./cookies/vimeo.txt, etc.
3. Use in downloads: --cookies /cookies/youtube.txt

See USER_GUIDE.md - Cookie Authentication for detailed instructions.

Preset Scripts

Status: Planned (see ROADMAP.md Priority 2). Scripts directory not yet implemented.

Once created, the /scripts/ directory will contain convenient wrappers for common download scenarios:

Current workaround: Use the API directly with custom arguments. See USER_GUIDE.md - Using Preset Scripts for API equivalents.

Scheduled Downloads / Subscriptions

Status: Planned (see ROADMAP.md Priority 5). Subscription system not yet implemented.

The planned subscription system will enable:

• Automated channel monitoring: Add channels to config/subscriptions.txt
• Daily cron job: Checks for new videos at 3 AM
• Smart updates: Uses --download-archive to skip already-downloaded videos
• Configurable limits: Download latest N videos from each channel

Current workaround: Manually trigger channel downloads with archive tracking:

curl -X POST "https://downloader.haiven.local/api/v1/exec" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.youtube.com/@channelname",
    "path": "/downloads",
    "args": "--download-archive /downloads/archive.txt --playlist-end 5"
  }'

Re-run this command periodically to fetch new videos. The archive file prevents re-downloads.

Monitoring

Prometheus Metrics

The service exposes metrics at https://downloader.haiven.local/metrics (port 3033, path /metrics).

Metrics available:
- HTTP request rates and latencies
- Download queue depth
- Active downloads count
- Success/failure rates

Prometheus scraping configuration:

# Configured via docker-compose.yml labels
labels:
  - "prometheus.scrape=true"
  - "prometheus.port=3033"
  - "prometheus.path=/metrics"

Grafana Dashboard

Status: Planned (see ROADMAP.md Priority 6).

A dedicated Grafana dashboard will display:
- Download count over time
- Average download duration
- Storage usage trends
- Error rate tracking
- Queue depth visualization

Current monitoring:
- Manual API checks: curl https://downloader.haiven.local/api/v1/active
- Container logs: docker logs -f video-downloader
- Uptime Kuma: Service availability at https://status.haiven.local

Health Check

# Docker health check (automated)
docker inspect video-downloader | jq '.[0].State.Health'

# Manual health verification
curl -I https://downloader.haiven.local

Maintenance

View Logs

docker logs -f video-downloader

Restart Service

docker compose restart

Update Service

docker compose pull
docker compose up -d

Update yt-dlp (if needed between image updates)

docker exec video-downloader pip install -U yt-dlp

Supported Sites

yt-dlp supports 1000+ sites including:
- YouTube, YouTube Music
- Vimeo, Dailymotion
- Twitter/X, TikTok, Instagram
- Twitch, Rumble
- Reddit, Facebook
- And many more: https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md

Quick Reference

Common Arguments

API Endpoints

Storage Paths

Resources

• USER_GUIDE.md - Complete user guide with examples and troubleshooting
• ROADMAP.md - Planned features and enhancements
• yt-dlp-web-ui GitHub
• yt-dlp Documentation
• yt-dlp Options Reference
• Implementation Plan

Generated by haiven-service-onboarding plugin | Last updated: 2025-12-06