Research Agent User Guide

A comprehensive guide for using the AI-powered iterative web research agent.

Table of Contents

  1. Quick Start
  2. Web UI
  3. API Usage
  4. CLI Usage
  5. Research Workflow
  6. Configuration Options
  7. Credential Management
  8. History Management
  9. Troubleshooting
  10. Best Practices

Quick Start

Access Points

Interface URL Purpose
Web UI https://research.haiven.local Interactive research interface
API https://research.haiven.local/api RESTful API endpoints
WebSocket wss://research.haiven.local/ws/research/{session_id} Real-time progress
Docs https://research.haiven.local/docs Swagger UI documentation

Your First Research

Via Web UI:
1. Navigate to https://research.haiven.local
2. Enter your research query (e.g., "RAG best practices 2024")
3. Optionally adjust max iterations and model
4. Click "Start Research"
5. Watch real-time progress and results

Via API:

curl -X POST https://research.haiven.local/api/research \
  -H "Content-Type: application/json" \
  -d '{"query": "RAG best practices 2024", "max_iterations": 5}'

Via CLI:

# Interactive mode
research-cli query "RAG best practices 2024"

# With options
research-cli query "RAG best practices 2024" \
  --max-iterations 3 \
  --model qwen3-30b-a3b-q8-abl \
  --auto-approve

Web UI

Home Page

The home page provides:
- Research input field - Enter your query here
- Configuration options - Max iterations, model selection, auto-approve toggle
- Recent research - Quick access to past research sessions

Progress Page

During research, the progress page shows:
- Current stage - Searching, crawling, synthesizing, etc.
- Progress bar - Overall completion percentage
- Live updates - Real-time status via WebSocket
- Iteration tracker - Current iteration vs maximum

Approval Page

When auto_approve=false, you'll see:
- Open questions - Questions identified for deeper research
- Edit capability - Modify questions before continuing
- Select/deselect - Choose which questions to pursue
- Continue/Complete - Resume research or finish early

Results Page

After completion:
- Final synthesis - Comprehensive answer to your query
- Source list - All sources used with quality scores
- Per-iteration breakdown - What was learned in each round
- Export options - Download as Markdown or JSON

API Usage

Start Research

# Basic request
curl -X POST https://research.haiven.local/api/research \
  -H "Content-Type: application/json" \
  -d '{
    "query": "How to implement vector search with PostgreSQL",
    "max_iterations": 5
  }'

# Response
{
  "session_id": "abc123-def456",
  "status": "pending",
  "ws_url": "/ws/research/abc123-def456",
  "message": "Research session started"
}

Advanced Options

curl -X POST https://research.haiven.local/api/research \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Kubernetes autoscaling best practices",
    "max_iterations": 3,
    "model": "qwen3-30b-a3b-q8-abl",
    "auto_approve": true,
    "domains": {
      "exclude": ["pinterest.com", "reddit.com"]
    },
    "force": true
  }'

Check Session Status

curl https://research.haiven.local/api/research/{session_id}

Approve Questions

curl -X POST https://research.haiven.local/api/research/{session_id}/approve \
  -H "Content-Type: application/json" \
  -d '{
    "approved_questions": [
      "What chunk sizes work best for RAG?",
      "How to handle multi-modal content?"
    ]
  }'

Find Similar Research

curl -X POST https://research.haiven.local/api/research/similar \
  -H "Content-Type: application/json" \
  -d '{
    "query": "RAG optimization techniques",
    "limit": 5
  }'

WebSocket Progress

Connect to WebSocket for real-time updates:

const ws = new WebSocket('wss://research.haiven.local/ws/research/' + sessionId);

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Event:', data.event);
  console.log('Payload:', data);

  // Event types:
  // - status_change: Research stage changed
  // - search_complete: Web search finished
  // - crawl_progress: Crawling X of Y URLs
  // - synthesis_update: Partial synthesis available
  // - questions_ready: Open questions for approval
  // - completed: Research finished
  // - error: Something went wrong
};

CLI Usage

Installation

The CLI is available inside the container:

docker exec -it research-agent research-cli --help

Or create an alias:

alias research='docker exec -it research-agent research-cli'

Commands

Query (Start Research)

# Basic query
research query "What is retrieval-augmented generation?"

# With options
research query "RAG best practices" \
  --max-iterations 3 \
  --model qwen3-30b-a3b-q8-abl \
  --auto-approve \
  --output markdown

# Exclude domains
research query "Python best practices" \
  --exclude pinterest.com \
  --exclude reddit.com

List Sessions

# List all
research list

# Filter by status
research list --status completed

# Limit results
research list --limit 10 --sort -created_at

Get Session

research get <session_id>

Find Similar

research similar "How does RAG work?"

History Management

# View history
research history

# Delete session
research delete <session_id>

# Cleanup old sessions
research cleanup --days 30 --status failed

Research Workflow

Stages

1. PENDING      -> Session created, research starting
2. SEARCHING    -> Querying SearXNG for relevant URLs
3. CRAWLING     -> Extracting content from URLs via Crawl4AI
4. CLEANING     -> Processing and deduplicating content
5. SYNTHESIZING -> LLM generating synthesis from sources
6. VALIDATING   -> Checking if more research needed
7. AWAITING_APPROVAL -> (if auto_approve=false) Waiting for user
8. COMPLETED    -> Research finished successfully
   FAILED       -> Error occurred

Iteration Flow

┌─────────────────────────────────────────────────────────────┐
│  Iteration 1                                                 │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────────┐  │
│  │ Search  │ → │ Crawl   │ → │ Clean   │ → │ Synthesize  │  │
│  └─────────┘   └─────────┘   └─────────┘   └─────────────┘  │
│                                                   │          │
│                                      ┌────────────▼────┐     │
│                                      │ Needs more?     │     │
│                                      └────────────┬────┘     │
│                                           │            │     │
│                                        YES│            │NO   │
│                                           ▼            │     │
│                               ┌───────────────────┐    │     │
│                               │ Generate questions│    │     │
│                               └─────────┬─────────┘    │     │
│                                         │              │     │
│                                         ▼              │     │
│                               ┌───────────────────┐    │     │
│                               │ Approval required?│    │     │
│                               └─────────┬─────────┘    │     │
│                                    │           │       │     │
│                                 YES│           │NO     │     │
│                                    ▼           ▼       │     │
│                               ┌────────┐   ┌───────┐   │     │
│                               │ Wait   │   │ Next  │   │     │
│                               │ approval│  │ iter  │   │     │
│                               └────────┘   └───┬───┘   │     │
└────────────────────────────────────────────────│───────┘     │
                                                 │             │
                                                 ▼             │
                                          ┌───────────┐        │
                                          │ Iteration │◄───────┘
                                          │    N+1    │
                                          └───────────┘

Configuration Options

Request Parameters

Parameter Type Default Description
query string required Research question (3-1000 chars)
max_iterations int 5 Max research rounds (1-10)
model string null LLM model via LiteLLM (null = default)
auto_approve bool false Auto-approve follow-up questions
domains.exclude array [] Domains to exclude from search
force bool false Skip deduplication check

Environment Variables

Variable Default Description
LITELLM_BASE_URL http://litellm:4000 LiteLLM proxy URL
SEARXNG_BASE_URL http://searxng:8080 SearXNG search URL
CRAWL4AI_BASE_URL http://crawl4ai:11235 Crawl4AI crawler URL
LANGFUSE_HOST http://langfuse:3000 Langfuse tracing URL
RESEARCH_AGENT_SECRET - Encryption key for credentials
DEFAULT_MODEL claude-sonnet-4-20250514 Default LLM model
DEFAULT_MAX_ITERATIONS 5 Default iteration count

Credential Management

Store credentials for sites requiring authentication.

Store Credentials

# Via API
curl -X POST https://research.haiven.local/api/credentials \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "username": "user@example.com",
    "password": "secret123",
    "credential_type": "basic"
  }'

# Via CLI
research credentials add example.com --username user@example.com
# (password prompted securely)

List Credentials

# Via API
curl https://research.haiven.local/api/credentials

# Via CLI
research credentials list

Delete Credentials

# Via API
curl -X DELETE https://research.haiven.local/api/credentials/example.com

# Via CLI
research credentials delete example.com

Security Properties

History Management

Browse History

# List completed research
curl "https://research.haiven.local/api/history?status=completed&limit=20"

# Search by query text
curl "https://research.haiven.local/api/history?query_text=RAG"

Get Details

curl https://research.haiven.local/api/history/{session_id}

Delete Sessions

# Single session
curl -X DELETE https://research.haiven.local/api/history/{session_id}

# Multiple sessions
curl -X DELETE "https://research.haiven.local/api/history?session_ids=id1&session_ids=id2"

Cleanup Old Sessions

# Dry run (see what would be deleted)
curl -X POST "https://research.haiven.local/api/history/cleanup?days=30&dry_run=true"

# Actually delete
curl -X POST "https://research.haiven.local/api/history/cleanup?days=30&dry_run=false"

# Only delete failed sessions
curl -X POST "https://research.haiven.local/api/history/cleanup?days=7&status=failed&dry_run=false"

Troubleshooting

Common Issues

"Similar research found"

Problem: API returns status: "similar_found" instead of starting new research.

Solution: Use force: true to bypass deduplication:

{"query": "...", "force": true}

Session stuck in "searching"

Problem: Research stuck at searching stage.

Check:
1. SearXNG health: curl http://searxng:8080/healthz
2. Network connectivity: docker exec research-agent ping searxng
3. Logs: docker logs research-agent --tail 100

Crawling failures

Problem: Many URLs fail to crawl.

Check:
1. Crawl4AI health: curl http://crawl4ai:11235/health
2. Target sites may be blocking crawlers
3. Consider using credentials for authenticated sites

LLM timeout

Problem: Synthesis takes too long or times out.

Solutions:
1. Use a faster model
2. Reduce content by limiting crawled sources
3. Increase timeout in config

Health Checks

# Comprehensive health
curl https://research.haiven.local/health

# Quick readiness
curl https://research.haiven.local/health/ready

# Liveness
curl https://research.haiven.local/health/live

Logs

# Container logs
docker logs research-agent -f

# Application logs in container
docker exec research-agent cat /data/logs/research.log

Langfuse Tracing

View all research traces in Langfuse:
1. Go to https://ai-ops.haiven.local
2. Navigate to Traces
3. Filter by research-agent service

Best Practices

Query Formulation

Good queries:
- "What are the best practices for RAG chunking strategies in 2024?"
- "How to implement vector search with PostgreSQL pgvector extension"
- "Compare Kubernetes HPA vs KEDA for autoscaling workloads"

Avoid:
- Too vague: "Tell me about AI" (too broad)
- Too specific: "What's the AWS S3 bucket name for X" (not research)
- Opinion-based: "What's the best programming language?" (subjective)

Iteration Settings

Use Case Max Iterations Auto Approve
Quick overview 1-2 true
Thorough research 5 false
Deep dive 10 false
Automated pipeline 3-5 true

Domain Exclusions

Commonly excluded domains for cleaner results:

{
  "domains": {
    "exclude": [
      "pinterest.com",
      "reddit.com",
      "quora.com",
      "twitter.com",
      "facebook.com"
    ]
  }
}

Caching & Deduplication

Resource Management

Support