Research Agent

Autonomous web research agent with a 9-state pipeline, work-hub task integration, knowledge base ingest of research outputs, and Langfuse tracing. Uses qwen3-embedding-4b for semantic deduplication and similarity search.

Quick Start

cd /mnt/apps/docker/ai/research-agent
cp .env.template .env
# Edit .env with your API keys
docker compose up -d
curl -sf http://localhost:8010/health

Service Identity

Field Value
Container research-agent
Domain research.haiven.site
Port 8010 (host) -> 8000 (container)
Category ai
Networks web, backend
GPU None
Status Live

Architecture

User Request
    |
    v
POST /research (query, task_id?)
    |
    v
9-State Pipeline
  PENDING -> SEARCHING -> CRAWLING -> CLEANING
  -> SYNTHESIZING -> VALIDATING -> [AWAITING_APPROVAL]
  -> COMPLETED / FAILED
    |
    +---> Work-hub PATCH (if task_id provided)
    |
    +---> haiven-knowledge POST /v1/ingest/text (auto-ingest on COMPLETED)
    |
    +---> Langfuse trace (full span tree)

Dependencies

Service Purpose URL
litellm LLM synthesis (all model calls) http://litellm:4000/v1
searxng Privacy-focused web search http://searxng:8080
crawl4ai Web content extraction http://crawl4ai:11235
langfuse-web Observability and tracing http://langfuse-web:3000
work-hub Task integration and artifact write-back http://work-hub:8030
haiven-knowledge KB ingest of research outputs http://haiven-knowledge:8022
redis Session cache and credential store redis://redis:6379

API Endpoints

Method Endpoint Description
POST /research Start new research session
GET /research/{session_id} Get session status and results
POST /research/{session_id}/approve Approve follow-up questions
GET /research List all research sessions
POST /research/similar Find semantically similar past research
GET /ws/research/{session_id} WebSocket live progress stream
GET /history Browse completed research history
DELETE /history/{session_id} Delete a research session
POST /history/cleanup Bulk cleanup by age/status
POST /credentials Store encrypted site credentials
GET /credentials List credential domains
DELETE /credentials/{domain} Remove site credentials
GET /health Comprehensive health check
GET /metrics Prometheus metrics

Pipeline States

PENDING -> SEARCHING -> CRAWLING -> CLEANING -> SYNTHESIZING
       -> VALIDATING -> AWAITING_APPROVAL -> COMPLETED / FAILED

Each state transition is emitted as a WebSocket event and recorded as a Langfuse span.

Configuration

Required Environment Variables

CRAWL4AI_API_TOKEN=your-token
LITELLM_API_KEY=your-key
LANGFUSE_PUBLIC_KEY=your-key
LANGFUSE_SECRET_KEY=your-key

Key Settings

Variable Default Description
DEFAULT_MODEL qwen3-30b-a3b-q8-abl Primary synthesis model
VALIDATION_MODEL falcon3-10b-instruct-f16 Quick validation checks
EMBEDDING_MODEL qwen3-embedding-4b Semantic similarity (vLLM always-on)
MAX_ITERATIONS 5 Research iteration ceiling
SIMILARITY_THRESHOLD 0.85 Deduplication threshold
WORKHUB_URL http://work-hub:8030 Work-hub base URL
KNOWLEDGE_URL http://haiven-knowledge:8022 Knowledge base URL

Work-Hub Integration

Pass task_id on research requests to wire artifact write-back:

curl -X POST http://localhost:8010/research \
  -H "Content-Type: application/json" \
  -d '{"query": "RAGAS evaluation frameworks", "task_id": "uuid-here", "auto_approve": true}'

On COMPLETED, the pipeline:
1. PATCHes the work-hub task with a research_output artifact containing session_id, query, sources, and summary
2. Injects a task_id tag into the Langfuse trace for cross-service correlation

Knowledge Base Auto-Ingest

Research outputs are automatically ingested into haiven-knowledge on completion:

POST http://haiven-knowledge:8022/v1/ingest/text
{
  "content": "<final_answer>",
  "topic": "<query[:100]>",
  "category": "research",
  "source": "research_agent",
  "source_application": "research_agent",
  "tags": ["auto-ingested", "research-output"]
}

Search for past research outputs via the knowledge base:

curl -X POST http://localhost:8022/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query": "RAGAS", "source_application": "research_agent", "limit": 5}'

Storage

All data persists in /mnt/storage/research-agent/:

Directory Purpose
data/ SQLite database (research.db)
cache/ Search result cache
crawl-cache/ Crawl4AI content cache
exports/ Session export files
archives/ Expired session archives
logs/ Application logs

Monitoring

Resource Limits

Resource Limit Reservation
CPU 4 cores 2 cores
Memory 4 GB 2 GB

Operations

# Restart
docker compose -f /mnt/apps/docker/ai/research-agent/docker-compose.yml restart research-agent

# Logs
docker logs -f research-agent

# Shell
docker exec -it research-agent bash

# Database backup (manual)
/mnt/apps/docker/ai/research-agent/scripts/backup.sh

OpenAPI Spec

/mnt/apps/docker/ai/research-agent/openapi.yaml

Interactive docs at https://research.haiven.site/docs.