Core Server Structure

Overview

The Web Intelligence MCP server follows the standard Nexus FastMCP pattern with Redis vault/operational connections.

File Location

/opt/mcp-servers/web/mcp_web_server.py

Size: ~11KB

Dependencies

FastMCP framework
Redis (vault: 6670, operational: 6671)
spider_rs (Rust web crawler with Python bindings)
credentials_helper (from shared/)

Server Initialization

from mcp.server.fastmcp import FastMCP
from spider_rs import Website
import redis

mcp = FastMCP("web")

Redis Connection Pattern

# Vault (writes) - password protected
vault_redis = redis.Redis(
    host='localhost',
    port=6670,
    password=vault_password,  # via credentials_helper
    decode_responses=True
)

# Operational (reads) - replica
operational_redis = redis.Redis(
    host='localhost',
    port=6671,
    password=None,
    decode_responses=True
)

Core Tools (Placeholders)

web.status

@mcp.tool()
async def status() -> str:
    """Get Web Intelligence server status and capabilities."""

Returns server health, spider_rs version, Redis connectivity.

web.crawl

@mcp.tool()
async def crawl(url: str, depth: int = 1, limit: int = 10) -> str:
    """Crawl a website starting from URL."""

Uses spider_rs Website.crawl() for recursive page discovery.

web.scrape

@mcp.tool()
async def scrape(url: str) -> str:
    """Scrape content from a single URL."""

Uses spider_rs Website.scrape() for single-page content extraction.

Architecture Notes

Server follows singleton pattern for Redis connections
spider_rs operations are async-compatible
Rate limiting handled at tool level
Timeout management via spider_rs configuration

/opt/mcp-servers/shared/credentials_helper.py - Password management
/opt/mcp-servers/shared/environment_config.json - Port definitions