Hard Boot Recovery Procedure

Phase 1: Verify Data Integrity

Before touching anything, verify data exists:

# Check all containers running
docker ps | grep nexus3-

# Verify data INSIDE containers
docker exec nexus3-kb-vault redis-cli -a "PASSWORD" DBSIZE

Key Discovery: Data persists on NVMe in /data/nexus3/{env}/vault/ - it's SAFE.

Phase 2: Update Environment (Per Environment)

Step 1: Backup Vault RDB

CRITICAL: Always backup before any changes!

ENV_NAME="kb"
STRONG_PASSWORD="tg6ap5"  # From Locker
VAULT_PORT="6625"
OP_PORT="6626"

# Force RDB save
docker exec nexus3-${ENV_NAME}-vault redis-cli -a "OLD_PASSWORD" SAVE

# Copy backup
sudo mkdir -p /tmp/redis_backups
sudo cp /data/nexus3/${ENV_NAME}/vault/dump.rdb \
    /tmp/redis_backups/${ENV_NAME}_backup_$(date +%Y%m%d_%H%M%S).rdb

# Verify backup size
ls -lh /tmp/redis_backups/${ENV_NAME}_backup_*

Why RDB? Redis ONLY accepts RDB format for restores. AOF won't work!

Step 2: Update Vault Container

# Stop and remove
docker stop nexus3-${ENV_NAME}-vault
docker rm nexus3-${ENV_NAME}-vault

# Recreate with strong password on nexus-network
docker run -d \
    --name nexus3-${ENV_NAME}-vault \
    --restart unless-stopped \
    --network nexus-network \
    -p ${VAULT_PORT}:6379 \
    -p 7${VAULT_PORT}:3000 \
    -v /data/nexus3/${ENV_NAME}/vault:/var/lib/falkordb/data \
    -e "REDIS_ARGS=--requirepass ${STRONG_PASSWORD} --appendonly yes --save 900 1 --save 300 10 --save 60 10000" \
    falkordb/falkordb:latest

# Wait and verify
sleep 5
redis-cli -p ${VAULT_PORT} -a "${STRONG_PASSWORD}" DBSIZE

Key count should match before update!

Step 3: Update Operational (With Replication)

# Stop and remove
docker stop nexus3-${ENV_NAME}-operational
docker rm nexus3-${ENV_NAME}-operational

# Recreate with replication configured
docker run -d \
    --name nexus3-${ENV_NAME}-operational \
    --restart unless-stopped \
    --network nexus-network \
    -p ${OP_PORT}:6379 \
    -p 7${OP_PORT}:3000 \
    -v /data/nexus3/${ENV_NAME}/operational:/var/lib/falkordb/data \
    -e "REDIS_ARGS=--requirepass ${STRONG_PASSWORD} --appendonly yes --replicaof nexus3-${ENV_NAME}-vault 6379 --masterauth ${STRONG_PASSWORD}" \
    falkordb/falkordb:latest

sleep 5

CRITICAL: The --replicaof and --masterauth parameters configure master-slave replication!

Step 4: Verify Replication

# Check operational is slave
redis-cli -p ${OP_PORT} -a "${STRONG_PASSWORD}" INFO replication | grep -E "role|master_link_status"

# Should show:
# role:slave
# master_link_status:up

# Verify key counts match
redis-cli -p ${VAULT_PORT} -a "${STRONG_PASSWORD}" DBSIZE
redis-cli -p ${OP_PORT} -a "${STRONG_PASSWORD}" DBSIZE

Step 5: Update MCP Server

# Edit MCP server file
# Example: /opt/mcp-servers/kb/mcp_kb_server.py

# Change fallback password:
# OLD: vault_password = 'KB3Vault2025'
# NEW: vault_password = 'tg6ap5'

Secure Environments (Locker, User)

Vault-only, no operational, no replication. Just update vault container.