If this saved your night 🌙, a ⭐ helps others find it.

This system wraps any long-running service with autonomous crash recovery. OpenClaw Gateway is the primary example — a self-hosted AI gateway for Claude/GPT-4 routing — but the watchdog/recovery architecture adapts to any service.

Your service crashes at midnight. A basic watchdog restarts it — but what if the config is corrupted? The API rate limit hit? A dependency broken?

Simple restart = crash loop. You get paged. Your weekend is ruined.

This system doesn't just restart — it understands and fixes root causes.

The gap that matters: when a crash loop is caused by something that can't be fixed by restarting alone, every other tool pages you. This one tries to fix it first.

Not ready to commit? Preview exactly what the installer will do — no changes made:

curl -fsSL https://raw.githubusercontent.com/Ramsbaby/openclaw-self-healing/main/install.sh | bash -s -- --dry-run

Sample output:

╔═══════════════════════════════════════════════════════════╗
║  🔍 DRY RUN — nothing will be installed or modified       ║
╚═══════════════════════════════════════════════════════════╝

[Pre-flight] All prerequisites present ✅

[Step 1] Directories that would be created: ~/.openclaw/...
[Step 2] Scripts that would be downloaded: gateway-watchdog.sh, ...
[Step 3] Environment file: ~/.openclaw/.env
[Step 4] LaunchAgents: ai.openclaw.watchdog, com.openclaw.healthcheck

  ✓ Level 0: Pre-flight validation              READY
  ✓ Level 1: KeepAlive (instant restart)         READY
  ✓ Level 2: Watchdog + HealthCheck (3-5 min)    READY
  ✓ Level 3: AI Emergency Recovery (auto-trigger) READY
  ✓ Level 4: Discord/Telegram Human Alert         READY

  Run without --dry-run to install.

• macOS 12+ or Linux (Ubuntu 20.04+ / systemd) or Docker
• OpenClaw Gateway installed and running
• Any major LLM — Claude CLI (default), OpenAI, Gemini, or Ollama. See LLM-Agnostic
• tmux, jq (brew install tmux jq or apt install tmux jq)

Note: While this was built for OpenClaw Gateway, the watchdog/recovery architecture works for any service. See docs/configuration.md to adapt it.

curl -fsSL https://raw.githubusercontent.com/ramsbaby/openclaw-self-healing/main/install.sh | bash

The installer walks you through everything:

╔═══════════════════════════════════════════════╗
║  🦞 OpenClaw Self-Healing System Installer    ║
╚═══════════════════════════════════════════════╝

[1/6] Checking prerequisites...          ✅
[2/6] Creating directories...            ✅
[3/6] Installing scripts...              ✅
[4/6] Configuring environment..
      Discord webhook URL (optional): https://discord.com/api/webhooks/...
      Gateway port [18789]: 
      Gateway token (auto-detected): ✅
[5/6] Installing Watchdog LaunchAgent... ✅
[6/6] Verifying installation...
      Health check: HTTP 200 ✅
      Chain: Watchdog → HealthCheck → Emergency Recovery ✅

🎉 Self-Healing System Active!

git clone https://github.com/Ramsbaby/openclaw-self-healing.git
cd openclaw-self-healing
cp .env.example .env   # edit with your config
docker compose up -d

See docs/DOCKER.md for full configuration guide.

# Kill your Gateway to test auto-recovery
kill -9 $(pgrep -f openclaw-gateway)

# Wait ~30 seconds, then check
curl http://localhost:18789/
# Expected: HTTP 200 ✅

graph TD
    A[🚀 LaunchAgent Starts Gateway] --> B[Level 0: Preflight]
    B -->|"Config valid"| C[exec gateway — launchd tracks PID]
    B -->|"Config invalid"| D[AI Recovery Session + backoff → retry]
    C --> E{Stable?}
    E -->|Repeated crashes| F[Level 1: KeepAlive]
    F -->|"Instant restart (0-30s)"| G{Stable?}
    G -->|Yes| Z[✅ Online]
    G -->|Repeated crashes| H[Level 2: Watchdog]
    H -->|"HTTP check every 3min"| I{Stable?}
    I -->|Yes| Z
    I -->|"30min continuous failure"| J[Level 3: AI Recovery]
    J -->|"Autonomous diagnosis & fix"| K{Fixed?}
    K -->|Yes| Z
    K -->|No| L[Level 4: Human Alert]
    L -->|"Discord / Slack / Telegram"| M[👤 Manual Fix]

    style A fill:#74c0fc
    style B fill:#74c0fc
    style Z fill:#51cf66
    style J fill:#4dabf7

Level 0 (new in v3.2): Catches config corruption, missing .env keys, and broken JSON before the gateway even starts — preventing crash loops from bad config entirely.

Based on an audit of 14 real incidents (Feb 2026):

9 of 14 incidents resolved fully autonomously. The remaining 5 escalated correctly to Level 4 — the system worked as designed.

Level 0: Preflight 🔍 (every cold start)
│  Validates binary, .env keys, JSON configs before exec
│  On failure: AI recovery session (tmux) + exponential backoff
│  scripts/gateway-preflight.sh
│
▼  passes
Level 1: KeepAlive ⚡ (0-30s)
│  Instant restart on any crash
│  Built into ai.openclaw.gateway.plist
│
▼  repeated failures
Level 2: Watchdog v4.1 🔍 (3-5 min)
│  HTTP + PID + memory monitoring every 3 min
│  Exponential backoff: 10s → 30s → 90s → 180s → 600s
│  Crash counter auto-decay after 6 hours
│
▼  30 minutes of continuous failure
Level 3: AI Emergency Recovery 🧠 (5-30 min)
│  Auto-triggered — no manual intervention
│  Supports Claude, GPT-4, Gemini, Ollama (OPENCLAW_LLM_PROVIDER)
│  PTY session: reads logs → diagnoses → fixes
│  Documents learnings for future incidents
│
▼  all automation fails
Level 4: Human Alert 🚨
   Discord/Slack/Telegram notification with full context
   Log paths + recovery report attached

Unified notification library supporting Discord, Slack, Telegram, and extensible adapters — one config, all channels.

# Auto-detects channel from available env vars
# Set one (or more) of:
DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."
TELEGRAM_BOT_TOKEN="..."  # + TELEGRAM_CHAT_ID

# Or explicitly force a channel
NOTIFICATION_CHANNEL="slack"

Implementation: scripts/lib/notify.sh

Level 3 Emergency Recovery is no longer Claude-only. Set OPENCLAW_LLM_PROVIDER in your .env:

# Switch to GPT-4o
echo 'OPENCLAW_LLM_PROVIDER=openai' >> ~/.openclaw/.env
echo 'OPENAI_API_KEY=sk-...'        >> ~/.openclaw/.env

# Fully offline with Ollama (no API key needed)
echo 'OPENCLAW_LLM_PROVIDER=ollama' >> ~/.openclaw/.env
echo 'OPENCLAW_LLM_MODEL=llama3.2'  >> ~/.openclaw/.env

Implementation: scripts/lib/llm-gateway.sh

Expose real-time recovery metrics for Grafana dashboards and alerting rules.

# Start the exporter (port 9090 by default)
bash scripts/start-metrics-exporter.sh start

# Test
curl -s http://localhost:9090/metrics

# Stop / restart
bash scripts/start-metrics-exporter.sh stop
bash scripts/start-metrics-exporter.sh restart

# Custom port
OPENCLAW_METRICS_PORT=8080 bash scripts/start-metrics-exporter.sh start

# prometheus.yml
scrape_configs:
  - job_name: 'openclaw'
    static_configs:
      - targets: ['localhost:9090']
    scrape_interval: 30s

# Alert: gateway down for >5 minutes
openclaw_gateway_healthy == 0

# Alert: recovery rate drops below 50%
openclaw_recovery_rate_percent < 50

Implementation: scripts/prometheus-exporter.py · scripts/start-metrics-exporter.sh

Generate a Markdown report of the last 7 days of incidents:

# Print to stdout
bash scripts/incident-digest.sh

# Print + post to Discord
bash scripts/incident-digest.sh --discord

Sample output:

## 📊 Weekly Incident Digest

**Period**: 2026-03-18 → 2026-03-25

| Metric | Value |
|--------|-------|
| Total incidents | 7 |
| Auto-resolved | 5 (71%) |
| Escalated to human | 2 |
| Autonomy rate | 71% |

✅ System performing well (autonomy rate on target)

✅ Done: 4-tier architecture · Claude AI integration · install.sh automation · Linux systemd · Level 2→3 auto-escalation · Discord/Telegram alerts · Preflight validation (v3.2) · LLM-agnostic layer — Claude, GPT-4, Gemini, Ollama (v3.3) · Prometheus metrics exporter (v3.3) · Multi-channel notifications — Discord/Slack/Telegram (v3.4) · Docker Compose support (v3.4) · --dry-run demo mode · Weekly incident digest

🚧 Next: Grafana dashboard template · Multi-node clusters

🔮 Future: Kubernetes Operator

🗳️ Vote on features →

Found this useful? A ⭐ on GitHub helps others discover it.

• 💬 Discussions — questions, ideas, show-and-tell
• 🐛 Issues — bug reports and feature requests
• 🤝 Contributing — PRs welcome

No secrets in code. .env for all webhooks. Lock files prevent races. All recoveries logged.

Level 3 AI access: OpenClaw config, gateway restart, log files — intentional for autonomous recovery.

All MIT licensed, all battle-tested on the same 24/7 production instance.

Bug reports, feature requests, docs improvements welcome. 📋 Contribution Guide →

Community: Discussions · Issues · Discord