Reference telegram bot robustness layers
| # | Layer | Where | Triggers when |
|---|---|---|---|
| 1 | Brain auto-retry | forge_telegram_brain.py::_call_sonnet_tracked |
Sonnet returns timeout / parse error / subprocess error. One retry after 1s sleep. |
| 2 | n8n call retry | forge_telegram_inbox_brain.py::n8n |
n8n returns 5xx OR connection error / timeout. One retry after 1s sleep. 4xx is NOT retried (real bugs need surfacing). |
| 3 | Truthfulness guard | forge_telegram_brain.py::_enforce_truthfulness |
Any tool result starts with a known failure pattern but the LLM's reply doesn't acknowledge it. Reply gets prefixed with "⚠️ Action(s) failed: ..." |
| 4 | Auto-fallback to Inbox | forge_telegram_capture_bot.py + coordinator |
brain.handle() returns one of Brain timed out, Brain error:, Brain call failed, Couldn't parse response. Bot calls tool_save_to_inbox directly with the raw user message so the thought is never lost. |
| 5 | systemd Restart=on-failure | /etc/systemd/system/forge-inbox-capture.service etc. |
Bot process crashes (uncaught exception). systemd restarts in 5s. |
| 6 | Watchdog timer | forge-bot-watchdog.timer (every 5 min, OnUnitActiveSec=5min) |
Any of capture / coordinator / remote-bridge service is inactive. Watchdog restarts it via sudo -n systemctl restart and pushes a notify alert. Critical-priority alert if restart fails. Logs to forge/logs/bot-watchdog.log. |
| 7 | Per-call timeout 90s | forge_telegram_brain.py::_call_sonnet_once |
Bumped from 45s 2026-04-29 after Justin saw a real timeout. Combined with Layer 1's retry = 2 attempts × 90s = 3 minutes of headroom on bad days. |
| 8 | Reaction UX | both bot files: react(), _typing_heartbeat() |
Bot reacts 👀 on receive, refreshes typing indicator every 4s, then 👌 on success or 🤔 on failure. Single-bubble final reply. No "Got it" noise. |
| 9 | Multi-intent in capture | forge_telegram_brain.py::CAPTURE_RULES |
Capture brain prompt now says "Multi-intent messages ARE expected and you MUST handle them in one shot. Up to 4 actions per iter is fine." Replaced the prior "if you find yourself chaining 3 tools you're doing too much, save to inbox" rule which deflected real intent. |
Failure modes the stack does NOT cover¶
- Anthropic API outage > 3 minutes: both retry attempts will time out. User gets the auto-Inbox fallback. Acceptable.
- n8n down > 3 seconds: both retry attempts fail. Tool returns error, truthfulness guard surfaces it. User sees "⚠️ Action failed".
- Notion API down: same as above, surfaced via truthfulness guard.
- Bot service crashes too often: systemd Restart=on-failure has no rate limit configured; runaway crash loops would hammer systemd. Mitigation: watchdog detects "still inactive after restart" and pushes a critical-priority alert.
Manual ops¶
# Run watchdog once on demand
sudo systemctl start forge-bot-watchdog.service
# Recent watchdog activity
tail /home/justinwieb/forge/logs/bot-watchdog.log
# All bot statuses at a glance
systemctl is-active forge-inbox-capture forge-lifeos-coordinator forge-remote-bridge
[Claude Code, bot-refiner-v2, 2026-04-29]