Feedback alert dedup allclear pattern
name: Mount watchdog alert dedup and all-clear pattern description: Alert dedup state machine: warn once, suppress repeats, escalate sustained, always send recovery type: feedback
The mount watchdog was sending 8 identical critical pages for a single overnight mount flap with no recovery notification. Justin called this out: repeated alerts with no escalation difference and no all-clear erode trust and get ignored.
The fix shipped: forge/data/mount-watchdog-state.json tracks per-mount state. Policy:
- First failed cycle: warning (suppress for 30 min)
- Sustained >30 min: critical (suppress for 60 min)
- Mount recovers after any alert: info RECOVERED page
- Re-page once per suppress window if still failing
Why: Justin explicitly said he stops trusting alerts when they flood with no meaningful signal change and never confirm resolution. 8 identical critical pages for one overnight flap is the failure mode.
How to apply: For any new monitor or watchdog script: (1) dedupe by tracking last-alert-time per resource in a state file, (2) escalate severity with duration rather than repeating the same level, (3) always send an all-clear when state returns to healthy after an alert was sent. This pattern is now canonical in forge_monitor_mount_watchdog.sh.
[auto-memory session 45a71d6c-0e15-44fc-8e2a-ef4c64b9f466, confidence 0.88, mode direct]
Historical (auto-merged from feedback_alert_dedup_and_allclear.md on 2026-05-05)¶
name: Alerts must deduplicate and send all-clear description: Repeated identical critical alerts for the same flapping condition erode trust; watchdogs must deduplicate and confirm recovery type: feedback
When a watchdog or monitor fires the same alert repeatedly for a sustained condition (e.g. mount flapping every 10 min overnight), Justin receives a flood of identical critical messages with no recovery confirmation. This actively erodes trust in the alerting system — he starts ignoring criticals.
Rule: every alerting script must implement:
1. Dedupe: suppress repeat alerts for the same condition for at least 30 min after first page.
2. Escalation tier: first failure = warning (monitoring), sustained > 30 min = critical (intervene).
3. All-clear: when condition resolves, send a recovery notification so Justin knows no action is needed.
Why: Justin explicitly said four morning alerts for the same mount issue made him question whether the system was worth trusting. The mount watchdog was technically correct but behaviorally useless — same wording for transient flap vs sustained outage, no recovery message.
How to apply: Before shipping any new monitor or watchdog script, verify it has a state file (or equivalent) tracking last-alert-time per condition, a tiered severity ramp, and an explicit recovery notify path. Apply retroactively when patching existing monitors (mount watchdog is the first candidate).
[auto-memory session 45a71d6c-0e15-44fc-8e2a-ef4c64b9f466, confidence 0.88, mode direct]