Skip to content

Telegram fleet audit + optimize brief, 2026-06-09

Worker brief: audit all five Telegram bots, ship safe optimizations, and stage a feature menu for Justin. Approved by Justin 2026-06-09 ("audit all telegram bots and optimize and add some features and tooling").

The fleet

Bot Service Brain / script
capture (inbox) forge-inbox-capture.service scripts/forge_telegram_inbox_brain.py
coordinator (lifeos) forge-lifeos-coordinator.service scripts/forge_telegram_brain.py
notify-out (outbound only, see reference_notify.md) scripts/forge_notify.sh path
remote-bridge forge-remote-bridge scripts/forge_telegram_remote_bridge.py
general-purpose (see reference_general_purpose_bot.md) scripts/forge_telegram_general_purpose_bot.py

Shared helpers: scripts/forge_telegram_format.py (formatting + boundary sanitizer), quota tracking via forge_quota_tracker.

Load-bearing constraints (read these memories before touching anything)

  • feedback_telegram_bot_token_ownership.md: ONE long-poller per token, ever. Never start a second poller to test.
  • feedback_telegram_bot_ux_reactions.md: reactions + typing indicator, not "Got it" text bubbles.
  • feedback_telegram_no_tables.md: no markdown tables to Telegram; convert at send boundary.
  • feedback_em_dash_sanitize_at_boundary.md: every LLM-to-Telegram boundary calls strip_em_dashes.
  • feedback_no_threads_with_subprocess_run.md: multiprocessing, not threads, around subprocess.run.
  • reference_telegram_bot_robustness_layers.md: nine defense layers exist; don't regress them.
  • feedback_capture_handles_oneshots.md: capture does single-intent actions end-to-end; done:false guard for external-data tools.
  • feedback_coordinator_checkin_tone.md: check-ins direct/observational, never motivational.
  • Note: capture + coordinator were restarted 2026-06-09 ~13:55 with a new Context API token loader (~/.forge-secrets/context-api.env); that change is committed and verified.

Scope

Phase A, audit (read-only): per bot: journal errors last 14d, restart counts, watchdog fires, brain latency (quota tracker JSONL has per-call latency), dead/unused tools in the tool lists, prompt drift vs the memories above, token/cost per day, UX gaps. Check forge-bot-watchdog.timer and daily canary results for chronic offenders. Write findings to this file under a Findings section, prioritized.

Phase B, optimize (ship without asking): dead code removal, retry/timeout tuning, latency wins (cache, model choice per feedback_parallel_subagents_haiku.md analog: cheap model for classify-only calls), prompt tightening, logging gaps, anything the audit flags as a regression vs the robustness layers. One service restart per change, verify active + send yourself a test message is NOT possible (you cannot message the bots), so verify via journal startup health + canary script if available.

Phase C, features (stage, then notify): draft a menu of 5 to 8 concrete feature/tooling ideas with effort estimates (probe-first per feedback_time_estimates_probe_first.md). Examples to evaluate, not mandates: voice-note transcription on capture, scheduled digest improvements, inline-keyboard quick actions on morning report, a /bots-health skill, per-bot quota budget alerts. Implement only the ones that are small AND zero-risk to live flows; for the rest, push the menu via scripts/forge_notify.sh info "Telegram fleet: feature menu" "<summary + path to this handoff>" and stop.

Boundaries

  • Never commit tokens; secrets live in ~/.forge-secrets/telegram-*.env.
  • No bot renames, no new bots, no token changes.
  • Don't touch scripts/forge_morning_report.py (another session owns it today) except to read.
  • Restarts: one at a time, verify systemctl is-active + 30s of clean journal before the next.
  • Commit in logical chunks with the repo's area: summary style.

Update this handoff with outcomes; register any new tooling in MEMORY.md per feedback_register_tools.md.

[Claude Code, forge-audit fix session]

Findings (Phase A, 2026-06-09, telegram-fleet session)

Audit window: 14 days of journals, full forge/logs/telegram-*.log history (rotates to ~May 15), quota JSONL May + June (5,606 rows).

Priority-ordered:

  1. Watchdog covered 3 of 5 long-pollers. The fleet actually runs five long-poll services (the brief's table misses one: forge-notify-bot.service is an inbound slash-command poller on the notify-outbound token, not outbound-only). forge_bot_watchdog.py guarded only capture, coordinator, remote-bridge; forge-general-purpose.service and forge-notify-bot.service had no auto-heal. SHIPPED: both added.
  2. TimeoutStopSec=15s + KillMode=mixed SIGKILLed in-flight brain calls on restart. SIGTERM stops the main poller gracefully, but an in-flight claude -p child (med 9 to 11s, p90 ~70s) outlives 15s and gets SIGKILLed; two Failed with result 'timeout' events in 14d (May 29, Jun 05 on capture + remote-bridge). SHIPPED: 90s on all five units.
  3. Benign long-poll timeouts logged at ERROR. getUpdates read timeouts (~2/day/bot) and the Jun 03 DNS blip land at ERROR in every bot log, polluting error greps (this audit had to filter them). SHIPPED: transient requests.Timeout/ConnectionError downgraded to WARNING in all five tg() copies.
  4. forge_notify.sh skipped the em-dash boundary sanitize (doctrine Section 9; callers are LLM-driven). SHIPPED: strip_em_dashes shell port added before any channel send.
  5. tg() is copy-pasted 6x (five fleet bots + autoagent reporter), a single-source-of-truth violation; the log-level fix in (3) had to be applied five times. Consolidation into forge_telegram_format.py is on the feature menu, not shipped (touches every bot plus a full restart cycle).
  6. Brain latency p90 is ~70s (coordinator n=162 med 10.7s p90 68.8s; capture n=49 med 8.8s p90 70.8s). Driver: each loop iteration shells out a fresh claude -p (cold CLI start per call). The 180s coordinator wall-clock budget and hot-tool-subset gating already bound the damage. Structural fix (persistent Anthropic SDK session + prompt caching) is on the menu.
  7. Tool catalog: 71 declared, 44 with zero invocations in the ~3.5-week log window. Not shipped as dead-code removal: several fire via separate timer scripts (weekly_review, followup_unscheduled, sync_workouts import the brain directly), and the coordinator already sees a hot subset with expand-on-demand. Flagged for a prune pass with Justin instead.
  8. Bridge quota rows with latency_ms=0 and 10 failures are all from May, pre-dating the duration_ms fallback landed in the 2026-05-11 Phoenix sweep. No June recurrence. No action.
  9. Restart counts, watchdog, canary all healthy: NRestarts=0 across the fleet, watchdog firing every 5 min, canary PASS 4/4 today 06:00. Old tool-result errors in logs (calendar invalid_grant May 15, Notion Tags property, n8n email-search 500) predate today's Context API token loader fix; watch only.

Phase B shipped (all verified)

Change Files Verification
Watchdog guards all 5 pollers scripts/forge_bot_watchdog.py 14:24 watchdog log: "ok all 5 bot services active"
TimeoutStopSec 15s to 90s 5 units in /etc/systemd/system/ (not in repo) + daemon-reload systemctl cat shows 90s on all five
Transient tg() errors to WARNING forge_telegram_capture_bot.py, _coordinator_bot.py, _general_purpose_bot.py, _notify_bot.py, _remote_bridge.py py_compile clean; rolling restarts one at a time, each active + 30s clean journal
notify.sh em-dash sanitize scripts/forge_notify.sh bash -n clean; sed verified on sample string
/bots-health skill (Phase C quick win) scripts/forge_bots_health.sh, .claude/skills/bots-health/SKILL.md end-to-end run, all sections render

All five services restarted clean (graceful SIGTERM shutdown observed on each), one at a time per the brief.

Phase C feature menu (probe-first estimates)

Staged for Justin to pick; only /bots-health (zero-risk, read-only) was built.

  1. Consolidate tg()/send() into forge_telegram_format.py. Removes the 6-way copy-paste. Probe done in this audit (blocks are byte-identical). Build ~1h + 5 rolling restarts. Medium risk: touches every bot's send path.
  2. Brain latency: persistent Anthropic SDK calls instead of claude -p per iteration, with prompt caching of the tool catalog. Expected p90 from ~70s to under 20s, and quota visibility improves (real token counts, not estimates). Probe ~1h (billing implications per feedback_subscription_over_api_for_automation.md), build 3 to 4h, staged rollout one persona at a time.
  3. Voice-note transcription on capture. File-download plumbing already exists in the capture bot; needs Telegram voice (.oga) to text via local whisper or API, then feed the brain. Probe 20 min, build 1 to 2h.
  4. Per-bot quota budget alerts. Daily per-invoker token budgets in the quota aggregator + forge_notify.sh warning on breach. ~1h, low risk (read-side only plus one notify path).
  5. Heartbeat liveness for all five bots. Only the coordinator has a stall heartbeat today; the watchdog's stall detection covers one bot of five. ~45 min, touches each poll loop + rolling restarts.
  6. Inline-keyboard quick actions on the morning report (ack / snooze / open-task). Blocked today: forge_morning_report.py is owned by another session; revisit after it lands. Est 1 to 2h once unblocked.
  7. Tool catalog prune pass with Justin. Walk the 44 zero-hit tools, retire or document each. ~1h interactive.

Status: COMPLETE 2026-06-09. Phases A + B done, menu staged, notify sent.

[Claude Code, telegram-fleet session, 2026-06-09]