Skip to content

Metrics stack, metrics.justinsforge.com (spec, awaiting go)

Goal: time-series metrics + dashboards + uptime checks + SMART disk health for the whole fleet. Complements forge-monitor (which stays the pager; this stack is history and trends, alerting added only where forge-monitor is blind).

Components

Piece Where Surface
Prometheus (TSDB, 90d retention) new LXC forge-metrics on Finn (Debian 12, 2 cores, 2GB) internal only
node_exporter every host: Finn, Console, all LXCs/VMs scraped by Prometheus
smartctl_exporter Finn (owns NVMe + 26TB HDD) SMART panels in Grafana, satisfies "SMART on metrics.justinsforge.com"
Grafana forge-metrics metrics.justinsforge.com (CF tunnel + Access, landing tile)
Uptime Kuma forge-metrics uptime.justinsforge.com (CF tunnel + Access, landing tile); external-perspective checks on all public hostnames

Tasks

  1. Create CT 111 forge-metrics, static IP, SSH alias, fleet.md row.
  2. Ansible-free fanout script scripts/forge_node_exporter_deploy.sh installs node_exporter on each host via existing SSH aliases (idempotent).
  3. smartctl_exporter on Finn bare metal (needs raw disk access).
  4. Prometheus scrape config generated from a single hosts list (single source of truth: data/metrics_hosts.json).
  5. Grafana: provisioned datasource + 3 dashboards as code (fleet overview, disk health/SMART, per-host drilldown). Admin behind CF Access.
  6. Uptime Kuma: monitors for every .justinsforge.com + .justinkrystal.com hostname, Telegram notify via existing notify bot on down/up (respect alert-dedup pattern).
  7. Grafana alerts ONLY for: disk >85% with 14d-to-full forecast, SMART pre-fail attributes. Route through forge_notify.sh webhook shim. Everything else stays forge-monitor's job.
  8. Cloudflare: /cf-add metrics ... + /cf-add uptime ..., CF Access policies, two landing tiles.
  9. Register: reference_forge_metrics.md + MEMORY.md index lines; smoke test (pull a metric via API, fail a fake monitor, confirm notify).

Verification: every host shows in fleet overview; SMART panel shows both Finn drives; killing nginx on a test host pages within 2 min via Kuma.

Estimate: probe 0.5h + build 3-4h.

[Claude Code]