Skip to content

name: reference-forge-metrics description: forge-metrics CT 111, Prometheus + Grafana (metrics.justinsforge.com) + Uptime Kuma (uptime.justinsforge.com); node_exporter fleet-wide, smartctl_exporter on Finn; deploy + provisioning scripts type: reference

forge-metrics stack (CT 111, 192.168.86.85, ssh metrics)

Built 2026-06-10. Time-series history + dashboards + external uptime checks + SMART disk health. forge-monitor stays the pager; Grafana alerts only where it's blind (trend forecasting, SMART).

Pieces

What Where Notes
Prometheus metrics:9090, docker metrics_prometheus_1 30s scrape, 90d retention. Targets generated from data/metrics_hosts.json (SINGLE SOURCE; edit it + rerun deploy)
Grafana https://metrics.justinsforge.com (CF Access) admin creds ~/.forge-secrets/grafana-metrics.env. Dashboards: Node Exporter Full (API-imported), Disk Health SMART (git: infra/metrics/grafana/dashboards/)
Uptime Kuma https://uptime.justinsforge.com (CF Access) creds ~/.forge-secrets/uptime-kuma.env; 10 HTTP monitors on public hostnames, Telegram down/up via notify-out bot token (send-only, no long-poll conflict)
node_exporter every fleet host :9100 installer scripts/forge_node_exporter_deploy.sh (idempotent, version-pinned). minecraft CT excluded while stopped (see _disabled_hosts in the json)
smartctl_exporter Finn :9633 systemd smartctl_exporter.service; covers nvme0/nvme1/sda

Operate

  • Deploy/converge stack after editing infra/metrics/ or data/metrics_hosts.json: scripts/forge_metrics_deploy.sh
  • Add/refresh Kuma monitors (edit PUBLIC_MONITOR_URLS first): ~/.venvs/kuma/bin/python scripts/forge_uptime_kuma_provision.py
  • Add a host: row in data/metrics_hosts.json, run node_exporter deploy, run metrics deploy.

Alerts (Grafana-managed, Telegram contact point)

  • forge-disk-full-14d: predict_linear says a filesystem hits full within 14 days
  • forge-smart-prefail: SMART status failed or NVMe critical warning
  • forge-backup-stale: forge_backup_last_success_timestamp_seconds older than 26h (metric written by forge_backup_nightly.sh via node_exporter textfile collector, /var/lib/node_exporter/forge_backup.prom on Console)

Dashboards

  • Fleet Overview (forge-fleet): Grafana home page; hosts-up, backup age, SMART-failing count, per-host CPU/mem/disk table + trends
  • Node Exporter Full: per-host deep dive (host dropdown)
  • Disk Health SMART: Finn drives
  • Kuma also carries 4 history-only internal monitors (lan-gateway ping, adguard-dns query, plex-direct, home-assistant-direct) with notifications OFF; forge-monitor stays the internal pager
  • Config: infra/metrics/grafana/provisioning/alerting/forge-alerts.yml (bot token via grafana.env interpolation)

Gotchas

  • Prometheus runs as nobody in-container: anything mounted into /etc/prometheus must be world-readable (deploy script chmods targets file; a 600 file = silent empty discovery + ERROR in docker logs only).
  • Grafana provisioning $ENV interpolation coerces numeric strings to numbers; telegram chatid must stay a literal quoted string in the yaml.
  • Provisioning yaml errors crash-loop the Grafana container; check docker logs metrics_grafana_1 first if metrics.justinsforge.com dies after a deploy.
  • Kuma monitors accept 301/302/401/403 as UP on purpose: CF Access-gated hosts answer those to anonymous probes while still proving tunnel + origin alive.
  • Negative-DNS trap (bit us 2026-06-10): if anything on the LAN queries a new hostname BEFORE its CNAME exists, AdGuard caches NXDOMAIN and LAN devices can't reach the site while the internet can. Order matters: create DNS first, then Kuma monitors; fix is ssh adguard "docker restart adguard-home".

[Claude Code]