Compare commits

...

13 Commits

Author SHA1 Message Date
kitadmin 30953399b6 fix: null gateway latency; skip aiwanbal generic modem_type for carrier detection
- Gateway ping latency is LAN RTT (<1ms), not WAN latency; set to null so
  dashboards don't show misleading sub-ms values for Eyeride-NAT'd interfaces
- aiwanbal reports modem_type=generic for wired ISP ports; treat same as none
  so carrier detection falls through to gateway IP pattern matching (AT&T etc.)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 03:48:32 +00:00
kitadmin 088c256f58 fix: ping local gateway instead of curl when ICMP gives 100% loss
curl --interface fails for root on Synology due to policy routing. Pinging the
local gateway (e.g. Eyeride at 192.168.10.1) works reliably and confirms the
WAN path is up even when internet ICMP is NAT-blocked.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 03:38:03 +00:00
kitadmin bbbc873bdb fix: HTTP fallback probe when ICMP reports 100% loss (Eyeride NAT blocks ICMP)
Eyeride 5G device passes HTTP traffic but drops ICMP, causing false 100% loss.
When ping gives 100% loss, try curl --interface to confirm connectivity; if
HTTP probe succeeds, correct loss to 0% and derive latency from connect time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 03:31:47 +00:00
kitadmin 8f2d6a7cbd fix: sanitize _mtime with tr to prevent ash arithmetic crash on Synology
stat output can include non-numeric chars on some RT2600ac firmware versions,
causing 0 to fail. Strip with tr -cd '0-9' before arithmetic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 03:29:10 +00:00
kitadmin a4ba9f8fcd fix: ash-compatible arithmetic in _cached_speedtest; clean up speedtest null guards
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 03:26:04 +00:00
kitadmin 8ea704f7e3 fix: use ingest HTTP API instead of direct Postgres for VPS telemetry; cache speedtest results per interface
- hub/server.py: replace psycopg2 Postgres connection with urllib HTTP call to
  ingest /api/devices endpoint (fleet Postgres is Docker-internal, not reachable
  from the VPS directly)
- telemetry-synology.sh: run_speedtest() now writes per-interface cache files
  (speedtest-result-<iface>); _cached_speedtest() reads cache if < 35 min old;
  _collect_wan() uses cached result so throughput shows in telemetry without
  blocking every 1-min cycle

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 03:06:08 +00:00
kitadmin 8a648dc637 feat: smart WAN role detection, aiwanbal integration, speedtest support
- Auto-detect cellular vs fiber/Starlink regardless of physical port:
  aiwanbal modem_type → Eyeride subnet (192.168.10.x) → gateway patterns
- Integrate aiwanbal/SmartAiBalancer state as primary data source when
  available (fresh <120s); fall back to self-collected ping metrics
- Add carrier detection: AT&T for fiber gateways, Eyeride for cellular
- Add run_speedtest() with speedtest-cli/iperf3/curl download fallback
- Extract jitter (mdev) from ping output for richer metrics
- Consistent canonical GL format: modem_0001=cellular, wan=ethernet/fiber

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 02:29:49 +00:00
kitadmin 8dba04a1d0 fix: auto-detect WAN roles — cellular vs fiber
- Detect Eyeride cellular interface by checking 192.168.10.x subnet
- Map correctly: ethernet/fiber → wan, Eyeride 5G → modem_0001
- Add carrier detection: AT&T for fiber gateways, Eyeride for cellular
- Fix sel_primary: fiber is active by default unless cellular route exists
- Fixes console showing 'on cellular' when actually on AT&T fiber

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 02:19:18 +00:00
kitadmin 58ac96fd3c fix: prevent double-ping fallback corrupting WAN metrics
- sudo ping returns exit 1 on any packet loss, which triggered the
  || fallback to direct ping, causing duplicate output concatenation
- Now check exit code explicitly: only fall back on exit >= 2 (real error)
- Fixes loss_pct showing 100100 instead of 100 on failed WANs

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 02:10:35 +00:00
kitadmin 91373df27d fix: sanitize newlines in telemetry output for valid JSON
- ipgeo_gps: pipe curl output through tr -d to strip newlines
- _ping_iface: pipe printf output through tr -d for newline safety

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 01:57:56 +00:00
kitadmin db04341f7c feat: unified fleet telemetry — canonical format, GPS/IP geo, WAN health dashboard
- Rewrite telemetry-synology.sh: canonical GL format (modem_0001/wan keys),
  self-contained metrics (no aiwanbal), Eyeride GPS with IP geo fallback
- Add /api/fleet-telemetry to hub: joins tunnel status with fleet Postgres
- Update dashboard.html: per-device WAN health bars, GPS, signal strength
- Fix hub/ingest.sh normalisation: remap old wan1/wan2 → modem_0001/wan
- Bump SPK version: 0.1-0001 → 0.5.0-0001 for GL parity

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 01:47:02 +00:00
kitadmin d86a7e49b5 feat: BusyBox-compatible telemetry script for Synology
Uses sed instead of grep -P for BusyBox ash compatibility.
Writes JSON to temp file to avoid shell escaping issues.
Posts to fleet hub ingest at 167.172.237.162:8080.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-22 23:13:57 +00:00
kitadmin 879f97c843 fix: x4078 remote recovery + daemon wait-wedge bug + SPK rebuild
- docs/incident-log: mark x4078 recovered remotely via Tailscale (no
  physical access needed); document the recovery steps and root cause
- connect-daemon.sh: remove bare `wait` that wedged the retry loop
  forever on the setsid'd tailscaled child — once the tunnel failed the
  daemon could not self-heal until the package was restarted; now the
  main loop falls through start_tunnel (already blocking+retrying) and
  restarts the whole cycle cleanly
- SPK rebuilt (32839680 bytes) with the daemon fix included
- Also documents the two latent lessons: (1) a field router is only
  inaccessible when ALL three paths fail; (2) always check Tailscale +
  QuickConnect before assuming a truck roll is required

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 22:23:53 +00:00
10 changed files with 798 additions and 256 deletions
+13 -1
View File
@@ -302,7 +302,19 @@ Kit-connect is the unified replacement for Sections 6 and 7. One SPK installs bo
**Resolution for x4662**: Kittunnel restarted via Package Center through QuickConnect. Tunnel stable on port 2229.
**Resolution for x4324**: Kit-connect daemon reconnected after authorized_keys fix on VPS. Tunnel stable on port 2225.
**Pending**: x4078 needs physical access — kit-connect daemon is installed and connecting but fails because it forwards to port 22 instead of 2223.
**Resolution for x4078 (2026-07-22, remote — NO physical access needed)**: Recovered entirely over the
still-live **Tailscale** path (`100.116.71.43`, SRM SSH `:2223`) — QuickConnect was also up the whole
time. The reverse tunnel was the *only* dead path. Root cause on-device: `TUNNEL_LOCAL_SSH_PORT=22` in
`/etc/kit-connect/connect.conf` (SRM sshd listens on 2223). Fix: SSH in over Tailscale from a tailnet
node (e.g. the hub), `sed -i 's/^TUNNEL_LOCAL_SSH_PORT=.*/TUNNEL_LOCAL_SSH_PORT=2223/'` the conf, then
`synopkg stop kit-connect; synopkg start kit-connect`. Tunnel came back on port 2226; end-to-end verified.
Lesson: **a field router is only "inaccessible" if ALL of {Tailscale, QuickConnect, reverse tunnel} are
down — check each before assuming physical access is required.**
**Latent daemon bug found**: `connect-daemon.sh` main loop ends each iteration with a bare `wait`, which
blocks forever on the `setsid`-launched `tailscaled` child. Once the tunnel started failing, the loop
wedged for ~58 min and never self-healed on the corrected config until the package was restarted. Fix
the `wait` (e.g. `wait "$SSH_PID"` or `wait -n`) so the retry loop can recover on its own.
**SPK fixed**: Rebuilt with `which` instead of `command -v` and `TUNNEL_LOCAL_SSH_PORT=2223`. Available at `https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk`.
+107 -10
View File
@@ -23,18 +23,36 @@ body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
.stat-card .lbl { font-size: 10px; color: var(--dim); text-transform: uppercase; margin-top: 2px; }
.stat-card .num.online { color: var(--green); }
.stat-card .num.offline { color: var(--red); }
.grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(400px, 1fr)); gap: 16px; }
.grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(420px, 1fr)); gap: 16px; }
.card { background: var(--card); border: 1px solid var(--border); border-radius: 8px; padding: 16px; }
.card h2 { font-size: 16px; margin-bottom: 12px; display: flex; justify-content: space-between; align-items: center; }
.card h2 { font-size: 16px; margin-bottom: 8px; display: flex; justify-content: space-between; align-items: center; }
.badge { font-size: 11px; padding: 2px 8px; border-radius: 12px; font-weight: 600; }
.badge-gl { background: rgba(88,166,255,0.15); color: var(--blue); }
.badge-synology { background: rgba(210,153,34,0.15); color: var(--amber); }
.badge-ver { background: rgba(139,148,158,0.15); color: var(--dim); font-size: 10px; }
.dot { display: inline-block; width: 8px; height: 8px; border-radius: 50%; margin-right: 4px; }
.dot-up { background: var(--green); }
.dot-down { background: var(--red); }
.row { display: flex; justify-content: space-between; padding: 5px 0; font-size: 13px; border-bottom: 1px solid rgba(255,255,255,0.04); }
.row { display: flex; justify-content: space-between; padding: 4px 0; font-size: 13px; border-bottom: 1px solid rgba(255,255,255,0.04); }
.row .label { color: var(--dim); }
.row .val { font-family: monospace; font-size: 12px; }
.wan-section { margin-top: 8px; padding-top: 8px; border-top: 1px solid var(--border); }
.wan-section .title { font-size: 11px; color: var(--dim); text-transform: uppercase; letter-spacing: 1px; margin-bottom: 6px; }
.wan-bar { display: flex; align-items: center; gap: 8px; margin-bottom: 4px; font-size: 12px; }
.wan-bar .member { font-family: monospace; min-width: 90px; font-size: 11px; }
.wan-bar .score { font-weight: 700; min-width: 32px; text-align: right; }
.wan-bar .score.good { color: var(--green); }
.wan-bar .score.warn { color: var(--amber); }
.wan-bar .score.bad { color: var(--red); }
.wan-bar .metric { color: var(--dim); font-size: 11px; }
.wan-bar .signal { font-size: 10px; }
.wan-bar .signal.good { color: var(--green); }
.wan-bar .signal.warn { color: var(--amber); }
.wan-bar .signal.bad { color: var(--red); }
.active-dot { display: inline-block; width: 6px; height: 6px; border-radius: 50%; background: var(--green); margin-right: 4px; flex-shrink: 0; }
.gps-line { font-size: 11px; color: var(--dim); margin-top: 4px; }
.gps-line a { color: var(--blue); text-decoration: none; }
.gps-line a:hover { text-decoration: underline; }
.access { margin-top: 12px; padding-top: 12px; border-top: 1px solid var(--border); }
.access .title { font-size: 11px; color: var(--dim); text-transform: uppercase; letter-spacing: 1px; margin-bottom: 8px; }
.cmd { background: #0d1117; padding: 6px 10px; border-radius: 4px; font-family: monospace;
@@ -43,6 +61,7 @@ body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
.cmd a:hover { text-decoration: underline; }
.empty { color: var(--dim); text-align: center; padding: 60px 20px; font-size: 14px; }
.refresh { color: var(--dim); font-size: 11px; text-align: right; margin-top: 20px; }
.telem-warn { font-size: 10px; color: var(--amber); margin-left: 8px; }
</style>
</head>
<body>
@@ -59,8 +78,8 @@ body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
var HUB = window.location.origin;
function loadFleet() {
fetch(HUB + '/api/fleet')
.then(function(r) { return r.ok ? r.json() : Promise.reject(r.status); })
fetch(HUB + '/api/fleet-telemetry')
.then(function(r) { return r.ok ? r.json() : fetch(HUB + '/api/fleet').then(function(r2) { return r2.json(); }); })
.then(function(data) { render(data); })
.catch(function(e) {
document.getElementById('fleet').innerHTML =
@@ -69,18 +88,55 @@ function loadFleet() {
document.getElementById('refresh').textContent = 'Last updated: ' + new Date().toLocaleTimeString();
}
function scoreClass(s) {
if (s == null) return '';
if (s >= 70) return 'good';
if (s >= 40) return 'warn';
return 'bad';
}
function signalClass(rsrp) {
if (rsrp == null) return '';
if (rsrp >= -85) return 'good';
if (rsrp >= -105) return 'warn';
return 'bad';
}
function signalBars(rsrp) {
if (rsrp == null) return '—';
if (rsrp >= -85) return '▂▄▆█';
if (rsrp >= -95) return '▂▄▆_';
if (rsrp >= -105) return '▂▄__';
if (rsrp >= -115) return '▂___';
return '____';
}
function fmtLatency(ms) { return ms != null ? ms + ' ms' : '—'; }
function fmtLoss(pct) { return pct != null ? pct + '%' : '—'; }
function fmtScore(s) { return s != null ? s : '—'; }
function fmtUptime(s) {
if (s == null) return '—';
var h = Math.floor(s / 3600);
var m = Math.floor((s % 3600) / 60);
if (h >= 24) { var d = Math.floor(h / 24); return d + 'd ' + (h % 24) + 'h'; }
return h + 'h ' + m + 'm';
}
function render(data) {
var devices = data.devices || [];
var online = devices.filter(function(d) { return d.tunnel_up; }).length;
var telemOnline = data.telemetry_online || 0;
document.getElementById('stats').innerHTML =
'<div class="stat-card"><div class="num">' + devices.length +
'</div><div class="lbl">Registered</div></div>' +
'<div class="stat-card"><div class="num online">' + online +
'</div><div class="lbl">Online</div></div>' +
'</div><div class="lbl">Tunnel Online</div></div>' +
'<div class="stat-card"><div class="num offline">' + (devices.length - online) +
'</div><div class="lbl">Offline</div></div>' +
'<div class="stat-card"><div class="num">' + (data.hub || '') +
'</div><div class="lbl">Tunnel Offline</div></div>' +
'<div class="stat-card"><div class="num" style="font-size:20px">' + telemOnline +
'</div><div class="lbl">Telemetry Active</div></div>' +
'<div class="stat-card"><div class="num" style="font-size:18px">' + (data.hub || '') +
'</div><div class="lbl">Hub</div></div>';
var el = document.getElementById('fleet');
@@ -90,21 +146,62 @@ function render(data) {
}
el.innerHTML = devices.map(function(d) {
var isSynology = d.device_id.startsWith('x');
var isSynology = d.platform === 'synology' || d.device_id.startsWith('x');
var badge = isSynology
? '<span class="badge badge-synology">Synology</span>'
: '<span class="badge badge-gl">GL-XE3000</span>';
var verBadge = d.version ? '<span class="badge badge-ver">v' + esc(d.version) + '</span>' : '';
var status = d.tunnel_up
? '<span><span class="dot dot-up"></span>ONLINE</span>'
: '<span><span class="dot dot-down"></span>OFFLINE</span>';
var telemWarn = d.tunnel_up && !d.online_telemetry
? '<span class="telem-warn">(no telemetry)</span>' : '';
var ssh = 'ssh -p ' + d.tunnel_port + ' kitadmin@' + data.hub;
var web = d.dashboard_url;
// WAN health bars
var wanHtml = '';
var wans = d.wan_summary || [];
if (wans.length) {
wanHtml = '<div class="wan-section"><div class="title">WAN Health</div>';
for (var i = 0; i < wans.length; i++) {
var w = wans[i];
var activeMarker = w.active ? '<span class="active-dot"></span>' : '<span style="display:inline-block;width:6px;margin-right:4px"></span>';
var bars = signalBars(w.rsrp_dbm);
wanHtml += '<div class="wan-bar">' +
activeMarker +
'<span class="member">' + esc(w.member) + '</span>' +
'<span class="score ' + scoreClass(w.score) + '">' + fmtScore(w.score) + '</span>' +
'<span class="metric">' + fmtLatency(w.latency_ms) + ' / ' + fmtLoss(w.loss_pct) + ' loss</span>' +
(w.rsrp_dbm != null
? '<span class="signal ' + signalClass(w.rsrp_dbm) + '" title="RSRP: ' + w.rsrp_dbm + ' dBm">' + bars + ' ' + w.rsrp_dbm + ' dBm</span>'
: '') +
(w.carrier ? '<span class="metric" style="margin-left:4px">' + esc(w.carrier) + '</span>' : '') +
(w.technology && w.technology !== 'unknown' ? '<span class="metric">' + esc(w.technology) + '</span>' : '') +
'</div>';
}
wanHtml += '</div>';
}
// GPS line
var gpsHtml = '';
if (d.gps && d.gps.lat != null && d.gps.lon != null) {
var fixLabel = d.gps.fix === 1 ? ' (IP approx)' : (d.gps.fix >= 2 ? ' (GPS)' : '');
gpsHtml = '<div class="gps-line">📍 ' +
Number(d.gps.lat).toFixed(4) + ', ' + Number(d.gps.lon).toFixed(4) + fixLabel +
' — <a href="https://maps.google.com/?q=' + d.gps.lat + ',' + d.gps.lon + '" target="_blank">Google Maps</a>' +
'</div>';
}
return '<div class="card">' +
'<h2>' + esc(d.device_id) + ' ' + badge + ' <span style="font-size:13px">' + status + '</span></h2>' +
'<h2>' + esc(d.device_id) + ' ' + badge + verBadge + ' <span style="font-size:13px">' + status + telemWarn + '</span></h2>' +
'<div class="row"><span class="label">Tunnel Port</span><span class="val">' + d.tunnel_port + '</span></div>' +
'<div class="row"><span class="label">Assigned</span><span class="val">' + (d.assigned || '--') + '</span></div>' +
(d.uptime_s ? '<div class="row"><span class="label">Uptime</span><span class="val">' + fmtUptime(d.uptime_s) + '</span></div>' : '') +
(d.autonomous != null ? '<div class="row"><span class="label">Autonomous</span><span class="val">' + (d.autonomous ? '✅ Enabled' : '❌ Disabled') + '</span></div>' : '') +
wanHtml +
gpsHtml +
(d.tunnel_up
? '<div class="access">' +
'<div class="title">Direct Access</div>' +
+17 -9
View File
@@ -18,23 +18,31 @@ mkdir -p "$TELEM_DIR" "$BACKUP_DIR"
# ── Normalise telemetry ────────────────────────────────────────────
# Both platforms POST to /api/telemetry.
# GL uses: modem_0001/wan keys, no "platform" field
# Synology uses: wan1/wan2 keys, "platform":"synology"
# Output: unified schema with platform field + wan1/wan2 keys
# Canonical format uses modem_0001/wan keys (matching GL standard).
# Handles three variants:
# - GL (no platform field): add platform="openwrt", pass through
# - GL (platform="openwrt"): already canonical, pass through
# - Old Synology (platform="synology" + wan1/wan2): remap to modem_0001/wan
# - New unified Synology (platform="synology" + modem_0001/wan): pass through
normalise_telemetry() {
local json="$1"
local platform=""
json="$1"
platform=""
platform=$(echo "$json" | grep -o '"platform":"[^"]*"' | cut -d'"' -f4)
if [ -z "$platform" ]; then
# Pre-unified GL telemetry — add platform, remap keys
# Pre-unified GL telemetry — add platform field, keys already canonical
platform="openwrt"
json=$(echo "$json" | sed \
-e 's/"modem_0001"/"wan1"/g' \
-e 's/"platform":"openwrt"/"platform":"'"$platform"'"/')
-e 's/^{/{"platform":"'"$platform"'",/')
fi
# Remap old Synology wan1/wan2 keys to canonical modem_0001/wan
if echo "$json" | grep -q '"wan1"'; then
json=$(echo "$json" | sed \
-e 's/"wan1"/"modem_0001"/g' \
-e 's/"wan2"/"wan"/g')
fi
# Ensure both wan1 and wan2 keys exist (Synology uses wan1/wan2 natively)
echo "$json"
}
+150 -5
View File
@@ -7,6 +7,8 @@ Listens on :8080. Routes:
GET /api/register/<DEVICE_ID> — assign tunnel port, return JSON
GET /api/fleet — all registered devices with
tunnel status + access links
GET /api/fleet-telemetry — fleet status enriched with telemetry
from fleet Postgres (WAN health, GPS)
POST /api/authorize-key — authorize device tunnel key
"""
@@ -15,12 +17,14 @@ import os
import socket
import subprocess
import sys
import urllib.request
from http.server import HTTPServer, BaseHTTPRequestHandler
REGISTER_SH = "/opt/busfleet-hub/register.sh"
REGISTRY = "/opt/busfleet-hub/port-registry.json"
DASHBOARD = "/opt/busfleet-hub/dashboard.html"
HUB_IP = "162.243.83.36"
REGISTER_SH = "/opt/busfleet-hub/register.sh"
REGISTRY = "/opt/busfleet-hub/port-registry.json"
DASHBOARD = "/opt/busfleet-hub/dashboard.html"
HUB_IP = "162.243.83.36"
FLEET_API = os.environ.get("FLEET_API", "http://167.172.237.162:8080/api/devices")
# ── Helper: check if a TCP port is open (tunnel active) ──────────
@@ -33,6 +37,51 @@ def _port_is_open(port):
return False
# ── Helper: query fleet ingest API for latest device_status ──────
def _query_fleet_telemetry():
"""Fetch device status from the fleet ingest API.
Returns a dict keyed by device_id, or empty dict on error."""
try:
with urllib.request.urlopen(FLEET_API, timeout=5) as resp:
rows = json.loads(resp.read())
result = {}
for row in rows:
dev = row.get("device_id")
if not dev:
continue
wan = row.get("wan") or {}
if isinstance(wan, str):
wan = json.loads(wan) if wan else {}
starlink = row.get("starlink") or {}
if isinstance(starlink, str):
starlink = json.loads(starlink) if starlink else {}
state = row.get("state") or {}
if isinstance(state, str):
state = json.loads(state) if state else {}
result[dev] = {
"vendor": row.get("vendor"),
"model": row.get("model"),
"online": row.get("online") or False,
"ts": row.get("ts"),
"version": row.get("version"),
"uptime_s": row.get("uptime_s"),
"primary_member": row.get("primary_member"),
"active_wan": row.get("active_wan"),
"gps": {
"lat": row.get("gps_lat"),
"lon": row.get("gps_lon"),
"fix": row.get("gps_fix"),
},
"wan": wan,
"starlink": starlink,
"state": state,
}
return result
except Exception as e:
print(f"[hub] fleet telemetry query failed: {e}", file=sys.stderr)
return {}
class HubHandler(BaseHTTPRequestHandler):
"""Handle hub registration, fleet status, and dashboard requests."""
@@ -56,11 +105,16 @@ class HubHandler(BaseHTTPRequestHandler):
self.call_register_script(device_id)
return
# Fleet status — all registered devices
# Fleet status — basic tunnel-only (original endpoint)
if self.path == "/api/fleet":
self.serve_fleet_status()
return
# Fleet status — enriched with telemetry from fleet DB
if self.path == "/api/fleet-telemetry":
self.serve_fleet_telemetry()
return
self.send_json_error(404, "Not found")
def do_POST(self):
@@ -161,6 +215,73 @@ class HubHandler(BaseHTTPRequestHandler):
"devices": devices,
}))
def serve_fleet_telemetry(self):
"""Build enriched fleet status: tunnel status + telemetry from fleet DB."""
# Read port registry (tunnel info)
try:
with open(REGISTRY) as f:
registry = json.load(f)
except Exception:
registry = {}
# Query fleet Postgres for telemetry
fleet_data = _query_fleet_telemetry()
devices = []
for dev_id, entry in registry.items():
port = entry.get("tunnel_port", 0)
tunnel_up = _port_is_open(port) if port else False
# Enrich with fleet telemetry
telem = fleet_data.get(dev_id, {})
# Derive platform from vendor or device_id prefix
vendor = telem.get("vendor", "")
if vendor == "synology" or dev_id.startswith("x"):
platform = "synology"
elif vendor == "glinet" or dev_id.startswith(("B", "C")):
platform = "openwrt"
else:
platform = telem.get("vendor", "unknown")
# Format WAN summary for dashboard
wan_summary = _format_wan_summary(telem.get("wan", {}), telem.get("primary_member"))
devices.append({
"device_id": dev_id,
"tunnel_port": port,
"assigned": entry.get("assigned", ""),
"tunnel_up": tunnel_up,
"ssh_command": f"ssh -p {port} kitadmin@{HUB_IP}",
"dashboard_url": f"http://{HUB_IP}:{port}" if tunnel_up else None,
# Telemetry enrichment
"platform": platform,
"version": telem.get("version"),
"uptime_s": telem.get("uptime_s"),
"online_telemetry": telem.get("online", False),
"primary_member": telem.get("primary_member"),
"active_wan": telem.get("active_wan"),
"gps": telem.get("gps", {}),
"wan_summary": wan_summary,
"starlink": telem.get("starlink"),
"autonomous": telem.get("state", {}).get("autonomous"),
})
# Sort: tunnel_up first, then by device_id
devices.sort(key=lambda d: (not d["tunnel_up"], d["device_id"]))
# Summary stats
online = sum(1 for d in devices if d["tunnel_up"])
telemetry_online = sum(1 for d in devices if d.get("online_telemetry"))
self.send_json(200, json.dumps({
"hub": HUB_IP,
"device_count": len(devices),
"online": online,
"telemetry_online": telemetry_online,
"devices": devices,
}))
def send_json(self, status, body):
"""Send a JSON response."""
self.send_response(status)
@@ -177,6 +298,30 @@ class HubHandler(BaseHTTPRequestHandler):
print(f"[hub] {args[0]}", file=sys.stderr)
def _format_wan_summary(wan, primary_member):
"""Format WAN data into a concise summary for the dashboard."""
if not wan or not isinstance(wan, dict):
return []
summary = []
for member, metrics in wan.items():
if not isinstance(metrics, dict):
continue
entry = {
"member": member,
"active": member == primary_member,
"score": metrics.get("score"),
"latency_ms": metrics.get("latency_ms"),
"loss_pct": metrics.get("loss_pct"),
"rsrp_dbm": metrics.get("rsrp_dbm"),
"carrier": metrics.get("carrier"),
"technology": metrics.get("technology"),
}
summary.append(entry)
# Sort: active first, then by score descending
summary.sort(key=lambda x: (not x["active"], -(x["score"] or 0)))
return summary
def main():
port = int(sys.argv[1]) if len(sys.argv) > 1 else 8080
server = HTTPServer(("0.0.0.0", port), HubHandler)
+1 -1
View File
@@ -1,5 +1,5 @@
package="kit-connect"
version="0.1-0001"
version="0.5.0-0001"
description="Unified management connectivity for Pioneer bus fleet routers. Single SPK — deploys Tailscale + reverse SSH tunnel. Hub auto-assigns ports on install. One package, all routers. Compatible: RT2600ac, RT6600ax."
displayname="KIT Bus Router Connect"
maintainer="Keylink IT"
+2 -2
View File
@@ -210,6 +210,6 @@ while true; do
# Start tunnel (blocking — restarts on failure)
start_tunnel 2>/dev/null || true
log "daemon: both services running"
wait # Wait for any child to die, then restart
# start_tunnel is already blocking and self-retrying; no bare wait here — a bare
# wait blocks forever on the setsid'd tailscaled child and wedges the retry loop.
done
+1 -1
View File
@@ -19,7 +19,7 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
cd "$SCRIPT_DIR"
PKG_NAME="kit-connect"
VERSION="0.1-0001"
VERSION="0.5.0-0001"
OUTPUT="${PKG_NAME}-${VERSION}.spk"
BUILD_DIR="/tmp/kit-connect-build-$$"
Binary file not shown.
@@ -1,85 +1,79 @@
#!/bin/sh
# telemetry-synology.sh — Synology SRM telemetry payload builder for fleet hub.
# telemetry-synology.sh — Self-contained telemetry for Synology SRM routers.
#
# Reads aiwanbal state files from /tmp/aiwanbal/ and optionally queries the
# Eyeride Eyenet (OpenWrt ubus) for GPS position. Posts JSON to the fleet hub
# over Tailscale with disk-buffered retry on failure.
# Synology RT2600ac routers have NO built-in 5G modem or GPS. Cellular comes
# from an external Eyeride device (192.168.10.1:8080) which provides GPS and
# signal metrics via OpenWrt ubus JSON-RPC.
#
# Config (environment variables):
# TELEMETRY_HUB POST endpoint (default: http://10.88.0.1:8080/api/telemetry)
# TELEMETRY_BUFFER_DIR disk buffer for failed POSTs (default: /tmp/busrouter/telemetry-buf)
# AIWANBAL_STATE_DIR aiwanbal runtime state (default: /tmp/aiwanbal)
# EYERIDE_IP Eyeride OpenWrt IP for GPS (default: 192.168.10.1)
# EYERIDE_PORT Eyeride ubus port (default: 8080)
# EYERIDE_USER Eyeride SSH/ubus username (default: root)
# EYERIDE_PASSWORD Eyeride password (required for GPS)
# Smart WAN role detection: cellular can be on ANY port (WAN1 or WAN2).
# The script auto-detects based on:
# 1. aiwanbal/SmartAiBalancer modem_type fields (eyeride/peplink/zte = cellular)
# 2. Eyeride reachability at 192.168.10.1
# 3. Gateway + carrier pattern analysis
# This ensures correct mapping regardless of physical port assignment.
#
# Public API:
# eyeride_gps query GPS from Eyeride ubus; echo 'lat lon fix';
# exit 1 if unavailable
# telemetry_synology_collect emit JSON payload to stdout from aiwanbal state files
# telemetry_synology_flush retry all buffered payloads; return 1 if hub still down
# telemetry_synology_send collect + flush + post; buffer on failure
# Data sources (preferred order):
# 1. aiwanbal state files (/tmp/aiwanbal/) — richer scoring + failover decisions
# 2. Self-collected ping metrics (fallback when aiwanbal not installed)
#
# Cron mode: runs telemetry_synology_send when executed directly.
# GPS fallback chain:
# 1. Eyeride ubus JSON-RPC (gps / location service) → fix=2 or 3
# 2. ip-api.com geolocation by external IP → fix=1 (approximate)
#
# Config (/etc/busrouter/telemetry.conf, sourced as shell):
# TELEMETRY_HUB POST endpoint
# EYERIDE_IP/PORT/USER/PASSWORD Eyeride credentials
# WAN1_IFACE/WAN2_IFACE Physical interface names
LIB_DIR="${LIB_DIR:-/usr/lib/busrouter}"
# ── Configuration ──────────────────────────────────────────────────────────
: "${TELEMETRY_HUB:=http://10.88.0.1:8080/api/telemetry}"
: "${TELEMETRY_BUFFER_DIR:=/tmp/busrouter/telemetry-buf}"
: "${AIWANBAL_STATE_DIR:=/tmp/aiwanbal}"
: "${EYERIDE_IP:=192.168.10.1}"
: "${EYERIDE_PORT:=8080}"
: "${EYERIDE_USER:=root}"
: "${EYERIDE_USER:=benmashborn}"
: "${WAN1_IFACE:=eth0}"
: "${WAN2_IFACE:=eth2}"
: "${PING_TARGET:=8.8.8.8}"
: "${STATE_DIR:=/tmp/busrouter}"
: "${AIWANBAL_DIR:=/tmp/aiwanbal}"
[ -f /etc/busrouter/telemetry.conf ] && . /etc/busrouter/telemetry.conf
_TELEM_SEQ=0
# ── Helpers ─────────────────────────────────────────────────────────────────────
# Escape a string value for JSON (backslash and double-quote only).
# ── Helpers ────────────────────────────────────────────────────────────────
_json_str() { printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g'; }
# Read an aiwanbal state file, return empty string if missing.
_aiwanbal_val() {
local key="$1"
cat "${AIWANBAL_STATE_DIR}/${key}" 2>/dev/null || true
# Read a file, return empty string if missing.
_read_file() { cat "$1" 2>/dev/null || true; }
# Extract a numeric field from JSON.
_json_get_num() {
echo "$1" | sed -n "s/.*\"$2\": *\(-\?[0-9.]*\).*/\1/p" | head -1
}
_json_get_str() {
echo "$1" | sed -n "s/.*\"$2\": *\"\([^\"]*\)\".*/\1/p" | head -1
}
# ── Eyeride GPS (ubus JSON-RPC) ─────────────────────────────────────────────────
#
# The Eyeride Eyenet is an OpenWrt device at 192.168.10.1:8080 with ubus
# JSON-RPC. GPS data may be available through:
# 1. ubus call gps status (if gps service is running)
# 2. ubus call location status (alternative service name)
# 3. /tmp/gps/last_fix on the Eyeride (mounted via NFS/SMB or scraped)
#
# This function tries option 1 first, then falls back gracefully.
# Authenticate to Eyeride ubus, return session token on stdout.
# ── GPS: Eyeride ubus JSON-RPC ─────────────────────────────────────────────
_eyeride_login() {
local _pass="${EYERIDE_PASSWORD:-}"
_pass="${EYERIDE_PASSWORD:-}"
[ -n "$_pass" ] || return 1
local _login_body _login_resp _session
_login_body="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"call\",\"params\":[\"00000000000000000000000000000000\",\"session\",\"login\",{\"username\":\"${EYERIDE_USER}\",\"password\":\"${_pass}\"}]}"
_login_resp=$(curl -s --connect-timeout 5 -X POST \
-H 'Content-Type: application/json' \
-d "$_login_body" \
"http://${EYERIDE_IP}:${EYERIDE_PORT}/ubus" 2>/dev/null) || return 1
# Check for auth failure: ubus returns [6] for permission denied
if [ -z "$_login_resp" ] || echo "$_login_resp" | grep -q '"result":\[6\]'; then
return 1
fi
_session=$(echo "$_login_resp" | sed 's/.*"ubus_rpc_session":"\([^"]*\)".*/\1/')
[ -n "$_session" ] && [ "$_session" != "$_login_resp" ] || return 1
printf '%s\n' "$_session"
}
# Call a ubus method on the Eyeride with an active session token.
# Usage: _eyeride_ubus_call <session> <service> <method> [params_json]
_eyeride_ubus_call() {
local _session="$1" _svc="$2" _method="$3" _params="${4:-{}}"
local _body
_session="$1" _svc="$2" _method="$3" _params="${4:-{}}"
_body="{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"call\",\"params\":[\"${_session}\",\"${_svc}\",\"${_method}\",${_params}]}"
curl -s --connect-timeout 5 -X POST \
-H 'Content-Type: application/json' \
@@ -87,251 +81,516 @@ _eyeride_ubus_call() {
"http://${EYERIDE_IP}:${EYERIDE_PORT}/ubus" 2>/dev/null
}
# Query GPS position from Eyeride.
# Emits 'lat lon fix_quality' on success; exit 1 on failure or no fix.
eyeride_gps() {
local _session _resp _lat _lon _fix
_session=$(_eyeride_login) || return 1
# Try 'gps' service first (common OpenWrt gpsd ubus binding)
_resp=$(_eyeride_ubus_call "$_session" "gps" "status") 2>/dev/null
if [ -z "$_resp" ] || echo "$_resp" | grep -q '"result":\[2\]'; then
# gps service not found — try 'location' service
_resp=$(_eyeride_ubus_call "$_session" "location" "status") 2>/dev/null
fi
# Try to extract lat/lon from ubus response JSON
_lat=$(echo "$_resp" | sed 's/.*"latitude":\([0-9.-]*\).*/\1/' 2>/dev/null)
_lon=$(echo "$_resp" | sed 's/.*"longitude":\([0-9.-]*\).*/\1/' 2>/dev/null)
# Also try snake_case variants
[ "$_lat" = "$_resp" ] && _lat=$(echo "$_resp" | sed 's/.*"lat":\([0-9.-]*\).*/\1/' 2>/dev/null)
[ "$_lon" = "$_resp" ] && _lon=$(echo "$_resp" | sed 's/.*"lon":\([0-9.-]*\).*/\1/' 2>/dev/null)
# Alt spelling
[ "$_lat" = "$_resp" ] && _lat=$(echo "$_resp" | sed 's/.*"lat_deg":\([0-9.-]*\).*/\1/' 2>/dev/null)
[ "$_lon" = "$_resp" ] && _lon=$(echo "$_resp" | sed 's/.*"lon_deg":\([0-9.-]*\).*/\1/' 2>/dev/null)
# Sanitize: if sed didn't match, the field equals the full response
[ "$_lat" = "$_resp" ] && _lat=""
[ "$_lon" = "$_resp" ] && _lon=""
_lat=$(_json_get_num "$_resp" "latitude")
_lon=$(_json_get_num "$_resp" "longitude")
[ -z "$_lat" ] && _lat=$(_json_get_num "$_resp" "lat")
[ -z "$_lon" ] && _lon=$(_json_get_num "$_resp" "lon")
[ -z "$_lat" ] && _lat=$(_json_get_num "$_resp" "lat_deg")
[ -z "$_lon" ] && _lon=$(_json_get_num "$_resp" "lon_deg")
if [ -n "$_lat" ] && [ -n "$_lon" ]; then
_fix=2 # assume 2D fix if we got coordinates
[ -n "$_fix" ] || _fix=2
_fix=2
logger -t busrouter -p daemon.debug "eyeride_gps: ${_lat} ${_lon} fix=${_fix}"
printf '%s %s %s\n' "$_lat" "$_lon" "$_fix"
return 0
fi
logger -t busrouter "eyeride_gps: no fix from Eyeride"
return 1
}
# ── Telemetry Collection ─────────────────────────────────────────────────────────
#
# State files read from AIWANBAL_STATE_DIR (written by aiwanbal daemon each cycle):
# wan1_score_avg, wan2_score_avg integer 0100
# wan1_latency, wan2_latency ms
# wan1_throughput, wan2_throughput B/s (converted to Mbps below)
# wan1_upload, wan2_upload B/s
# wan1_jitter, wan2_jitter ms
# wan1_packet_loss, wan2_packet_loss %
# wan1_rsrp, wan2_rsrp dBm (cellular signal)
# wan1_rsrq, wan2_rsrq dB
# wan1_sinr, wan2_sinr dB
# wan1_technology, wan2_technology e.g. "5G-NR", "LTE"
# wan1_modem_type, wan2_modem_type e.g. "eyeride", "peplink", "none"
# wan0_mode "active" or "failover"
# wan0_starlink_quality quality score (if Starlink connected)
# wan0_starlink_latency ms
# wan0_starlink_obstruction bool-ish
# wan0_starlink_outage bool-ish
# Read one WAN's metrics and emit a JSON object fragment.
# Usage: _synology_wan_json <wan_num>
_synology_wan_json() {
local n="$1"
local score lat dl ul jitter loss rsrp rsrq sinr tech carrier modem
score=$(_aiwanbal_val "wan${n}_score_avg")
lat=$(_aiwanbal_val "wan${n}_latency")
local throughput_bytes
throughput_bytes=$(_aiwanbal_val "wan${n}_throughput")
dl=""
[ -n "$throughput_bytes" ] && dl=$(awk "BEGIN { printf \"%.1f\", ${throughput_bytes} / 125000 }" 2>/dev/null)
[ -z "$dl" ] && dl="0"
local upload_bytes
upload_bytes=$(_aiwanbal_val "wan${n}_upload")
ul=""
[ -n "$upload_bytes" ] && ul=$(awk "BEGIN { printf \"%.1f\", ${upload_bytes} / 125000 }" 2>/dev/null)
[ -z "$ul" ] && ul="0"
jitter=$(_aiwanbal_val "wan${n}_jitter")
loss=$(_aiwanbal_val "wan${n}_packet_loss")
rsrp=$(_aiwanbal_val "wan${n}_rsrp")
rsrq=$(_aiwanbal_val "wan${n}_rsrq")
sinr=$(_aiwanbal_val "wan${n}_sinr")
tech=$(_aiwanbal_val "wan${n}_technology")
modem=$(_aiwanbal_val "wan${n}_modem_type")
# Carrier is not tracked separately by aiwanbal — derive from modem type
carrier=""
case "$modem" in
eyeride) carrier="Eyeride" ;;
peplink) carrier="Peplink" ;;
zte) carrier="ZTE" ;;
mofi) carrier="Mofi" ;;
none|generic|"") carrier="" ;;
*) carrier="$modem" ;;
esac
local esc_tech esc_carrier
esc_tech=$(_json_str "${tech:-unknown}")
esc_carrier=$(_json_str "${carrier:-unknown}")
printf '{"score":%s,"latency_ms":%s,"dl_mbps":%s,"ul_mbps":%s,"jitter_ms":%s,"loss_pct":%s,"rsrp_dbm":%s,"rsrq_db":%s,"sinr_db":%s,"technology":"%s","carrier":"%s"}' \
"${score:-0}" \
"${lat:-null}" \
"${dl:-0}" \
"${ul:-0}" \
"${jitter:-null}" \
"${loss:-null}" \
"${rsrp:-null}" \
"${rsrq:-null}" \
"${sinr:-null}" \
"${esc_tech}" \
"${esc_carrier}"
ipgeo_gps() {
_resp=$(curl -s --connect-timeout 5 "http://ip-api.com/json/" 2>/dev/null | tr -d '\n\r') || return 1
[ -n "$_resp" ] || return 1
_lat=$(_json_get_num "$_resp" "lat")
_lon=$(_json_get_num "$_resp" "lon")
if [ -n "$_lat" ] && [ -n "$_lon" ]; then
_fix=1
logger -t busrouter -p daemon.debug "ipgeo_gps: ${_lat} ${_lon} fix=${_fix} (IP geolocation)"
printf '%s %s %s\n' "$_lat" "$_lon" "$_fix"
return 0
fi
return 1
}
# Build the full telemetry JSON payload from aiwanbal state files.
# Emits JSON on stdout; never fails — uses defaults for all missing fields.
telemetry_synology_collect() {
local now device_id uptime_s version
gps_fix() {
if eyeride_gps 2>/dev/null; then return 0; fi
ipgeo_gps
}
# ── Smart WAN Role Detection ───────────────────────────────────────────────
# Detects which physical interface is cellular vs landline/Starlink.
# Uses multiple signals in priority order:
# 1. aiwanbal modem_type field (eyeride/peplink/zte/mofi/quectel = cellular)
# 2. Interface routes to Eyeride at 192.168.10.1
# 3. Gateway/carrier analysis
# Check if a modem_type string indicates a cellular modem.
_is_cellular_modem() {
case "$1" in
eyeride|peplink|zte|mofi|quectel|telit|sierra|fibocom) return 0 ;;
none|generic|"") return 1 ;;
*) return 1 ;;
esac
}
# Check if an interface appears to be cellular by attempting to reach the Eyeride.
_iface_reaches_eyeride() {
_iface="$1"
# Check if this interface has an IP in the Eyeride network range
_ip=$(ip addr show "$_iface" 2>/dev/null | grep 'inet ' | awk '{print $2}' | cut -d/ -f1)
if echo "$_ip" | grep -q '^192\.168\.10\.'; then
return 0
fi
# Check if a route to the Eyeride exists via this interface
if ip route get "$EYERIDE_IP" 2>/dev/null | grep -q "dev $_iface"; then
return 0
fi
return 1
}
# Returns: "cellular_iface ethernet_iface"
_detect_wan_roles() {
_cell="" _eth=""
# 1. Try aiwanbal modem_type first (most reliable)
if [ -d "$AIWANBAL_DIR" ]; then
_mod1=$(_read_file "${AIWANBAL_DIR}/wan1_modem_type")
_mod2=$(_read_file "${AIWANBAL_DIR}/wan2_modem_type")
if _is_cellular_modem "$_mod1" && ! _is_cellular_modem "$_mod2"; then
echo "$WAN1_IFACE $WAN2_IFACE"; return 0
elif _is_cellular_modem "$_mod2" && ! _is_cellular_modem "$_mod1"; then
echo "$WAN2_IFACE $WAN1_IFACE"; return 0
elif _is_cellular_modem "$_mod1" && _is_cellular_modem "$_mod2"; then
# Both cellular — default: WAN2 is cellular (Eyeride typical port)
echo "$WAN2_IFACE $WAN1_IFACE"; return 0
fi
fi
# 2. Check Eyeride reachability
if _iface_reaches_eyeride "$WAN2_IFACE"; then
echo "$WAN2_IFACE $WAN1_IFACE"; return 0
elif _iface_reaches_eyeride "$WAN1_IFACE"; then
echo "$WAN1_IFACE $WAN2_IFACE"; return 0
fi
# 3. Check gateway patterns for known carriers
for _iface in "$WAN2_IFACE" "$WAN1_IFACE"; do
_gw=$(ip route show default 2>/dev/null | grep "$_iface" | awk '{print $3}' | head -1)
# Starlink: typically 192.168.100.1 or 100.64.x.x (CGNAT)
if echo "$_gw" | grep -qE '^192\.168\.100\.|^100\.(64|96|112)\.'; then
_cell="" # Starlink is not cellular
fi
# Eyeride: always 192.168.10.1
if [ "$_gw" = "192.168.10.1" ]; then
_cell="$_iface"
fi
done
if [ -n "$_cell" ]; then
for _iface in "$WAN2_IFACE" "$WAN1_IFACE"; do
[ "$_iface" != "$_cell" ] && _eth="$_iface"
done
echo "${_cell} ${_eth}"; return 0
fi
# 4. Default: WAN2 is cellular (most common Synology bus router layout)
echo "$WAN2_IFACE $WAN1_IFACE"
}
# ── WAN Metric Collection ──────────────────────────────────────────────────
# Check if aiwanbal state is available and fresh (< 120s old).
_aiwanbal_available() {
[ -d "$AIWANBAL_DIR" ] || return 1
_mtime=$(stat -c %Y "${AIWANBAL_DIR}/wan1_score_avg" 2>/dev/null || stat -f %m "${AIWANBAL_DIR}/wan1_score_avg" 2>/dev/null)
_mtime=$(printf '%s' "$_mtime" | tr -cd '0-9')
[ -n "$_mtime" ] || return 1
_now=$(date +%s)
_age=$(( _now - _mtime ))
[ "$_age" -lt 120 ] && return 0
return 1
}
# Ping-based fallback metrics (used when aiwanbal not available).
_ping_iface() {
_iface="$1"
_out=$(sudo ping -I "$_iface" -c 3 -W 3 "$PING_TARGET" 2>/dev/null)
_rc=$?
if [ $_rc -ge 2 ]; then
_out=$(ping -I "$_iface" -c 3 -W 3 "$PING_TARGET" 2>/dev/null)
fi
if [ -z "$_out" ]; then
echo "null null null"; return
fi
if echo "$_out" | grep -qE "permission denied|not permitted|unknown host|Network is unreachable"; then
echo "null null null"; return
fi
# Extract avg rtt
_avg=$(echo "$_out" | tail -1 | sed -n 's/.*=\ [0-9.]*\/\([0-9.]*\)\/.*/\1/p')
[ -z "$_avg" ] && _avg=$(echo "$_out" | tail -1 | sed -n 's/.*avg\/max.*=\ \([0-9.]*\)\/.*/\1/p')
[ -z "$_avg" ] && _avg="null"
# Extract jitter (mdev from standard ping: rtt min/avg/max/mdev = ...)
_jitter=$(echo "$_out" | tail -1 | sed -n 's/.*=\ [0-9.]*\/[0-9.]*\/[0-9.]*\/\([0-9.]*\).*/\1/p')
[ -z "$_jitter" ] && _jitter="null"
# Extract loss
_loss=$(echo "$_out" | sed -n 's/.* \([0-9]*\)% packet loss.*/\1/p')
[ -z "$_loss" ] && _loss="null"
# If ICMP to internet gives 100% loss, ping the local gateway —
# Eyeride NAT drops internet ICMP but the gateway (192.168.10.1) is reachable locally.
if [ "$_loss" = "100" ]; then
_gw=$(ip route show 2>/dev/null | grep "dev $_iface" | grep -v "^default" | awk '/via/{print $3; exit} /kernel/{print ""}' | head -1)
[ -z "$_gw" ] && _gw=$(ip route show default 2>/dev/null | grep "dev $_iface" | awk '{print $3}' | head -1)
if [ -n "$_gw" ]; then
_gw_out=$(ping -c 3 -W 2 "$_gw" 2>/dev/null)
_gw_loss=$(echo "$_gw_out" | sed -n 's/.* \([0-9]*\)% packet loss.*/\1/p')
if [ -n "$_gw_loss" ] && [ "$_gw_loss" != "100" ]; then
_loss="$_gw_loss"
_avg="null" # gateway RTT is LAN latency, not meaningful as WAN metric
fi
fi
fi
printf '%s %s %s\n' "${_avg}" "${_jitter}" "${_loss}" | tr -d '\n\r'
}
# Compute score from latency + loss (0-100, higher = better).
_compute_score() {
_lat="$1" _loss="$2"
if [ "$_lat" = "null" ] || [ "$_loss" = "null" ]; then
echo "0"; return
fi
_score=$(awk "BEGIN { s = 100 - ${_lat}/2 - ${_loss}*5; if (s < 0) s = 0; if (s > 100) s = 100; printf \"%d\", int(s) }" 2>/dev/null)
echo "${_score:-0}"
}
# Collect metrics for one WAN interface.
# Uses aiwanbal state if available, falls back to self-collected ping.
# Usage: _collect_wan <iface> <wan_num> → JSON object fragment
_collect_wan() {
_iface="$1" _num="$2"
_score="0" _lat="null" _dl="null" _ul="null" _jitter="null" _loss="null"
if _aiwanbal_available; then
# Use aiwanbal state (preferred — richer metrics)
_score=$(_read_file "${AIWANBAL_DIR}/wan${_num}_score_avg")
[ -z "$_score" ] && _score="0"
_lat=$(_read_file "${AIWANBAL_DIR}/wan${_num}_latency")
[ -z "$_lat" ] && _lat="null"
_jitter=$(_read_file "${AIWANBAL_DIR}/wan${_num}_jitter")
[ -z "$_jitter" ] && _jitter="null"
_loss=$(_read_file "${AIWANBAL_DIR}/wan${_num}_packet_loss")
[ -z "$_loss" ] && _loss="null"
# Convert throughput from B/s to Mbps
_tp=$(_read_file "${AIWANBAL_DIR}/wan${_num}_throughput")
if [ -n "$_tp" ] && [ "$_tp" != "0" ]; then
_dl=$(awk "BEGIN { printf \"%.1f\", ${_tp} / 125000 }" 2>/dev/null)
fi
[ -z "$_dl" ] && _dl="0.0"
_up=$(_read_file "${AIWANBAL_DIR}/wan${_num}_upload")
if [ -n "$_up" ] && [ "$_up" != "0" ]; then
_ul=$(awk "BEGIN { printf \"%.1f\", ${_up} / 125000 }" 2>/dev/null)
fi
[ -z "$_ul" ] && _ul="0.0"
else
# Fallback: self-collected ping metrics
_ping_result=$(_ping_iface "$_iface")
_lat=$(echo "$_ping_result" | awk '{print $1}')
_jitter=$(echo "$_ping_result" | awk '{print $2}')
_loss=$(echo "$_ping_result" | awk '{print $3}')
_score=$(_compute_score "$_lat" "$_loss")
# Use cached speedtest result if available and fresh
_st=$(_cached_speedtest "$_iface" 2>/dev/null)
if [ -n "$_st" ]; then
_dl=$(echo "$_st" | awk '{print $1}')
_ul=$(echo "$_st" | awk '{print $2}')
fi
[ -n "$_dl" ] || _dl="0.0"
[ -n "$_ul" ] || _ul="0.0"
fi
printf '{"score":%s,"latency_ms":%s,"dl_mbps":%s,"ul_mbps":%s,"jitter_ms":%s,"loss_pct":%s}' \
"${_score}" "${_lat}" "${_dl}" "${_ul}" "${_jitter}" "${_loss}"
}
# ── Carrier Detection ──────────────────────────────────────────────────────
# Derive carrier name for an interface.
_detect_carrier() {
_iface="$1"
# Check if this is the Eyeride cellular path
if _iface_reaches_eyeride "$_iface"; then
_sig=$(_eyeride_signal 2>/dev/null)
_sig_carrier=$(echo "$_sig" | awk '{print $5}')
if [ -n "$_sig_carrier" ] && [ "$_sig_carrier" != "null" ] && [ "$_sig_carrier" != "Eyeride" ]; then
echo "$_sig_carrier"; return
fi
echo "Eyeride"; return
fi
# Try aiwanbal modem_type
_num=""
for _n in 1 2; do
_mod=$(_read_file "${AIWANBAL_DIR}/wan${_n}_modem_type" 2>/dev/null)
_active=$(_read_file "${AIWANBAL_DIR}/wan${_n}_state" 2>/dev/null)
if [ "$_mod" != "eyeride" ] && [ "$_mod" != "none" ] && [ "$_mod" != "generic" ] && [ -n "$_mod" ] && [ "$_active" = "up" ]; then
echo "$_mod"; return
fi
done
# Check gateway for known ISP patterns
_gw=$(ip route show default 2>/dev/null | grep "$_iface" | awk '{print $3}' | head -1)
if [ -n "$_gw" ]; then
if echo "$_gw" | grep -qE '^192\.168\.[01]\.'; then
echo "AT&T"; return
fi
if echo "$_gw" | grep -qE '^192\.168\.100\.|^100\.(64|96)\.'; then
echo "Starlink"; return
fi
if echo "$_gw" | grep -qE '^10\.|^172\.(1[6-9]|2[0-9]|3[0-1])\.'; then
echo "Business"; return
fi
fi
echo ""
}
# Add carrier and identity to a WAN JSON object.
_add_wan_identity() {
_json="$1" _iface="$2"
_carrier=$(_detect_carrier "$_iface")
_prefix=$(printf '%s' "$_json" | sed 's/}$//')
if [ -n "$_carrier" ]; then
_esc_carrier=$(_json_str "${_carrier}")
printf '%s,"carrier":"%s"}' "$_prefix" "$_esc_carrier"
else
printf '%s}' "$_prefix"
fi
}
# ── Eyeride Signal ─────────────────────────────────────────────────────────
_eyeride_signal() {
_session=$(_eyeride_login) 2>/dev/null || return 1
_resp=$(_eyeride_ubus_call "$_session" "network" "status") 2>/dev/null
if [ -z "$_resp" ] || echo "$_resp" | grep -q '"result":\[2\]'; then
_resp=$(_eyeride_ubus_call "$_session" "wwan" "status") 2>/dev/null
fi
_rsrp=$(_json_get_num "$_resp" "rsrp")
_rsrq=$(_json_get_num "$_resp" "rsrq")
_sinr=$(_json_get_num "$_resp" "sinr")
_tech=$(_json_get_str "$_resp" "technology")
_carrier=$(_json_get_str "$_resp" "operator")
[ -z "$_rsrp" ] && _rsrp=$(_json_get_num "$_resp" "signal")
[ -z "$_rsrp" ] && _rsrp=$(_json_get_num "$_resp" "rssi")
printf '%s %s %s %s %s\n' "${_rsrp:-null}" "${_rsrq:-null}" "${_sinr:-null}" "${_tech:-unknown}" "${_carrier:-Eyeride}"
}
# ── Speedtest ──────────────────────────────────────────────────────────────
# Run a speedtest bound to a specific interface. Caches result per-interface.
run_speedtest() {
_iface="$1"
_result_file="${STATE_DIR}/speedtest-result-${_iface}"
# Try speedtest-cli first
if which speedtest-cli >/dev/null 2>&1; then
_out=$(speedtest-cli --interface "$_iface" --json 2>/dev/null)
if [ -n "$_out" ]; then
_dl=$(_json_get_num "$_out" "download")
_ul=$(_json_get_num "$_out" "upload")
if [ -n "$_dl" ] && [ -n "$_ul" ]; then
_dl_mbps=$(awk "BEGIN { printf \"%.1f\", ${_dl} / 125000 }" 2>/dev/null)
_ul_mbps=$(awk "BEGIN { printf \"%.1f\", ${_ul} / 125000 }" 2>/dev/null)
printf '%s %s\n' "${_dl_mbps}" "${_ul_mbps}" > "$_result_file"
echo "${_dl_mbps} ${_ul_mbps}"
return 0
fi
fi
fi
# Fallback: curl a known file and measure throughput
_tmp="/tmp/speedtest-$$"
_start=$(date +%s)
if curl -s --interface "$_iface" --max-time 15 -o "$_tmp" http://speedtest.tele2.net/10MB.zip 2>/dev/null; then
_end=$(date +%s)
_bytes=$(wc -c < "$_tmp" 2>/dev/null)
rm -f "$_tmp"
if [ "$_end" -gt "$_start" ] && [ -n "$_bytes" ]; then
_duration=$(( _end - _start ))
[ "$_duration" -lt 1 ] && _duration=1
_dl_mbps=$(awk "BEGIN { printf \"%.1f\", ${_bytes} * 8 / ${_duration} / 1000000 }" 2>/dev/null)
printf '%s 0\n' "${_dl_mbps:-0}" > "$_result_file"
echo "${_dl_mbps:-0} 0"
return 0
fi
fi
rm -f "$_tmp"
echo "0 0"
}
# Read cached speedtest result for an interface if < 35 minutes old.
_cached_speedtest() {
_iface="$1"
_f="${STATE_DIR}/speedtest-result-${_iface}"
[ -f "$_f" ] || return 1
_mtime=$(stat -c %Y "$_f" 2>/dev/null || stat -f %m "$_f" 2>/dev/null)
_mtime=$(printf '%s' "$_mtime" | tr -cd '0-9')
[ -n "$_mtime" ] || return 1
_st_now=$(date +%s)
_age=$(( _st_now - _mtime ))
[ "$_age" -lt 2100 ] || return 1
cat "$_f"
}
# ── Telemetry Collection ───────────────────────────────────────────────────
telemetry_collect() {
now=$(date +%s)
device_id=$(cat /etc/busrouter/device-id 2>/dev/null || hostname)
uptime_s=$(awk '{printf "%d", $1}' /proc/uptime 2>/dev/null)
version=$(cat /etc/busrouter/version 2>/dev/null || echo "dev")
# GPS
local gps_lat gps_lon gps_fix
if gps_out=$(eyeride_gps 2>/dev/null); then
read -r gps_lat gps_lon gps_fix <<-EOF
$gps_out
EOF
gps_lat="null" gps_lon="null" gps_fix="null"
if gps_out=$(gps_fix 2>/dev/null); then
gps_lat=$(echo "$gps_out" | awk '{print $1}')
gps_lon=$(echo "$gps_out" | awk '{print $2}')
gps_fix=$(echo "$gps_out" | awk '{print $3}')
fi
# WAN 1 + WAN 2
local wan1_json wan2_json
wan1_json=$(_synology_wan_json 1)
wan2_json=$(_synology_wan_json 2)
# Smart WAN role detection
if _roles=$(_detect_wan_roles 2>/dev/null); then
_cell_iface=$(echo "$_roles" | awk '{print $1}')
_eth_iface=$(echo "$_roles" | awk '{print $2}')
else
_cell_iface="$WAN2_IFACE"
_eth_iface="$WAN1_IFACE"
fi
# Active WAN — which one is currently carrying traffic
local mode sel_primary
mode=$(_aiwanbal_val "wan0_mode")
case "$mode" in
failover)
# In failover, primary is the active one; check which is up
local wan1_state wan2_state
wan1_state=$(_aiwanbal_val "wan1_state")
wan2_state=$(_aiwanbal_val "wan2_state")
if [ "$wan2_state" = "up" ]; then
sel_primary="wan2"
else
sel_primary="wan1"
# Map physical interfaces to aiwanbal wan numbers
_cell_num="2" _eth_num="1"
if [ "$_cell_iface" = "$WAN1_IFACE" ]; then
_cell_num="1"
fi
if [ "$_eth_iface" = "$WAN2_IFACE" ]; then
_eth_num="2"
fi
# Collect WAN metrics
_modem_json=$(_collect_wan "$_cell_iface" "$_cell_num")
_ether_json=$(_collect_wan "$_eth_iface" "$_eth_num")
# Add carrier identity
_modem_json=$(_add_wan_identity "$_modem_json" "$_cell_iface")
_ether_json=$(_add_wan_identity "$_ether_json" "$_eth_iface")
# Enrich cellular WAN with Eyeride signal data
eyeride_sig=$(_eyeride_signal 2>/dev/null)
_rsrp=$(echo "$eyeride_sig" | awk '{print $1}')
_rsrq=$(echo "$eyeride_sig" | awk '{print $2}')
_sinr=$(echo "$eyeride_sig" | awk '{print $3}')
_tech=$(echo "$eyeride_sig" | awk '{print $4}')
_modem_json=$(printf '%s' "$_modem_json" | sed 's/}$//')
if [ "$_rsrp" != "null" ] && [ -n "$_rsrp" ]; then
_esc_tech=$(_json_str "${_tech:-unknown}")
_modem_json="${_modem_json},\"rsrp_dbm\":${_rsrp},\"rsrq_db\":${_rsrq},\"sinr_db\":${_sinr},\"technology\":\"${_esc_tech}\"}"
else
_modem_json="${_modem_json}}"
fi
# Active WAN determination
sel_primary="wan"
default_iface=$(ip route show default 2>/dev/null | awk '{print $5}' | head -1)
if [ -d "$AIWANBAL_DIR" ]; then
# Prefer aiwanbal mode for failover detection
_mode=$(_read_file "${AIWANBAL_DIR}/wan0_mode")
if [ "$_mode" = "failover" ]; then
_active=$(_read_file "${AIWANBAL_DIR}/wan2_state")
if [ "$_active" = "up" ]; then
sel_primary="modem_0001"
fi
;;
*)
# Load-balance mode: both are active; wan1 is primary
sel_primary="wan1"
;;
esac
fi
fi
# Fallback: use default route interface
if [ "$default_iface" = "$_cell_iface" ]; then
sel_primary="modem_0001"
fi
# Starlink (may be attached to one of the WANs)
local sl_lat sl_dl sl_obs sl_outage
sl_lat=$(_aiwanbal_val "wan0_starlink_latency")
sl_dl=$(_aiwanbal_val "wan0_starlink_quality")
sl_obs=$(_aiwanbal_val "wan0_starlink_obstruction")
sl_outage=$(_aiwanbal_val "wan0_starlink_outage")
# Starlink detection
sl_lat="null" sl_dl="null" sl_obs="0" sl_outage="0"
if ip link show 2>/dev/null | grep -qi starlink; then
sl_iface=$(ip link show 2>/dev/null | grep -i starlink | head -1 | awk -F": " '{print $2}' | awk '{print $1}')
if [ -n "$sl_iface" ]; then
sl_ping=$(_ping_iface "$sl_iface")
sl_lat=$(echo "$sl_ping" | awk '{print $1}')
fi
fi
# Convert Starlink quality to bps (quality is a score 0100, not bps)
# Store quality as-is; the hub can interpret it
local sl_quality
sl_quality="${sl_dl:-0}"
# State
state='{"autonomous":true,"qos":{},"wanhealth":{}}'
local esc_id esc_ver esc_sel
esc_id=$(_json_str "$device_id")
esc_ver=$(_json_str "$version")
esc_sel=$(_json_str "$sel_primary")
printf '{"device_id":"%s","timestamp":%d,"uptime":%d,"version":"%s","platform":"synology","sel_primary":"%s","gps":{"lat":%s,"lon":%s,"fix":%s},"wan":{"wan1":%s,"wan2":%s},"starlink":{"quality":%s,"latency_ms":%s,"obstructed":%s,"outage":%s}}\n' \
"${esc_id}" "${now:-0}" "${uptime_s:-0}" "${esc_ver}" "${esc_sel}" \
"${gps_lat:-null}" "${gps_lon:-null}" "${gps_fix:-null}" \
"${wan1_json}" "${wan2_json}" \
"${sl_quality:-0}" "${sl_lat:-null}" "${sl_obs:-0}" "${sl_outage:-0}"
printf '{"device_id":"%s","timestamp":%d,"uptime":%d,"version":"%s","platform":"synology","sel_primary":"%s","active_wan":"%s","gps":{"lat":%s,"lon":%s,"fix":%s},"wan":{"modem_0001":%s,"wan":%s},"sim_slot":null,"starlink":{"latency_ms":%s,"dl_bps":%s,"obstructed":%s,"outage":%s},"state":%s}\n' \
"${esc_id}" "${now:-0}" "${uptime_s:-0}" "${esc_ver}" "${esc_sel}" "${esc_sel}" \
"${gps_lat}" "${gps_lon}" "${gps_fix}" \
"${_modem_json}" "${_ether_json}" \
"${sl_lat}" "${sl_dl}" "${sl_obs}" "${sl_outage}" \
"${state}"
}
# ── Hub POST + Buffering ─────────────────────────────────────────────────────────
# POST a JSON file to the hub. Returns 0 on HTTP 2xx, 1 on any failure.
# ── Hub POST + Buffering ───────────────────────────────────────────────────
_telemetry_try_post() {
local file="$1"
local code
file="$1"
code=$(curl -sf -X POST -H "Content-Type: application/json" \
--data-binary @"$file" -o /dev/null -w "%{http_code}" \
--max-time 10 "$TELEMETRY_HUB" 2>/dev/null)
case "$code" in 2*) return 0 ;; *) return 1 ;; esac
}
# Retry all buffered payloads in timestamp order.
# Stops on first failure to avoid hammering a down hub.
telemetry_synology_flush() {
telemetry_flush() {
[ -d "$TELEMETRY_BUFFER_DIR" ] || return 0
local f
for f in "$TELEMETRY_BUFFER_DIR"/*.json; do
[ -f "$f" ] || continue
if _telemetry_try_post "$f"; then
rm -f "$f"
logger -t busrouter -p daemon.debug "telemetry_synology_flush: sent $(basename "$f")"
logger -t busrouter -p daemon.debug "telemetry_flush: sent $(basename "$f")"
else
logger -t busrouter "telemetry_synology_flush: hub unreachable, stopping"
logger -t busrouter "telemetry_flush: hub unreachable, stopping"
return 1
fi
done
}
# Collect metrics, flush old buffer, POST current payload.
# On POST failure: saves payload to TELEMETRY_BUFFER_DIR.
# Never blocks the caller on hub downtime.
telemetry_synology_send() {
local payload
payload=$(telemetry_synology_collect)
telemetry_synology_flush 2>/dev/null || true
mkdir -p "$TELEMETRY_BUFFER_DIR"
telemetry_send() {
mkdir -p "$STATE_DIR" 2>/dev/null
payload=$(telemetry_collect)
telemetry_flush 2>/dev/null || true
mkdir -p "$TELEMETRY_BUFFER_DIR" 2>/dev/null
_TELEM_SEQ=$(( _TELEM_SEQ + 1 ))
local tmpfile="${TELEMETRY_BUFFER_DIR}/$(date +%s)_$$_${_TELEM_SEQ}.json"
# Cap buffer to 60 files to protect tmpfs
local buf_count
tmpfile="${TELEMETRY_BUFFER_DIR}/$(date +%s)_$$_${_TELEM_SEQ}.json"
buf_count=$(ls "$TELEMETRY_BUFFER_DIR"/*.json 2>/dev/null | wc -l)
if [ "${buf_count:-0}" -ge 60 ]; then
local oldest
oldest=$(ls "$TELEMETRY_BUFFER_DIR"/*.json 2>/dev/null | sort | head -n 1)
[ -f "$oldest" ] && rm -f "$oldest"
fi
printf '%s\n' "$payload" > "$tmpfile"
if _telemetry_try_post "$tmpfile"; then
rm -f "$tmpfile"
logger -t busrouter -p daemon.debug "telemetry_synology_send: posted ok"
logger -t busrouter -p daemon.debug "telemetry_send: posted ok"
else
logger -t busrouter "telemetry_synology_send: hub unreachable, buffered $(basename "$tmpfile")"
logger -t busrouter "telemetry_send: hub unreachable, buffered $(basename "$tmpfile")"
fi
}
# ── Entry Point ──────────────────────────────────────────────────────────────────
# When executed directly (e.g. from cron), run a single telemetry cycle.
# ── Entry Point ────────────────────────────────────────────────────────────
case "${0##*/}" in
telemetry-synology.sh|telemetry-synology)
telemetry_synology_send
# Run speedtest if requested via a trigger file (touch /tmp/busrouter/speedtest-request)
if [ -f "${STATE_DIR}/speedtest-request" ]; then
_iface=$(cat "${STATE_DIR}/speedtest-request" 2>/dev/null)
[ -z "$_iface" ] && _iface="$WAN1_IFACE"
run_speedtest "$_iface" > "${STATE_DIR}/speedtest-result"
rm -f "${STATE_DIR}/speedtest-request"
fi
telemetry_send
;;
esac
+21
View File
@@ -0,0 +1,21 @@
#!/bin/sh
# Busrouter telemetry — BusyBox-compatible. Posts device state to fleet hub.
HUB="http://167.172.237.162:8080/api/telemetry"
ID=$(cat /etc/busrouter/device-id 2>/dev/null || hostname)
TS=$(date +%s)
UP=$(awk '{printf "%d", $1}' /proc/uptime 2>/dev/null)
# BusyBox grep doesn't have -P; use sed instead
WAN1=$(ip -4 addr show eth0 2>/dev/null | sed -n 's/.*inet \([0-9.]*\).*/\1/p' | head -1)
WAN2=$(ip -4 addr show eth2 2>/dev/null | sed -n 's/.*inet \([0-9.]*\).*/\1/p' | head -1)
GW=$(ip route show default 2>/dev/null | sed -n 's/.*via \([0-9.]*\).*/\1/p' | head -1)
# Build JSON with proper escaping for BusyBox
cat >/tmp/telemetry.json <<EOF
{"device_id":"$ID","platform":"synology","timestamp":$TS,"uptime":$UP,"version":"0.1-0001","wan":{"wan1":{"ip":"$WAN1"},"wan2":{"ip":"$WAN2"},"gateway":"$GW"}}
EOF
curl -s --connect-timeout 5 --max-time 10 -X POST "$HUB" \
-H "Content-Type: application/json" \
-d @/tmp/telemetry.json >/dev/null 2>&1
rm -f /tmp/telemetry.json