kit-busrouter/docs/superpowers/specs/2026-06-30-pioneer-bus-router-design.md

Pioneer Bus Router — Project 1: On-Device Multi-WAN Balancer

Status: Design approved (2026-06-30) · Owner: Keylink IT · Client: Pioneer Coach Buses Target hardware: GL.iNet GL-XE3000 (Puli AX) · OpenWrt 21.02-SNAPSHOT · MediaTek MT7981 (ARMv8) · kernel 5.4

---

1. Context & Goal

Pioneer Coach Buses is replacing its eyeride solution with a fleet of GL-XE3000 routers (3 in the next month, 194 eventually). Each bus needs high-quality, highly stable internet from multiple uplinks, with automatic, intelligent use of whichever links are healthy.

This project (Project 1) is the on-router software: the multi-WAN balancing brain plus the telemetry agent that later feeds the central fleet console (Project 2). It is a port of the existing Synology SRM package aiwanbal to OpenWrt — we reuse the logic, rewrite the platform layer.

Scope decomposition (whole program)

1. Project 1 — On-router multi-WAN balancer (this spec)
2. Project 2 — Central fleet console (map, status, reporting, remote config, OTA) — separate spec
3. Project 3 — Starlink deep integration & reporting — folded partly into P1, extended in P2
4. Project 4 — Fleet provisioning / mass rollout of 194 units — separate spec

---

2. Hardware constraints (load-bearing)

• Single 5G modem, dual-SIM single-standby. The Quectel RM520N-GL has two SIM slots but only one SIM is active at a time. Therefore a single XE3000 cannot simultaneously balance T-Mobile and Verizon. Decision: run one active cellular WAN and treat the second SIM as failover.
• Ports: 2.5 GbE WAN, 1 GbE LAN, USB 2.0, microSD. GNSS/GPS available via the modem.
• Upstream observed in the field unit: cellular rmnet_mhi0 on a carrier CGNAT address (e.g. 21.188.231.63/25) — no public inbound, which is why remote management uses an outbound WireGuard tunnel (see §6).

---

3. Per-bus network design

WAN	Link	Role
WAN1 — Cellular 5G	internal modem, dual-SIM single-standby	Balanced. Auto-switch to the other SIM when the active carrier degrades past threshold.
WAN2 — Starlink	2.5 GbE WAN port → Starlink	Balanced.

• Normal state: 5G + Starlink balanced together (2 live WANs) via mwan3.
• Degradation paths: bad SIM → switch SIM (hysteresis); bad cellular overall → lean on Starlink; bad Starlink → lean on cellular. Never strand a bus offline.
• Downstream: LAN → Omada fully-managed switch. Roof antenna provides 5G + GPS + Wi-Fi.

---

4. Platform approach

• Base OS: GL.iNet stock firmware (OpenWrt 21.02). We do not build a custom firmware image — this keeps GL's security/stability updates flowing and the Quectel modem fully supported.
• Delivery: our software ships as a branded opkg package + init.d service, installed on top. Persists across reboots, OTA-updatable, survives GL firmware updates when done carefully.
• Balancing engine: mwan3 (OpenWrt standard weighted multi-WAN + failover). Our daemon tunes mwan3 member weights instead of Synology SmartWAN slots / srm.json.
• Branding: applied to the layer customers see — local dashboard, SSID/login identity, hostname, and the fleet console — not a full firmware rebrand.

Reuse map (from aiwanbal)

Reused logic	Rewritten platform glue
6-factor scoring + rolling average	SmartWAN weights → mwan3 member weights
3-layer dead-gateway detection (DNS→ping quorum→HTTP)	SRM srm.json/tctool → mwan3/tc
Adaptive, data-cap-aware speed testing (Ookla + curl)	synopkg lifecycle → opkg + init.d
Modem signal monitoring (RSRP/RSRQ/SINR)	SRM modem drivers → GL modem manager / QMI / AT
Starlink gRPC helper	SRM-embedded UI → LuCI/GL plugin

---

5. Components (each independently testable)

1. Scoring daemon (busrouter-daemon) — polls each WAN's latency / throughput / jitter / loss / signal, computes the 6-factor score with rolling 3-sample average. Adapted from 2 generic WANs to cellular + Starlink. Writes desired weights.
2. WAN controller (busrouter-mwan) — translates scores → mwan3 member weights + failover; ports the 3-layer dead-gateway check. Hysteresis so weights don't flap; minimizes conntrack-flush resets.
3. SIM manager (busrouter-modem) — reads RSRP/RSRQ/SINR via the GL modem manager; auto-switches the active SIM when the live carrier degrades past threshold, with cooldown/hysteresis to prevent ping-pong. ("Another 5G when signal gets bad.")
4. Starlink integration (busrouter-starlink) — gRPC to the dish (status, obstruction, outage), feeding both scoring and telemetry.
5. Speed-test engine (busrouter-speedtest) — Ookla CLI + curl fallback, adaptive interval, data-cap-aware on cellular (metered SIM skips auto-tests).
6. Telemetry agent (busrouter-agent) — collects GPS (modem GNSS), WAN states/scores, active SIM, signal, throughput, Starlink health, uptime; posts to the fleet hub over the WireGuard tunnel, and pulls remote config / OTA triggers. Defines the contract Project 2 consumes.
7. Branded local UI — lightweight LuCI / GL-plugin status + diagnostics page (live WAN status, scores, signal, GPS, manual speed test, failover state). Read-mostly — real control is centralized — plus emergency local overrides for field techs.

---

6. Management plane (provisioned 2026-06-30)

Remote access and telemetry ride a single WireGuard hub, which doubles as the future fleet console host (Project 2). Buses dial out as WG clients — CGNAT-safe.

        ┌──────── WireGuard hub: kit-fleet-hub ────────┐
        │  DO droplet · 167.172.237.162 · nyc3         │
        │  s-1vcpu-1gb ($6/mo, resize as fleet grows)  │
        │  wg0 10.88.0.1/24 · ListenPort 51820         │
        │  hub pubkey 4QAauZ9gS/wZ/u0Wf66OcJFGo4er...  │
        └──────────────────────────────────────────────┘
            ▲                  ▲                  ▲
   bus01 = 10.88.0.2     LAB (jump via hub)   fleet console (later, same box)
   (GL-XE3000)           ssh -J hub → router  10.88.0.1

• Per-bus WG client config stored on the hub at /root/busXX-wireguard.conf, imported into GL's WireGuard client UI. AllowedIPs = 10.88.0.0/24 only — user internet stays on the balanced WANs; the tunnel is pure out-of-band management.
• LAB access: the LAB SSHes routers via the hub as a jump host (ssh -J hub root@10.88.0.2), key-based once the LAB public key is installed on the router.
• Local dev access: local Claude Code on the operator PC deploys/tests over LAN (192.168.8.1) — no tunnel needed for the fast build/test loop.
• Division of labor: local PC = build/deploy over LAN; LAB = remote SSH + management over WireGuard.

---

7. Config & error handling

• Master config under /etc/busrouter/*.conf, persists across upgrades.
• Fail-safe: if our daemon dies, both WANs stay up via mwan3 defaults — a software fault never takes a bus offline.
• Hysteresis on both SIM-switching and weight changes; cooldown windows to damp oscillation.
• Telemetry agent degrades gracefully when the hub is unreachable (buffer + retry); never blocks routing.

---

8. Device identity (provisioning defaults)

• SSID X0000 / Pioneer123 (2.4G + 5G), admin user pioadmin / Pioneer321!, branded hostname.
• Captured as a reproducible provisioning profile so all units are identical (formalized in Project 4).
• Router admin (current test unit): web pioadmin / Pioneer321!; SSH/root password kitPLANE1!!.

---

9. Testing strategy

• Unit: shell logic (scoring, hysteresis, dead-gateway decision) tested in a VM / dev container.
• Bench: single XE3000 test unit (bus01, WG 10.88.0.2) — iterate via local LAN push (CGI/HTML hot, daemon needs service restart), mirroring aiwanbal's live-push workflow.
• Field pilot: 3 buses before scaling. Validate failover, SIM-switch, Starlink handoff, telemetry.

---

10. Out of scope (later projects)

Fleet console UI/map/reporting, OTA distribution infrastructure, mass provisioning of 194 units, Omada switch configuration, Starlink account/dish provisioning.

11. Decisions locked

• Cellular: one active SIM, the other as failover (no simultaneous dual-carrier). ✅
• Engine: mwan3. ✅
• Base: GL.iNet stock firmware + branded opkg layer (no custom firmware image). ✅
• Mgmt transport: WireGuard, buses dial out to a shared hub that becomes the fleet console host. ✅
• Local UI: branded, read-mostly; control centralized. ✅
• Hub sizing: s-1vcpu-1gb now (~3 buses), resize up as the fleet grows. ✅