# Pioneer Bus Router — Project 1: On-Device Multi-WAN Balancer

**Status:** Design approved (2026-06-30) · **Owner:** Keylink IT · **Client:** Pioneer Coach Buses
**Target hardware:** GL.iNet GL-XE3000 (Puli AX) · OpenWrt 21.02-SNAPSHOT · MediaTek MT7981 (ARMv8) · kernel 5.4

---

## 1. Context & Goal

Pioneer Coach Buses is replacing its **eyeride** solution with a fleet of GL-XE3000 routers (3 in the
next month, **194 eventually**). Each bus needs **high-quality, highly stable internet** from multiple
uplinks, with automatic, intelligent use of whichever links are healthy.

This project (Project 1) is the **on-router software**: the multi-WAN balancing brain plus the telemetry
agent that later feeds the central fleet console (Project 2). It is a **port** of the existing Synology
SRM package [`aiwanbal`](https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer) to OpenWrt — we reuse
the *logic*, rewrite the *platform layer*.

### Scope decomposition (whole program)
1. **Project 1 — On-router multi-WAN balancer (this spec)**
2. Project 2 — Central fleet console (map, status, reporting, remote config, OTA) — *separate spec*
3. Project 3 — Starlink deep integration & reporting — *folded partly into P1, extended in P2*
4. Project 4 — Fleet provisioning / mass rollout of 194 units — *separate spec*

---

## 2. Hardware constraints (load-bearing)

- **Single 5G modem, dual-SIM *single-standby*.** The Quectel RM520N-GL has two SIM slots but only **one
  SIM is active at a time**. Therefore a single XE3000 **cannot** simultaneously balance T-Mobile *and*
  Verizon. Decision: run **one active cellular WAN** and treat the second SIM as **failover**.
- **Ports:** 2.5 GbE WAN, 1 GbE LAN, USB 2.0, microSD. GNSS/GPS available via the modem.
- **Upstream observed in the field unit:** cellular `rmnet_mhi0` on a **carrier CGNAT address**
  (e.g. `21.188.231.63/25`) — no public inbound, which is why remote management uses an **outbound**
  WireGuard tunnel (see §6).

---

## 3. Per-bus network design

| WAN | Link | Role |
|-----|------|------|
| **WAN1 — Cellular 5G** | internal modem, dual-SIM single-standby | Balanced. Auto-switch to the other SIM when the active carrier degrades past threshold. |
| **WAN2 — Starlink** | 2.5 GbE WAN port → Starlink | Balanced. |

- **Normal state:** 5G + Starlink **balanced together** (2 live WANs) via `mwan3`.
- **Degradation paths:** bad SIM → switch SIM (hysteresis); bad cellular overall → lean on Starlink;
  bad Starlink → lean on cellular. Never strand a bus offline.
- **Downstream:** LAN → Omada fully-managed switch. Roof antenna provides 5G + GPS + Wi-Fi.

---

## 4. Platform approach

- **Base OS:** **GL.iNet stock firmware** (OpenWrt 21.02). We do **not** build a custom firmware image —
  this keeps GL's security/stability updates flowing and the Quectel modem fully supported.
- **Delivery:** our software ships as a **branded `opkg` package + `init.d` service**, installed on top.
  Persists across reboots, OTA-updatable, survives GL firmware updates when done carefully.
- **Balancing engine:** **`mwan3`** (OpenWrt standard weighted multi-WAN + failover). Our daemon tunes
  `mwan3` member weights instead of Synology SmartWAN slots / `srm.json`.
- **Branding:** applied to the layer customers see — local dashboard, SSID/login identity, hostname, and
  the fleet console — **not** a full firmware rebrand.

### Reuse map (from `aiwanbal`)
| Reused logic | Rewritten platform glue |
|---|---|
| 6-factor scoring + rolling average | SmartWAN weights → `mwan3` member weights |
| 3-layer dead-gateway detection (DNS→ping quorum→HTTP) | SRM `srm.json`/`tctool` → `mwan3`/`tc` |
| Adaptive, data-cap-aware speed testing (Ookla + curl) | `synopkg` lifecycle → `opkg` + `init.d` |
| Modem signal monitoring (RSRP/RSRQ/SINR) | SRM modem drivers → GL modem manager / QMI / AT |
| Starlink gRPC helper | SRM-embedded UI → LuCI/GL plugin |

---

## 5. Components (each independently testable)

1. **Scoring daemon** (`busrouter-daemon`) — polls each WAN's latency / throughput / jitter / loss /
   signal, computes the 6-factor score with rolling 3-sample average. Adapted from 2 generic WANs to
   **cellular + Starlink**. Writes desired weights.
2. **WAN controller** (`busrouter-mwan`) — translates scores → `mwan3` member weights + failover; ports
   the 3-layer dead-gateway check. **Hysteresis** so weights don't flap; minimizes conntrack-flush resets.
3. **SIM manager** (`busrouter-modem`) — reads RSRP/RSRQ/SINR via the GL modem manager; **auto-switches
   the active SIM** when the live carrier degrades past threshold, with cooldown/hysteresis to prevent
   ping-pong. ("Another 5G when signal gets bad.")
4. **Starlink integration** (`busrouter-starlink`) — gRPC to the dish (status, obstruction, outage),
   feeding both scoring and telemetry.
5. **Speed-test engine** (`busrouter-speedtest`) — Ookla CLI + curl fallback, adaptive interval,
   **data-cap-aware on cellular** (metered SIM skips auto-tests).
6. **Telemetry agent** (`busrouter-agent`) — collects **GPS (modem GNSS)**, WAN states/scores, active SIM,
   signal, throughput, Starlink health, uptime; posts to the fleet hub over the **WireGuard tunnel**, and
   pulls remote config / OTA triggers. Defines the **contract** Project 2 consumes.
7. **Branded local UI** — lightweight LuCI / GL-plugin status + diagnostics page (live WAN status, scores,
   signal, GPS, manual speed test, failover state). **Read-mostly** — real control is centralized — plus
   emergency local overrides for field techs.

---

## 6. Management plane (provisioned 2026-06-30)

Remote access and telemetry ride a single **WireGuard hub**, which **doubles as the future fleet console
host** (Project 2). Buses dial **out** as WG clients — CGNAT-safe.

```
        ┌──────── WireGuard hub: kit-fleet-hub ────────┐
        │  DO droplet · 167.172.237.162 · nyc3         │
        │  s-1vcpu-1gb ($6/mo, resize as fleet grows)  │
        │  wg0 10.88.0.1/24 · ListenPort 51820         │
        │  hub pubkey 4QAauZ9gS/wZ/u0Wf66OcJFGo4er...  │
        └──────────────────────────────────────────────┘
            ▲                  ▲                  ▲
   bus01 = 10.88.0.2     LAB (jump via hub)   fleet console (later, same box)
   (GL-XE3000)           ssh -J hub → router  10.88.0.1
```

- **Per-bus WG client config** stored on the hub at `/root/busXX-wireguard.conf`, imported into GL's
  WireGuard client UI. `AllowedIPs = 10.88.0.0/24` only — **user internet stays on the balanced WANs**;
  the tunnel is pure out-of-band management.
- **LAB access:** the LAB SSHes routers via the hub as a **jump host** (`ssh -J hub root@10.88.0.2`),
  key-based once the LAB public key is installed on the router.
- **Local dev access:** local Claude Code on the operator PC deploys/tests over LAN (`192.168.8.1`) — no
  tunnel needed for the fast build/test loop.
- **Division of labor:** local PC = build/deploy over LAN; LAB = remote SSH + management over WireGuard.

---

## 7. Config & error handling

- Master config under `/etc/busrouter/*.conf`, **persists across upgrades**.
- **Fail-safe:** if our daemon dies, both WANs stay up via `mwan3` defaults — a software fault never takes
  a bus offline.
- **Hysteresis** on both SIM-switching and weight changes; cooldown windows to damp oscillation.
- Telemetry agent degrades gracefully when the hub is unreachable (buffer + retry); never blocks routing.

---

## 8. Device identity (provisioning defaults)

- SSID `X0000` / `Pioneer123` (2.4G + 5G), admin user `pioadmin` / `Pioneer321!`, branded hostname.
- Captured as a **reproducible provisioning profile** so all units are identical (formalized in Project 4).
- Router admin (current test unit): web `pioadmin` / `Pioneer321!`; SSH/root password `kitPLANE1!!`.

---

## 9. Testing strategy

- **Unit:** shell logic (scoring, hysteresis, dead-gateway decision) tested in a VM / dev container.
- **Bench:** single XE3000 test unit (`bus01`, WG `10.88.0.2`) — iterate via local LAN push (CGI/HTML hot,
  daemon needs service restart), mirroring `aiwanbal`'s live-push workflow.
- **Field pilot:** 3 buses before scaling. Validate failover, SIM-switch, Starlink handoff, telemetry.

---

## 10. Out of scope (later projects)

Fleet console UI/map/reporting, OTA distribution infrastructure, mass provisioning of 194 units, Omada
switch configuration, Starlink account/dish provisioning.

## 11. Decisions locked

- **Cellular:** one active SIM, the other as failover (no simultaneous dual-carrier). ✅
- **Engine:** `mwan3`. ✅
- **Base:** GL.iNet stock firmware + branded `opkg` layer (no custom firmware image). ✅
- **Mgmt transport:** WireGuard, buses dial out to a shared hub that becomes the fleet console host. ✅
- **Local UI:** branded, read-mostly; control centralized. ✅
- **Hub sizing:** `s-1vcpu-1gb` now (~3 buses), resize up as the fleet grows. ✅