allen
f15ac69925
Includes: - package/ GL-XE3000 kit-busrouter (opkg) - scripts/provision.sh (GL) and provision-synology.sh (Synology) - syno-balance/ — new WAN balancer replacing aiwanbal (SmartWAN adapter) - kit-connect/ — unified connectivity SPK (Tailscale + reverse SSH) - docs/deployment/synology-rt2600ac-checklist.md — 62-point checklist - docs/provisioning/device-identity.md — fleet identity spec - docs/pilot/checklist.md — field pilot validation - x4078_20260721.dss — reference config backup Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
166 lines
8.9 KiB
Markdown
166 lines
8.9 KiB
Markdown
# Pioneer Bus Router — Project 1: On-Device Multi-WAN Balancer
|
|
|
|
**Status:** Design approved (2026-06-30) · **Owner:** Keylink IT · **Client:** Pioneer Coach Buses
|
|
**Target hardware:** GL.iNet GL-XE3000 (Puli AX) · OpenWrt 21.02-SNAPSHOT · MediaTek MT7981 (ARMv8) · kernel 5.4
|
|
|
|
---
|
|
|
|
## 1. Context & Goal
|
|
|
|
Pioneer Coach Buses is replacing its **eyeride** solution with a fleet of GL-XE3000 routers (3 in the
|
|
next month, **194 eventually**). Each bus needs **high-quality, highly stable internet** from multiple
|
|
uplinks, with automatic, intelligent use of whichever links are healthy.
|
|
|
|
This project (Project 1) is the **on-router software**: the multi-WAN balancing brain plus the telemetry
|
|
agent that later feeds the central fleet console (Project 2). It is a **port** of the existing Synology
|
|
SRM package [`aiwanbal`](https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer) to OpenWrt — we reuse
|
|
the *logic*, rewrite the *platform layer*.
|
|
|
|
### Scope decomposition (whole program)
|
|
1. **Project 1 — On-router multi-WAN balancer (this spec)**
|
|
2. Project 2 — Central fleet console (map, status, reporting, remote config, OTA) — *separate spec*
|
|
3. Project 3 — Starlink deep integration & reporting — *folded partly into P1, extended in P2*
|
|
4. Project 4 — Fleet provisioning / mass rollout of 194 units — *separate spec*
|
|
|
|
---
|
|
|
|
## 2. Hardware constraints (load-bearing)
|
|
|
|
- **Single 5G modem, dual-SIM *single-standby*.** The Quectel RM520N-GL has two SIM slots but only **one
|
|
SIM is active at a time**. Therefore a single XE3000 **cannot** simultaneously balance T-Mobile *and*
|
|
Verizon. Decision: run **one active cellular WAN** and treat the second SIM as **failover**.
|
|
- **Ports:** 2.5 GbE WAN, 1 GbE LAN, USB 2.0, microSD. GNSS/GPS available via the modem.
|
|
- **Upstream observed in the field unit:** cellular `rmnet_mhi0` on a **carrier CGNAT address**
|
|
(e.g. `21.188.231.63/25`) — no public inbound, which is why remote management uses an **outbound**
|
|
WireGuard tunnel (see §6).
|
|
|
|
---
|
|
|
|
## 3. Per-bus network design
|
|
|
|
| WAN | Link | Role |
|
|
|-----|------|------|
|
|
| **WAN1 — Cellular 5G** | internal modem, dual-SIM single-standby | Balanced. Auto-switch to the other SIM when the active carrier degrades past threshold. |
|
|
| **WAN2 — Starlink** | 2.5 GbE WAN port → Starlink | Balanced. |
|
|
|
|
- **Normal state:** 5G + Starlink **balanced together** (2 live WANs) via `mwan3`.
|
|
- **Degradation paths:** bad SIM → switch SIM (hysteresis); bad cellular overall → lean on Starlink;
|
|
bad Starlink → lean on cellular. Never strand a bus offline.
|
|
- **Downstream:** LAN → Omada fully-managed switch. Roof antenna provides 5G + GPS + Wi-Fi.
|
|
|
|
---
|
|
|
|
## 4. Platform approach
|
|
|
|
- **Base OS:** **GL.iNet stock firmware** (OpenWrt 21.02). We do **not** build a custom firmware image —
|
|
this keeps GL's security/stability updates flowing and the Quectel modem fully supported.
|
|
- **Delivery:** our software ships as a **branded `opkg` package + `init.d` service**, installed on top.
|
|
Persists across reboots, OTA-updatable, survives GL firmware updates when done carefully.
|
|
- **Balancing engine:** **`mwan3`** (OpenWrt standard weighted multi-WAN + failover). Our daemon tunes
|
|
`mwan3` member weights instead of Synology SmartWAN slots / `srm.json`.
|
|
- **Branding:** applied to the layer customers see — local dashboard, SSID/login identity, hostname, and
|
|
the fleet console — **not** a full firmware rebrand.
|
|
|
|
### Reuse map (from `aiwanbal`)
|
|
| Reused logic | Rewritten platform glue |
|
|
|---|---|
|
|
| 6-factor scoring + rolling average | SmartWAN weights → `mwan3` member weights |
|
|
| 3-layer dead-gateway detection (DNS→ping quorum→HTTP) | SRM `srm.json`/`tctool` → `mwan3`/`tc` |
|
|
| Adaptive, data-cap-aware speed testing (Ookla + curl) | `synopkg` lifecycle → `opkg` + `init.d` |
|
|
| Modem signal monitoring (RSRP/RSRQ/SINR) | SRM modem drivers → GL modem manager / QMI / AT |
|
|
| Starlink gRPC helper | SRM-embedded UI → LuCI/GL plugin |
|
|
|
|
---
|
|
|
|
## 5. Components (each independently testable)
|
|
|
|
1. **Scoring daemon** (`busrouter-daemon`) — polls each WAN's latency / throughput / jitter / loss /
|
|
signal, computes the 6-factor score with rolling 3-sample average. Adapted from 2 generic WANs to
|
|
**cellular + Starlink**. Writes desired weights.
|
|
2. **WAN controller** (`busrouter-mwan`) — translates scores → `mwan3` member weights + failover; ports
|
|
the 3-layer dead-gateway check. **Hysteresis** so weights don't flap; minimizes conntrack-flush resets.
|
|
3. **SIM manager** (`busrouter-modem`) — reads RSRP/RSRQ/SINR via the GL modem manager; **auto-switches
|
|
the active SIM** when the live carrier degrades past threshold, with cooldown/hysteresis to prevent
|
|
ping-pong. ("Another 5G when signal gets bad.")
|
|
4. **Starlink integration** (`busrouter-starlink`) — gRPC to the dish (status, obstruction, outage),
|
|
feeding both scoring and telemetry.
|
|
5. **Speed-test engine** (`busrouter-speedtest`) — Ookla CLI + curl fallback, adaptive interval,
|
|
**data-cap-aware on cellular** (metered SIM skips auto-tests).
|
|
6. **Telemetry agent** (`busrouter-agent`) — collects **GPS (modem GNSS)**, WAN states/scores, active SIM,
|
|
signal, throughput, Starlink health, uptime; posts to the fleet hub over the **WireGuard tunnel**, and
|
|
pulls remote config / OTA triggers. Defines the **contract** Project 2 consumes.
|
|
7. **Branded local UI** — lightweight LuCI / GL-plugin status + diagnostics page (live WAN status, scores,
|
|
signal, GPS, manual speed test, failover state). **Read-mostly** — real control is centralized — plus
|
|
emergency local overrides for field techs.
|
|
|
|
---
|
|
|
|
## 6. Management plane (provisioned 2026-06-30)
|
|
|
|
Remote access and telemetry ride a single **WireGuard hub**, which **doubles as the future fleet console
|
|
host** (Project 2). Buses dial **out** as WG clients — CGNAT-safe.
|
|
|
|
```
|
|
┌──────── WireGuard hub: kit-fleet-hub ────────┐
|
|
│ DO droplet · 167.172.237.162 · nyc3 │
|
|
│ s-1vcpu-1gb ($6/mo, resize as fleet grows) │
|
|
│ wg0 10.88.0.1/24 · ListenPort 51820 │
|
|
│ hub pubkey 4QAauZ9gS/wZ/u0Wf66OcJFGo4er... │
|
|
└──────────────────────────────────────────────┘
|
|
▲ ▲ ▲
|
|
bus01 = 10.88.0.2 LAB (jump via hub) fleet console (later, same box)
|
|
(GL-XE3000) ssh -J hub → router 10.88.0.1
|
|
```
|
|
|
|
- **Per-bus WG client config** stored on the hub at `/root/busXX-wireguard.conf`, imported into GL's
|
|
WireGuard client UI. `AllowedIPs = 10.88.0.0/24` only — **user internet stays on the balanced WANs**;
|
|
the tunnel is pure out-of-band management.
|
|
- **LAB access:** the LAB SSHes routers via the hub as a **jump host** (`ssh -J hub root@10.88.0.2`),
|
|
key-based once the LAB public key is installed on the router.
|
|
- **Local dev access:** local Claude Code on the operator PC deploys/tests over LAN (`192.168.8.1`) — no
|
|
tunnel needed for the fast build/test loop.
|
|
- **Division of labor:** local PC = build/deploy over LAN; LAB = remote SSH + management over WireGuard.
|
|
|
|
---
|
|
|
|
## 7. Config & error handling
|
|
|
|
- Master config under `/etc/busrouter/*.conf`, **persists across upgrades**.
|
|
- **Fail-safe:** if our daemon dies, both WANs stay up via `mwan3` defaults — a software fault never takes
|
|
a bus offline.
|
|
- **Hysteresis** on both SIM-switching and weight changes; cooldown windows to damp oscillation.
|
|
- Telemetry agent degrades gracefully when the hub is unreachable (buffer + retry); never blocks routing.
|
|
|
|
---
|
|
|
|
## 8. Device identity (provisioning defaults)
|
|
|
|
- SSID `X0000` / `Pioneer123` (2.4G + 5G), admin user `pioadmin` / `Pioneer321!`, branded hostname.
|
|
- Captured as a **reproducible provisioning profile** so all units are identical (formalized in Project 4).
|
|
- Router admin (current test unit): web `pioadmin` / `Pioneer321!`; SSH/root password `kitPLANE1!!`.
|
|
|
|
---
|
|
|
|
## 9. Testing strategy
|
|
|
|
- **Unit:** shell logic (scoring, hysteresis, dead-gateway decision) tested in a VM / dev container.
|
|
- **Bench:** single XE3000 test unit (`bus01`, WG `10.88.0.2`) — iterate via local LAN push (CGI/HTML hot,
|
|
daemon needs service restart), mirroring `aiwanbal`'s live-push workflow.
|
|
- **Field pilot:** 3 buses before scaling. Validate failover, SIM-switch, Starlink handoff, telemetry.
|
|
|
|
---
|
|
|
|
## 10. Out of scope (later projects)
|
|
|
|
Fleet console UI/map/reporting, OTA distribution infrastructure, mass provisioning of 194 units, Omada
|
|
switch configuration, Starlink account/dish provisioning.
|
|
|
|
## 11. Decisions locked
|
|
|
|
- **Cellular:** one active SIM, the other as failover (no simultaneous dual-carrier). ✅
|
|
- **Engine:** `mwan3`. ✅
|
|
- **Base:** GL.iNet stock firmware + branded `opkg` layer (no custom firmware image). ✅
|
|
- **Mgmt transport:** WireGuard, buses dial out to a shared hub that becomes the fleet console host. ✅
|
|
- **Local UI:** branded, read-mostly; control centralized. ✅
|
|
- **Hub sizing:** `s-1vcpu-1gb` now (~3 buses), resize up as the fleet grows. ✅
|