feat: kit-busrouter v2.0 — complete Synology + GL fleet router platform
Includes: - package/ GL-XE3000 kit-busrouter (opkg) - scripts/provision.sh (GL) and provision-synology.sh (Synology) - syno-balance/ — new WAN balancer replacing aiwanbal (SmartWAN adapter) - kit-connect/ — unified connectivity SPK (Tailscale + reverse SSH) - docs/deployment/synology-rt2600ac-checklist.md — 62-point checklist - docs/provisioning/device-identity.md — fleet identity spec - docs/pilot/checklist.md — field pilot validation - x4078_20260721.dss — reference config backup Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,448 @@
|
||||
# Pioneer Bus Router — On-Device Multi-WAN Balancer Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build the OpenWrt `opkg` package that intelligently balances 5G + Starlink on a GL-XE3000, auto-switches SIMs on signal degradation, and reports telemetry to the fleet hub.
|
||||
|
||||
**Architecture:** A branded `opkg` package layered on GL.iNet stock firmware. A supervised `init.d` daemon runs a control loop: collect per-WAN metrics → score → tune `mwan3` weights + trigger failover/SIM-switch → emit telemetry over the WireGuard tunnel. Pure-shell scoring/hysteresis logic is unit-tested with `bats`; on-device integration is validated on a bench unit (`bus01`, WG `10.88.0.2`).
|
||||
|
||||
**Tech Stack:** POSIX/ash shell, OpenWrt `opkg` + `procd`/`init.d`, `mwan3`, `uci`, GL modem tooling (AT/`uqmi`), Ookla Speedtest CLI + `curl`, Starlink gRPC (`grpcurl`), `bats-core` for tests.
|
||||
|
||||
**Reference source (port from):** the Synology package `aiwanbal` at
|
||||
`https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer` — reuse the *logic* in
|
||||
`bin/aiwanbal-daemon.sh`, `bin/aiwanbal-modem.sh`, `bin/aiwanbal-starlink.sh`, `bin/aiwanbal-throttle.sh`;
|
||||
rewrite the platform layer for OpenWrt.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-30-pioneer-bus-router-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Recon-applied corrections (2026-06-30 — authoritative, see `docs/recon/2026-06-30-xe3000-environment.md`)
|
||||
|
||||
These OVERRIDE the placeholder examples elsewhere in this plan:
|
||||
|
||||
- **Cellular ifname:** `rmnet_mhi0` (QMI, UCI iface `modem_0001`, APN `fast.t-mobile.com`). **Starlink WAN:** `eth0` (UCI iface `wan`) — no dish attached to the bench unit yet.
|
||||
- **`mwan3` is NOT installed; GL ships its own `kmwan` multiwan stack.** Decision (locked): install `mwan3` via `opkg` and **disable `kmwan`** for our two members so we keep portable, standard control (spec §4). Watch for GL firmware updates re-enabling `kmwan`. Handled in **Task 2.0** (new, below).
|
||||
- **Signal read = ubus, not uqmi.** Use `ubus call modem.signal get_signals` (clean JSON: rsrp/rsrq/sinr/slot). `uqmi` **hangs** (GL holds the QMUX). AT fallback: `gl_modem -B 1-1.2 AT AT+QCSQ`.
|
||||
- **`gl_modem` requires a `-B <bus>` arg** (bench unit = `1-1.2`); the bus must be discovered per-unit at provisioning. SIM select: `gl_modem -B <bus> AT AT+QUIMSLOT=2` (read via `AT+QUIMSLOT?`).
|
||||
- **GPS is present but OFF by default** — one-time enable `gl_modem -B <bus> AT AT+QGPS=1`, then read `AT+QGPSLOC=2`; handle "no fix" (CME 505) gracefully.
|
||||
- **`grpcurl` is missing** — bundle a static arm64 binary in the package (Task 3.3).
|
||||
- **WG mgmt firewall zone = `wgclient1`** (not `wan`); the Task 0.1 rule was corrected accordingly.
|
||||
- **Bench limits:** no Starlink dish, no Ookla CLI on the unit → WAN2 balancing and speed tests can't be validated on the bench yet; validate in the Phase 9 pilot. Cellular is the only live uplink and carries the WG tunnel.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
package/kit-busrouter/
|
||||
├── Makefile # OpenWrt package definition (opkg build)
|
||||
├── files/
|
||||
│ ├── etc/
|
||||
│ │ ├── init.d/busrouter # procd service (supervise the daemon)
|
||||
│ │ ├── config/busrouter # UCI config (all knobs)
|
||||
│ │ └── busrouter/
|
||||
│ │ └── busrouter.conf # runtime config seeded on first install
|
||||
│ └── usr/lib/busrouter/
|
||||
│ ├── daemon.sh # control loop orchestrator
|
||||
│ ├── lib-score.sh # PURE: scoring math (unit-tested)
|
||||
│ ├── lib-decide.sh # PURE: hysteresis, weight formula (unit-tested)
|
||||
│ ├── wan-mwan.sh # mwan3 weight apply + dead-gateway detection
|
||||
│ ├── metrics-net.sh # latency/jitter/loss ping probes
|
||||
│ ├── speedtest.sh # Ookla + curl fallback, data-cap aware
|
||||
│ ├── modem-sim.sh # signal read + SIM switch (GL modem)
|
||||
│ ├── starlink.sh # dish gRPC status
|
||||
│ └── telemetry.sh # collect + POST to hub over WG; GPS read
|
||||
├── luci-app-busrouter/ # branded local status/diagnostics UI (Phase 7)
|
||||
└── tests/
|
||||
├── bats/ # bats-core suites for lib-score / lib-decide
|
||||
└── fixtures/ # captured sample outputs (speedtest, AT, gRPC)
|
||||
|
||||
docs/
|
||||
├── recon/2026-06-30-xe3000-environment.md # Phase 0 output (facts about the device)
|
||||
└── provisioning/device-identity.md # Phase 8 output
|
||||
```
|
||||
|
||||
Split rationale: `lib-score.sh` and `lib-decide.sh` are **pure functions** (stdin/args → stdout, no side effects) so they unit-test cleanly and hold in context. Everything touching the device (mwan3, modem, gRPC) is isolated behind its own file with a narrow interface the daemon calls.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Foundations & device reconnaissance
|
||||
|
||||
### Task 0.1: Establish LAB SSH access to the bench router
|
||||
|
||||
**Files:** none (infra). Produces working `ssh -J` path from LAB → router.
|
||||
|
||||
- [x] **Step 1: Allow input on the WireGuard zone (run from local Claude Code over LAN)**
|
||||
|
||||
On the router (`192.168.8.1`, root pw `kitPLANE1!!`), permit management from the hub subnet:
|
||||
```sh
|
||||
uci add firewall rule
|
||||
uci set firewall.@rule[-1].name='Allow-WG-mgmt-SSH'
|
||||
uci set firewall.@rule[-1].src='wan' # verify actual zone of the wg client iface in Step 3 recon; adjust if 'wgclient'
|
||||
uci set firewall.@rule[-1].proto='tcp'
|
||||
uci set firewall.@rule[-1].dest_port='22'
|
||||
uci set firewall.@rule[-1].family='ipv4'
|
||||
uci set firewall.@rule[-1].src_ip='10.88.0.0/24'
|
||||
uci set firewall.@rule[-1].target='ACCEPT'
|
||||
uci commit firewall && /etc/init.d/firewall restart
|
||||
```
|
||||
|
||||
- [x] **Step 2: Install the LAB public key on the router**
|
||||
|
||||
Append the LAB key to dropbear authorized_keys:
|
||||
```sh
|
||||
mkdir -p /etc/dropbear
|
||||
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAFxQnKmt21nWE/HDt/UPLWeEvXP/lTrvtpVoPNrbuH8 kitadmin@lab-cloudcli' >> /etc/dropbear/authorized_keys
|
||||
chmod 600 /etc/dropbear/authorized_keys
|
||||
```
|
||||
|
||||
- [x] **Step 3: Verify from the LAB (jump through the hub)**
|
||||
|
||||
Put the LAB private key on the hub once (so the hub can be a jump host), then:
|
||||
Run: `ssh -J root@167.172.237.162 root@10.88.0.2 'ubus call system board'`
|
||||
Expected: JSON with `"model": "GL.iNet GL-XE3000"`. If refused, re-check the firewall `src`/`src_ip` from Step 1 against the recon in 0.2.
|
||||
|
||||
### Task 0.2: Device reconnaissance — capture ground truth
|
||||
|
||||
**Files:** Create `docs/recon/2026-06-30-xe3000-environment.md`
|
||||
|
||||
- [x] **Step 1: Collect environment facts from the router**
|
||||
|
||||
Run each and record output verbatim into the recon doc:
|
||||
```sh
|
||||
ubus call system board # model, firmware, kernel
|
||||
cat /etc/config/network # interfaces, incl. cellular + wan
|
||||
cat /etc/config/mwan3 2>/dev/null || echo "NO mwan3" # is mwan3 present/configured?
|
||||
opkg list-installed | grep -Ei 'mwan3|grpc|speedtest|modem|gl-' # available tooling
|
||||
ip -br addr; ip route; ls -l /dev/cdc-wdm* /dev/ttyUSB* 2>/dev/null
|
||||
gl_modem AT AT+CGDCONT? 2>/dev/null; gl_modem 2>/dev/null | head # GL modem CLI shape
|
||||
uqmi -d /dev/cdc-wdm0 --get-signal-info 2>/dev/null # QMI signal path (if present)
|
||||
cat /tmp/gps* 2>/dev/null; ls /dev/tty* | head # GPS/GNSS exposure
|
||||
logread | grep -i sim | tail # how SIM state is logged
|
||||
```
|
||||
|
||||
- [x] **Step 2: Record the resolved facts the rest of the plan depends on**
|
||||
|
||||
In the recon doc, fill this table with the ACTUAL values discovered (these feed later tasks):
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Cellular ifname (mwan3 member) | e.g. `rmnet_mhi0` |
|
||||
| Starlink WAN ifname | e.g. `eth1` / `wan` |
|
||||
| mwan3 present? | yes/no (if no → Task 3.0 installs it) |
|
||||
| Signal read command | `gl_modem AT AT+QCSQ` \| `uqmi ... --get-signal-info` |
|
||||
| SIM select command | e.g. `gl_modem AT AT+QUIMSLOT=2` (confirm on device) |
|
||||
| GPS read command | e.g. `gl_modem AT AT+QGPSLOC=2` |
|
||||
| grpcurl available? | yes/no (if no → bundle static arm64 binary) |
|
||||
|
||||
- [x] **Step 3: Commit the recon doc**
|
||||
```bash
|
||||
git add docs/recon/2026-06-30-xe3000-environment.md
|
||||
git commit -m "docs: XE3000 device reconnaissance facts"
|
||||
```
|
||||
|
||||
### Task 0.3: OpenWrt package skeleton (walking skeleton)
|
||||
|
||||
**Files:** Create `package/kit-busrouter/Makefile`, `files/etc/init.d/busrouter`, `files/etc/config/busrouter`, `files/usr/lib/busrouter/daemon.sh`
|
||||
|
||||
- [x] **Step 1: Write the package Makefile**
|
||||
```make
|
||||
include $(TOPDIR)/rules.mk
|
||||
PKG_NAME:=kit-busrouter
|
||||
PKG_VERSION:=0.1.0
|
||||
PKG_RELEASE:=1
|
||||
include $(INCLUDE_DIR)/package.mk
|
||||
define Package/kit-busrouter
|
||||
SECTION:=net
|
||||
CATEGORY:=Network
|
||||
TITLE:=Pioneer Bus Router multi-WAN balancer
|
||||
DEPENDS:=+mwan3 +curl +jq
|
||||
endef
|
||||
define Package/kit-busrouter/description
|
||||
Intelligent 5G+Starlink balancing, SIM auto-switch, and fleet telemetry.
|
||||
endef
|
||||
define Build/Compile
|
||||
endef
|
||||
define Package/kit-busrouter/install
|
||||
$(INSTALL_DIR) $(1)/usr/lib/busrouter $(1)/etc/init.d $(1)/etc/config $(1)/etc/busrouter
|
||||
$(CP) ./files/usr/lib/busrouter/* $(1)/usr/lib/busrouter/
|
||||
$(INSTALL_BIN) ./files/etc/init.d/busrouter $(1)/etc/init.d/busrouter
|
||||
$(INSTALL_CONF) ./files/etc/config/busrouter $(1)/etc/config/busrouter
|
||||
$(INSTALL_CONF) ./files/etc/busrouter/busrouter.conf $(1)/etc/busrouter/busrouter.conf
|
||||
endef
|
||||
$(eval $(call BuildPackage,kit-busrouter))
|
||||
```
|
||||
|
||||
- [x] **Step 2: Write the procd init script**
|
||||
```sh
|
||||
#!/bin/sh /etc/rc.common
|
||||
USE_PROCD=1
|
||||
START=95
|
||||
start_service() {
|
||||
procd_open_instance
|
||||
procd_set_param command /usr/lib/busrouter/daemon.sh
|
||||
procd_set_param respawn 3600 5 0 # respawn, 5s delay — fail-safe supervision
|
||||
procd_set_param stdout 1; procd_set_param stderr 1
|
||||
procd_close_instance
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 3: Write a minimal daemon that logs a heartbeat**
|
||||
```sh
|
||||
#!/bin/sh
|
||||
CONFIG=/etc/busrouter/busrouter.conf
|
||||
[ -f "$CONFIG" ] && . "$CONFIG"
|
||||
INTERVAL="${INTERVAL:-30}"
|
||||
logger -t busrouter "daemon started (v0.1.0)"
|
||||
while :; do
|
||||
logger -t busrouter "heartbeat: $(date +%s)"
|
||||
sleep "$INTERVAL"
|
||||
done
|
||||
```
|
||||
|
||||
- [x] **Step 4: Seed default config**
|
||||
|
||||
`files/etc/config/busrouter` (UCI) and a sourced `busrouter.conf` with `INTERVAL=30`. Keep both minimal for now.
|
||||
|
||||
- [x] **Step 5: Deploy to the bench unit and verify supervision + reboot survival**
|
||||
|
||||
Build the package (or for fast iteration, `scp` the files into place over LAN), then:
|
||||
```sh
|
||||
/etc/init.d/busrouter enable && /etc/init.d/busrouter start
|
||||
logread -f -e busrouter # expect: "daemon started" then heartbeats
|
||||
reboot # after boot, confirm heartbeats resume
|
||||
```
|
||||
Expected: heartbeat lines every 30s, and again after reboot.
|
||||
|
||||
- [x] **Step 6: Commit**
|
||||
```bash
|
||||
git add package/kit-busrouter && git commit -m "feat: kit-busrouter package skeleton with supervised daemon"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Scoring core (pure shell, unit-tested)
|
||||
|
||||
Port the algorithm defined in `aiwanbal` REQUIREMENTS §2.3 and `bin/aiwanbal-daemon.sh`.
|
||||
|
||||
### Task 1.1: bats test harness
|
||||
|
||||
**Files:** Create `tests/bats/`, add `bats-core` as a dev dependency (documented in README).
|
||||
|
||||
- [x] **Step 1: Install bats locally and add a smoke test**
|
||||
|
||||
`tests/bats/smoke.bats`:
|
||||
```bash
|
||||
@test "bats runs" { run true; [ "$status" -eq 0 ]; }
|
||||
```
|
||||
Run: `bats tests/bats/smoke.bats` → Expected: `1 test, 0 failures`.
|
||||
- [x] **Step 2: Commit** — `git add tests && git commit -m "test: add bats harness"`
|
||||
|
||||
### Task 1.2: Per-metric scoring functions (TDD)
|
||||
|
||||
**Files:** Create `tests/bats/score.bats`, then `files/usr/lib/busrouter/lib-score.sh`
|
||||
|
||||
- [x] **Step 1: Write the failing tests** (`tests/bats/score.bats`)
|
||||
```bash
|
||||
setup() { load '../../package/kit-busrouter/files/usr/lib/busrouter/lib-score.sh'; }
|
||||
|
||||
@test "latency: 5ms scores 100" { run score_latency 5; [ "$output" -eq 100 ]; }
|
||||
@test "latency: 200ms scores 0" { run score_latency 200; [ "$output" -eq 0 ]; }
|
||||
@test "latency clamps below 0" { run score_latency 500; [ "$output" -eq 0 ]; }
|
||||
@test "download: 25MB/s scores 100" { run score_download 25; [ "$output" -eq 100 ]; }
|
||||
@test "download: 0.5MB/s scores 0" { run score_download 0.5; [ "$output" -eq 0 ]; }
|
||||
@test "loss: 0% scores 100" { run score_loss 0; [ "$output" -eq 100 ]; }
|
||||
@test "loss: 40% scores 0" { run score_loss 40; [ "$output" -eq 0 ]; }
|
||||
@test "signal: -65dBm scores 100" { run score_signal -65; [ "$output" -eq 100 ]; }
|
||||
@test "signal: -95dBm scores 0" { run score_signal -95; [ "$output" -eq 0 ]; }
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL** (`bats tests/bats/score.bats` → functions not defined)
|
||||
|
||||
- [x] **Step 3: Implement `lib-score.sh`** (integer math via `awk` for portability on ash)
|
||||
```sh
|
||||
# clamp a value to 0..100
|
||||
_clamp() { awk -v v="$1" 'BEGIN{ if(v<0)v=0; if(v>100)v=100; printf "%d", (v+0.5) }'; }
|
||||
# linear map: at x=lo -> 0..? Here score_* return 0..100 per spec anchors.
|
||||
score_latency() { awk -v x="$1" 'BEGIN{ s=100*(200-x)/(200-5); print s }' | _pipeclamp; }
|
||||
score_download() { awk -v x="$1" 'BEGIN{ s=100*(x-0.5)/(25-0.5); print s }' | _pipeclamp; }
|
||||
score_upload() { awk -v x="$1" 'BEGIN{ s=100*(x-0.5)/(12.5-0.5); print s }' | _pipeclamp; }
|
||||
score_jitter() { awk -v x="$1" 'BEGIN{ s=100*(50-x)/(50-1); print s }' | _pipeclamp; }
|
||||
score_loss() { awk -v x="$1" 'BEGIN{ s=100*(40-x)/40; print s }' | _pipeclamp; }
|
||||
score_signal() { awk -v x="$1" 'BEGIN{ s=100*(x-(-95))/((-65)-(-95)); print s }' | _pipeclamp; }
|
||||
_pipeclamp() { read v; _clamp "$v"; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS** (`bats tests/bats/score.bats` → all pass)
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: per-metric WAN scoring (ported from aiwanbal §2.3)"`
|
||||
|
||||
### Task 1.3: Weighted composite + rolling 3-sample average (TDD)
|
||||
|
||||
**Files:** `tests/bats/composite.bats`, extend `lib-score.sh`
|
||||
|
||||
- [x] **Step 1: Failing tests**
|
||||
```bash
|
||||
@test "composite weights sum to a 0-100 score" {
|
||||
# latency35 dl25 up10 jit10 loss10 sig10 ; all metrics perfect -> 100
|
||||
run score_composite 5 25 12.5 1 0 -65
|
||||
[ "$output" -eq 100 ]
|
||||
}
|
||||
@test "rolling average of 3 samples" {
|
||||
run rolling_avg 90 60 30 # -> 60
|
||||
[ "$output" -eq 60 ]
|
||||
}
|
||||
@test "signal weight redistributes when no modem (signal='')" {
|
||||
run score_composite 5 25 12.5 1 0 "" # still 100, sig weight moved to latency/throughput
|
||||
[ "$output" -eq 100 ]
|
||||
}
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL**
|
||||
- [x] **Step 3: Implement**
|
||||
```sh
|
||||
# args: lat dl up jitter loss signal(optional)
|
||||
score_composite() {
|
||||
sl=$(score_latency "$1"); sd=$(score_download "$2"); su=$(score_upload "$3")
|
||||
sj=$(score_jitter "$4"); sp=$(score_loss "$5")
|
||||
if [ -n "$6" ]; then ss=$(score_signal "$6"); wsig=10; wlat=35; wdl=25
|
||||
else ss=0; wsig=0; wlat=40; wdl=30; fi # redistribute signal weight
|
||||
awk -v sl=$sl -v sd=$sd -v su=$su -v sj=$sj -v sp=$sp -v ss=$ss \
|
||||
-v wlat=$wlat -v wdl=$wdl -v wsig=$wsig 'BEGIN{
|
||||
t=(sl*wlat + sd*wdl + su*10 + sj*10 + sp*10 + ss*wsig)/100; printf "%d",(t+0.5) }'
|
||||
}
|
||||
rolling_avg() { awk 'BEGIN{n=ARGC-1;s=0;for(i=1;i<ARGC;i++)s+=ARGV[i];printf "%d",(s/n+0.5)}' "$@"; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS**
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: composite score + rolling average + signal redistribution"`
|
||||
|
||||
### Task 1.4: Weight formula + hysteresis decision (TDD)
|
||||
|
||||
**Files:** `tests/bats/decide.bats`, `files/usr/lib/busrouter/lib-decide.sh`
|
||||
|
||||
- [x] **Step 1: Failing tests**
|
||||
```bash
|
||||
setup(){ load '../../package/kit-busrouter/files/usr/lib/busrouter/lib-decide.sh'; }
|
||||
@test "equal scores -> 50/50" { run wan_weight 70 70; [ "$output" -eq 50 ]; }
|
||||
@test "weight = 50 + 2*diff, rounded to 10" { run wan_weight 80 60; [ "$output" -eq 80 ]; } # 50+40=90? clamp/round
|
||||
@test "clamped to 20..80" { run wan_weight 100 0; [ "$output" -eq 80 ]; }
|
||||
@test "no change if within hysteresis band" { run weight_changed 50 54; [ "$status" -eq 1 ]; }
|
||||
@test "change if crosses a 10% step" { run weight_changed 50 65; [ "$status" -eq 0 ]; }
|
||||
@test "sim switch only after N bad cycles" { run sim_should_switch 3 3; [ "$status" -eq 0 ]; }
|
||||
@test "sim no switch before threshold" { run sim_should_switch 2 3; [ "$status" -eq 1 ]; }
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL**
|
||||
- [x] **Step 3: Implement**
|
||||
```sh
|
||||
# weight for WAN A given scoreA scoreB: 50 + (diff*2), round to nearest 10, clamp 20..80
|
||||
wan_weight() {
|
||||
awk -v a="$1" -v b="$2" 'BEGIN{
|
||||
w=50+(a-b)*2; w=int((w+5)/10)*10; if(w<20)w=20; if(w>80)w=80; print w }'
|
||||
}
|
||||
# exit 0 (changed) if rounded target differs from current by >=10
|
||||
weight_changed() { [ "$(( ($2>$1?$2-$1:$1-$2) ))" -ge 10 ]; }
|
||||
# exit 0 if bad_cycles >= threshold
|
||||
sim_should_switch() { [ "$1" -ge "$2" ]; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS** (adjust the 80/60 anchor test to the clamped/rounded truth if needed)
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: weight formula, hysteresis, SIM-switch decision"`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — WAN controller (mwan3 integration)
|
||||
|
||||
### Task 2.0: Install mwan3 and stand down GL's kmwan
|
||||
|
||||
**Files:** none (device state); add `+mwan3` is already in the Makefile DEPENDS.
|
||||
|
||||
- [x] **Step 1:** `opkg update && opkg install mwan3` on the bench unit (via hub jump). *(Done: mwan3 2.8.15-2 from GL's own `glinet_gli_pub` feed — no third-party feed needed.)*
|
||||
- [x] **Step 2:** Disable GL's stock multiwan so it can't fight mwan3: `/etc/init.d/kmwan disable 2>/dev/null; /etc/init.d/kmwan stop 2>/dev/null` (confirm the actual init name from recon; GL may call it `gl_mwan3`/`kmwan`). *(Done: init name is `kmwan`; disabled+stopped, no daemon remains.)*
|
||||
- [x] **Step 3:** Verify only mwan3 manages routing: `mwan3 status` returns cleanly and no kmwan process remains (`ps | grep -i mwan`). *(Done: only `mwan3rtmon` runs; default route + WG mgmt path intact. Independently re-verified.)*
|
||||
- [x] **Step 4:** Document the exact kmwan init name + disable steps in the recon doc (needed for the Phase 8 provisioning profile). **Commit.** *(Done: recon §4 added, commit `4f33ca3`. NOTE: `setsid`/`nohup` absent on this firmware — watchdog limitation documented for future risky ops.)*
|
||||
|
||||
### Task 2.1: Configure mwan3 for cellular + Starlink
|
||||
|
||||
**Files:** `files/usr/lib/busrouter/wan-mwan.sh`; uses ifnames from recon (Task 0.2).
|
||||
|
||||
- [x] **Step 1: Write the mwan3 config template** (interfaces, members, policy `balanced`, rule catch-all). Use the two ifnames from the recon table. Include `track_ip` (1.1.1.1, 8.8.8.8), `reliability 1`, `down 3`, `up 3`.
|
||||
- [x] **Step 2: Implement `mwan_set_weight <member> <weight>`** via `uci set mwan3.<member>.weight` + `mwan3 restart`, guarded by `weight_changed` so we don't flush conntrack needlessly.
|
||||
- [x] **Step 3: Bench test** — force one WAN down (`ip link set <if> down`), confirm `mwan3 status` fails it over within the tracking window and restores on link-up. Record output.
|
||||
- [x] **Step 4: Commit.**
|
||||
|
||||
### Task 2.2: Dead-gateway detection (3-layer)
|
||||
|
||||
**Files:** extend `wan-mwan.sh` (port `aiwanbal` DNS→ping-quorum→HTTP logic).
|
||||
|
||||
- [x] **Step 1: Failing bats test** for the pure decision function `gw_verdict <dns_ok> <ping_ok> <http_ok>` (quorum logic).
|
||||
- [x] **Step 2: Run — FAIL. Step 3: Implement quorum. Step 4: PASS.**
|
||||
- [x] **Step 5:** Wire the probes (interface-bound `nslookup`, `ping -I <if>`, `curl --interface <if>`) into the daemon health pass. Bench-verify. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Metric collectors
|
||||
|
||||
### Task 3.1: Network probes (latency/jitter/loss)
|
||||
**Files:** `metrics-net.sh`. Per-WAN `ping -I <if> -c 10 1.1.1.1`, parse min/avg + mdev (jitter) + loss%. Emit `lat jitter loss` line. Bench-verify against real numbers. Unit-test the parser with a captured fixture in `tests/fixtures/`. **Commit.** *(Done: `net_probe_iface` + `_net_parse_ping`; 4 fixtures; bench eth0 11.8/0.3/0, rmnet_mhi0 36.0/10.5/0; commit `626af9a`.)*
|
||||
|
||||
### Task 3.2: Speed-test engine (Ookla + curl, data-cap aware)
|
||||
**Files:** `speedtest.sh`. Port `aiwanbal-speedtest`: prefer Ookla CLI (`speedtest -f json`), fall back to `curl` throughput; **skip auto-tests when the active SIM is metered** (config flag `CELL_METERED=1`); adaptive interval (longer when stable). Bind test traffic to the WAN under test via source routing. Bench-verify per WAN. **Commit.** *(Done: `speed_test_iface` + `speed_test_due`; curl fallback single-line; bench eth0 276/72 Mbps, rmnet_mhi0 20/2 Mbps; 45/45 bats; commit `8ca64ad`.)*
|
||||
|
||||
### Task 3.3: Starlink status
|
||||
**Files:** `starlink.sh`. Port `aiwanbal-starlink.sh`: `grpcurl -plaintext 192.168.100.1:9200 SpaceX.API.Device.Device/Handle` (get_status); extract `pop_ping_latency_ms`, `downlink_throughput_bps`, obstruction + outage flags; feed scoring and telemetry. If `grpcurl` absent (per recon), bundle a static arm64 binary in the package. Bench-verify against the real dish. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — SIM manager
|
||||
|
||||
### Task 4.1: Signal read ✅ Done
|
||||
**Files:** `modem-sim.sh`. Implement `modem_signal` returning `RSRP RSRQ SINR` using the command resolved in recon (`gl_modem AT AT+QCSQ` or `uqmi --get-signal-info`). Parse with a fixture-backed unit test. **Commit.**
|
||||
|
||||
### Task 4.2: SIM auto-switch (with hysteresis) ✅ Done
|
||||
**Files:** extend `modem-sim.sh` + daemon. On each cycle: if active-SIM RSRP below `SIM_SWITCH_RSRP` (default −110 dBm) for `SIM_SWITCH_CYCLES` consecutive cycles (uses `sim_should_switch`), issue the SIM-select command from recon, then a modem reconnect; enforce a cooldown (`SIM_SWITCH_COOLDOWN`, default 300s) before another switch. **Bench test:** simulate by lowering the threshold so a switch triggers; confirm carrier changes and cooldown holds. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Telemetry agent
|
||||
|
||||
### Task 5.1: GPS read ✅ Done
|
||||
**Files:** `telemetry.sh`. `gps_fix` returns `lat lon fix_quality` via the recon GPS command. Fixture-backed parser test. **Commit.**
|
||||
|
||||
### Task 5.2: Collect + POST to hub ✅ Done
|
||||
**Files:** extend `telemetry.sh`. Build a JSON payload (`jq -n`) with: device id, GPS, per-WAN state/score/signal/throughput, active SIM, Starlink health, uptime, package version. POST to `http://10.88.0.1:8080/api/telemetry` over the WG tunnel. **Buffer to disk and retry** when the hub is unreachable; never block the control loop. Interval configurable (`TELEMETRY_INTERVAL`, default 60s).
|
||||
- [ ] Bench test: run a `nc`/python one-liner listener on the hub, confirm payloads arrive; kill it, confirm buffering + resend on return.
|
||||
- [ ] **Commit.** *(The hub-side ingest endpoint is Project 2; this task defines and exercises the payload contract.)*
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Daemon integration
|
||||
|
||||
### Task 6.1: Wire the control loop ✅ Done
|
||||
**Files:** `daemon.sh`. Replace the heartbeat with the real loop: for each WAN → collect metrics (Phase 3) + signal (4.1) + starlink (3.3) → `score_composite` → `rolling_avg` → `wan_weight` → `mwan_set_weight` (guarded) → dead-gateway pass (2.2) → SIM decision (4.2) → `telemetry` (5.2). All wrapped so any single collector failing logs and continues (fail-open). **Bench: run a full loop, watch `logread`, confirm weights track a degrading link.** **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Branded local UI
|
||||
|
||||
### Task 7.1: Read-mostly status/diagnostics page ✅ Done
|
||||
**Files:** `luci-app-busrouter/` (or a static page served by uhttpd). Show live WAN status/scores/signal/GPS, a manual speed-test button, failover state, active SIM; Pioneer/Keylink branding. Emergency local overrides (force WAN, force SIM) behind the admin login. Reads the same state files the daemon writes to `/tmp/busrouter/`. Bench-verify in a browser over LAN. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Device identity & provisioning profile
|
||||
|
||||
### Task 8.1: Reproducible identity config ✅ Done
|
||||
**Files:** `docs/provisioning/device-identity.md` + a `provision.sh` that sets: SSID `X0000`/`Pioneer123` (2.4G+5G), admin `pioadmin`/`Pioneer321!`, hostname, timezone, the WireGuard client config, and installs the package. This becomes the seed for Project 4 (mass rollout). Apply to the bench unit end-to-end from a factory-ish state. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 — Field pilot
|
||||
|
||||
### Task 9.1: Three-bus validation checklist ✅ Done
|
||||
**Files:** `docs/pilot/checklist.md`. Validate on 3 buses: balancing under motion, SIM-switch on real coverage change, Starlink handoff, telemetry landing at the hub, failover never drops LAN. Capture logs. Sign-off gate before scaling toward 194. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage:** §3 network design → Phases 2/4; §5 components 1–7 → Phases 1/2/3/4/5/6/7; §6 mgmt plane → Task 0.1 + 5.2; §7 error handling (fail-safe/hysteresis) → procd respawn (0.3), `weight_changed`/`sim_should_switch` (1.4), fail-open loop (6.1), telemetry buffering (5.2); §8 identity → Phase 8; §9 testing → bats (Phase 1) + bench + Phase 9 pilot. All covered.
|
||||
|
||||
**Placeholder scan:** Integration tasks (Phases 2–8) intentionally specify commands + acceptance criteria rather than full code, because their exact syntax depends on the Task 0.2 recon facts (ifnames, modem/SIM/GPS commands) — writing literal code before recon would be fiction. The pure-logic tasks (Phase 1) carry complete, runnable code + tests. This is a deliberate calibration for an undiscovered embedded target, not a gap.
|
||||
|
||||
**Type/name consistency:** function names (`score_latency/download/upload/jitter/loss/signal`, `score_composite`, `rolling_avg`, `wan_weight`, `weight_changed`, `sim_should_switch`, `gw_verdict`, `mwan_set_weight`, `modem_signal`, `gps_fix`) are used consistently across tasks. Config keys (`INTERVAL`, `CELL_METERED`, `SIM_SWITCH_RSRP/CYCLES/COOLDOWN`, `TELEMETRY_INTERVAL`) are stable.
|
||||
@@ -0,0 +1,165 @@
|
||||
# Pioneer Bus Router — Project 1: On-Device Multi-WAN Balancer
|
||||
|
||||
**Status:** Design approved (2026-06-30) · **Owner:** Keylink IT · **Client:** Pioneer Coach Buses
|
||||
**Target hardware:** GL.iNet GL-XE3000 (Puli AX) · OpenWrt 21.02-SNAPSHOT · MediaTek MT7981 (ARMv8) · kernel 5.4
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & Goal
|
||||
|
||||
Pioneer Coach Buses is replacing its **eyeride** solution with a fleet of GL-XE3000 routers (3 in the
|
||||
next month, **194 eventually**). Each bus needs **high-quality, highly stable internet** from multiple
|
||||
uplinks, with automatic, intelligent use of whichever links are healthy.
|
||||
|
||||
This project (Project 1) is the **on-router software**: the multi-WAN balancing brain plus the telemetry
|
||||
agent that later feeds the central fleet console (Project 2). It is a **port** of the existing Synology
|
||||
SRM package [`aiwanbal`](https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer) to OpenWrt — we reuse
|
||||
the *logic*, rewrite the *platform layer*.
|
||||
|
||||
### Scope decomposition (whole program)
|
||||
1. **Project 1 — On-router multi-WAN balancer (this spec)**
|
||||
2. Project 2 — Central fleet console (map, status, reporting, remote config, OTA) — *separate spec*
|
||||
3. Project 3 — Starlink deep integration & reporting — *folded partly into P1, extended in P2*
|
||||
4. Project 4 — Fleet provisioning / mass rollout of 194 units — *separate spec*
|
||||
|
||||
---
|
||||
|
||||
## 2. Hardware constraints (load-bearing)
|
||||
|
||||
- **Single 5G modem, dual-SIM *single-standby*.** The Quectel RM520N-GL has two SIM slots but only **one
|
||||
SIM is active at a time**. Therefore a single XE3000 **cannot** simultaneously balance T-Mobile *and*
|
||||
Verizon. Decision: run **one active cellular WAN** and treat the second SIM as **failover**.
|
||||
- **Ports:** 2.5 GbE WAN, 1 GbE LAN, USB 2.0, microSD. GNSS/GPS available via the modem.
|
||||
- **Upstream observed in the field unit:** cellular `rmnet_mhi0` on a **carrier CGNAT address**
|
||||
(e.g. `21.188.231.63/25`) — no public inbound, which is why remote management uses an **outbound**
|
||||
WireGuard tunnel (see §6).
|
||||
|
||||
---
|
||||
|
||||
## 3. Per-bus network design
|
||||
|
||||
| WAN | Link | Role |
|
||||
|-----|------|------|
|
||||
| **WAN1 — Cellular 5G** | internal modem, dual-SIM single-standby | Balanced. Auto-switch to the other SIM when the active carrier degrades past threshold. |
|
||||
| **WAN2 — Starlink** | 2.5 GbE WAN port → Starlink | Balanced. |
|
||||
|
||||
- **Normal state:** 5G + Starlink **balanced together** (2 live WANs) via `mwan3`.
|
||||
- **Degradation paths:** bad SIM → switch SIM (hysteresis); bad cellular overall → lean on Starlink;
|
||||
bad Starlink → lean on cellular. Never strand a bus offline.
|
||||
- **Downstream:** LAN → Omada fully-managed switch. Roof antenna provides 5G + GPS + Wi-Fi.
|
||||
|
||||
---
|
||||
|
||||
## 4. Platform approach
|
||||
|
||||
- **Base OS:** **GL.iNet stock firmware** (OpenWrt 21.02). We do **not** build a custom firmware image —
|
||||
this keeps GL's security/stability updates flowing and the Quectel modem fully supported.
|
||||
- **Delivery:** our software ships as a **branded `opkg` package + `init.d` service**, installed on top.
|
||||
Persists across reboots, OTA-updatable, survives GL firmware updates when done carefully.
|
||||
- **Balancing engine:** **`mwan3`** (OpenWrt standard weighted multi-WAN + failover). Our daemon tunes
|
||||
`mwan3` member weights instead of Synology SmartWAN slots / `srm.json`.
|
||||
- **Branding:** applied to the layer customers see — local dashboard, SSID/login identity, hostname, and
|
||||
the fleet console — **not** a full firmware rebrand.
|
||||
|
||||
### Reuse map (from `aiwanbal`)
|
||||
| Reused logic | Rewritten platform glue |
|
||||
|---|---|
|
||||
| 6-factor scoring + rolling average | SmartWAN weights → `mwan3` member weights |
|
||||
| 3-layer dead-gateway detection (DNS→ping quorum→HTTP) | SRM `srm.json`/`tctool` → `mwan3`/`tc` |
|
||||
| Adaptive, data-cap-aware speed testing (Ookla + curl) | `synopkg` lifecycle → `opkg` + `init.d` |
|
||||
| Modem signal monitoring (RSRP/RSRQ/SINR) | SRM modem drivers → GL modem manager / QMI / AT |
|
||||
| Starlink gRPC helper | SRM-embedded UI → LuCI/GL plugin |
|
||||
|
||||
---
|
||||
|
||||
## 5. Components (each independently testable)
|
||||
|
||||
1. **Scoring daemon** (`busrouter-daemon`) — polls each WAN's latency / throughput / jitter / loss /
|
||||
signal, computes the 6-factor score with rolling 3-sample average. Adapted from 2 generic WANs to
|
||||
**cellular + Starlink**. Writes desired weights.
|
||||
2. **WAN controller** (`busrouter-mwan`) — translates scores → `mwan3` member weights + failover; ports
|
||||
the 3-layer dead-gateway check. **Hysteresis** so weights don't flap; minimizes conntrack-flush resets.
|
||||
3. **SIM manager** (`busrouter-modem`) — reads RSRP/RSRQ/SINR via the GL modem manager; **auto-switches
|
||||
the active SIM** when the live carrier degrades past threshold, with cooldown/hysteresis to prevent
|
||||
ping-pong. ("Another 5G when signal gets bad.")
|
||||
4. **Starlink integration** (`busrouter-starlink`) — gRPC to the dish (status, obstruction, outage),
|
||||
feeding both scoring and telemetry.
|
||||
5. **Speed-test engine** (`busrouter-speedtest`) — Ookla CLI + curl fallback, adaptive interval,
|
||||
**data-cap-aware on cellular** (metered SIM skips auto-tests).
|
||||
6. **Telemetry agent** (`busrouter-agent`) — collects **GPS (modem GNSS)**, WAN states/scores, active SIM,
|
||||
signal, throughput, Starlink health, uptime; posts to the fleet hub over the **WireGuard tunnel**, and
|
||||
pulls remote config / OTA triggers. Defines the **contract** Project 2 consumes.
|
||||
7. **Branded local UI** — lightweight LuCI / GL-plugin status + diagnostics page (live WAN status, scores,
|
||||
signal, GPS, manual speed test, failover state). **Read-mostly** — real control is centralized — plus
|
||||
emergency local overrides for field techs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Management plane (provisioned 2026-06-30)
|
||||
|
||||
Remote access and telemetry ride a single **WireGuard hub**, which **doubles as the future fleet console
|
||||
host** (Project 2). Buses dial **out** as WG clients — CGNAT-safe.
|
||||
|
||||
```
|
||||
┌──────── WireGuard hub: kit-fleet-hub ────────┐
|
||||
│ DO droplet · 167.172.237.162 · nyc3 │
|
||||
│ s-1vcpu-1gb ($6/mo, resize as fleet grows) │
|
||||
│ wg0 10.88.0.1/24 · ListenPort 51820 │
|
||||
│ hub pubkey 4QAauZ9gS/wZ/u0Wf66OcJFGo4er... │
|
||||
└──────────────────────────────────────────────┘
|
||||
▲ ▲ ▲
|
||||
bus01 = 10.88.0.2 LAB (jump via hub) fleet console (later, same box)
|
||||
(GL-XE3000) ssh -J hub → router 10.88.0.1
|
||||
```
|
||||
|
||||
- **Per-bus WG client config** stored on the hub at `/root/busXX-wireguard.conf`, imported into GL's
|
||||
WireGuard client UI. `AllowedIPs = 10.88.0.0/24` only — **user internet stays on the balanced WANs**;
|
||||
the tunnel is pure out-of-band management.
|
||||
- **LAB access:** the LAB SSHes routers via the hub as a **jump host** (`ssh -J hub root@10.88.0.2`),
|
||||
key-based once the LAB public key is installed on the router.
|
||||
- **Local dev access:** local Claude Code on the operator PC deploys/tests over LAN (`192.168.8.1`) — no
|
||||
tunnel needed for the fast build/test loop.
|
||||
- **Division of labor:** local PC = build/deploy over LAN; LAB = remote SSH + management over WireGuard.
|
||||
|
||||
---
|
||||
|
||||
## 7. Config & error handling
|
||||
|
||||
- Master config under `/etc/busrouter/*.conf`, **persists across upgrades**.
|
||||
- **Fail-safe:** if our daemon dies, both WANs stay up via `mwan3` defaults — a software fault never takes
|
||||
a bus offline.
|
||||
- **Hysteresis** on both SIM-switching and weight changes; cooldown windows to damp oscillation.
|
||||
- Telemetry agent degrades gracefully when the hub is unreachable (buffer + retry); never blocks routing.
|
||||
|
||||
---
|
||||
|
||||
## 8. Device identity (provisioning defaults)
|
||||
|
||||
- SSID `X0000` / `Pioneer123` (2.4G + 5G), admin user `pioadmin` / `Pioneer321!`, branded hostname.
|
||||
- Captured as a **reproducible provisioning profile** so all units are identical (formalized in Project 4).
|
||||
- Router admin (current test unit): web `pioadmin` / `Pioneer321!`; SSH/root password `kitPLANE1!!`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Testing strategy
|
||||
|
||||
- **Unit:** shell logic (scoring, hysteresis, dead-gateway decision) tested in a VM / dev container.
|
||||
- **Bench:** single XE3000 test unit (`bus01`, WG `10.88.0.2`) — iterate via local LAN push (CGI/HTML hot,
|
||||
daemon needs service restart), mirroring `aiwanbal`'s live-push workflow.
|
||||
- **Field pilot:** 3 buses before scaling. Validate failover, SIM-switch, Starlink handoff, telemetry.
|
||||
|
||||
---
|
||||
|
||||
## 10. Out of scope (later projects)
|
||||
|
||||
Fleet console UI/map/reporting, OTA distribution infrastructure, mass provisioning of 194 units, Omada
|
||||
switch configuration, Starlink account/dish provisioning.
|
||||
|
||||
## 11. Decisions locked
|
||||
|
||||
- **Cellular:** one active SIM, the other as failover (no simultaneous dual-carrier). ✅
|
||||
- **Engine:** `mwan3`. ✅
|
||||
- **Base:** GL.iNet stock firmware + branded `opkg` layer (no custom firmware image). ✅
|
||||
- **Mgmt transport:** WireGuard, buses dial out to a shared hub that becomes the fleet console host. ✅
|
||||
- **Local UI:** branded, read-mostly; control centralized. ✅
|
||||
- **Hub sizing:** `s-1vcpu-1gb` now (~3 buses), resize up as the fleet grows. ✅
|
||||
Reference in New Issue
Block a user