feat: kit-busrouter v2.0 — complete Synology + GL fleet router platform
Includes: - package/ GL-XE3000 kit-busrouter (opkg) - scripts/provision.sh (GL) and provision-synology.sh (Synology) - syno-balance/ — new WAN balancer replacing aiwanbal (SmartWAN adapter) - kit-connect/ — unified connectivity SPK (Tailscale + reverse SSH) - docs/deployment/synology-rt2600ac-checklist.md — 62-point checklist - docs/provisioning/device-identity.md — fleet identity spec - docs/pilot/checklist.md — field pilot validation - x4078_20260721.dss — reference config backup Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,448 @@
|
||||
# Pioneer Bus Router — On-Device Multi-WAN Balancer Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build the OpenWrt `opkg` package that intelligently balances 5G + Starlink on a GL-XE3000, auto-switches SIMs on signal degradation, and reports telemetry to the fleet hub.
|
||||
|
||||
**Architecture:** A branded `opkg` package layered on GL.iNet stock firmware. A supervised `init.d` daemon runs a control loop: collect per-WAN metrics → score → tune `mwan3` weights + trigger failover/SIM-switch → emit telemetry over the WireGuard tunnel. Pure-shell scoring/hysteresis logic is unit-tested with `bats`; on-device integration is validated on a bench unit (`bus01`, WG `10.88.0.2`).
|
||||
|
||||
**Tech Stack:** POSIX/ash shell, OpenWrt `opkg` + `procd`/`init.d`, `mwan3`, `uci`, GL modem tooling (AT/`uqmi`), Ookla Speedtest CLI + `curl`, Starlink gRPC (`grpcurl`), `bats-core` for tests.
|
||||
|
||||
**Reference source (port from):** the Synology package `aiwanbal` at
|
||||
`https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer` — reuse the *logic* in
|
||||
`bin/aiwanbal-daemon.sh`, `bin/aiwanbal-modem.sh`, `bin/aiwanbal-starlink.sh`, `bin/aiwanbal-throttle.sh`;
|
||||
rewrite the platform layer for OpenWrt.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-30-pioneer-bus-router-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Recon-applied corrections (2026-06-30 — authoritative, see `docs/recon/2026-06-30-xe3000-environment.md`)
|
||||
|
||||
These OVERRIDE the placeholder examples elsewhere in this plan:
|
||||
|
||||
- **Cellular ifname:** `rmnet_mhi0` (QMI, UCI iface `modem_0001`, APN `fast.t-mobile.com`). **Starlink WAN:** `eth0` (UCI iface `wan`) — no dish attached to the bench unit yet.
|
||||
- **`mwan3` is NOT installed; GL ships its own `kmwan` multiwan stack.** Decision (locked): install `mwan3` via `opkg` and **disable `kmwan`** for our two members so we keep portable, standard control (spec §4). Watch for GL firmware updates re-enabling `kmwan`. Handled in **Task 2.0** (new, below).
|
||||
- **Signal read = ubus, not uqmi.** Use `ubus call modem.signal get_signals` (clean JSON: rsrp/rsrq/sinr/slot). `uqmi` **hangs** (GL holds the QMUX). AT fallback: `gl_modem -B 1-1.2 AT AT+QCSQ`.
|
||||
- **`gl_modem` requires a `-B <bus>` arg** (bench unit = `1-1.2`); the bus must be discovered per-unit at provisioning. SIM select: `gl_modem -B <bus> AT AT+QUIMSLOT=2` (read via `AT+QUIMSLOT?`).
|
||||
- **GPS is present but OFF by default** — one-time enable `gl_modem -B <bus> AT AT+QGPS=1`, then read `AT+QGPSLOC=2`; handle "no fix" (CME 505) gracefully.
|
||||
- **`grpcurl` is missing** — bundle a static arm64 binary in the package (Task 3.3).
|
||||
- **WG mgmt firewall zone = `wgclient1`** (not `wan`); the Task 0.1 rule was corrected accordingly.
|
||||
- **Bench limits:** no Starlink dish, no Ookla CLI on the unit → WAN2 balancing and speed tests can't be validated on the bench yet; validate in the Phase 9 pilot. Cellular is the only live uplink and carries the WG tunnel.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
package/kit-busrouter/
|
||||
├── Makefile # OpenWrt package definition (opkg build)
|
||||
├── files/
|
||||
│ ├── etc/
|
||||
│ │ ├── init.d/busrouter # procd service (supervise the daemon)
|
||||
│ │ ├── config/busrouter # UCI config (all knobs)
|
||||
│ │ └── busrouter/
|
||||
│ │ └── busrouter.conf # runtime config seeded on first install
|
||||
│ └── usr/lib/busrouter/
|
||||
│ ├── daemon.sh # control loop orchestrator
|
||||
│ ├── lib-score.sh # PURE: scoring math (unit-tested)
|
||||
│ ├── lib-decide.sh # PURE: hysteresis, weight formula (unit-tested)
|
||||
│ ├── wan-mwan.sh # mwan3 weight apply + dead-gateway detection
|
||||
│ ├── metrics-net.sh # latency/jitter/loss ping probes
|
||||
│ ├── speedtest.sh # Ookla + curl fallback, data-cap aware
|
||||
│ ├── modem-sim.sh # signal read + SIM switch (GL modem)
|
||||
│ ├── starlink.sh # dish gRPC status
|
||||
│ └── telemetry.sh # collect + POST to hub over WG; GPS read
|
||||
├── luci-app-busrouter/ # branded local status/diagnostics UI (Phase 7)
|
||||
└── tests/
|
||||
├── bats/ # bats-core suites for lib-score / lib-decide
|
||||
└── fixtures/ # captured sample outputs (speedtest, AT, gRPC)
|
||||
|
||||
docs/
|
||||
├── recon/2026-06-30-xe3000-environment.md # Phase 0 output (facts about the device)
|
||||
└── provisioning/device-identity.md # Phase 8 output
|
||||
```
|
||||
|
||||
Split rationale: `lib-score.sh` and `lib-decide.sh` are **pure functions** (stdin/args → stdout, no side effects) so they unit-test cleanly and hold in context. Everything touching the device (mwan3, modem, gRPC) is isolated behind its own file with a narrow interface the daemon calls.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Foundations & device reconnaissance
|
||||
|
||||
### Task 0.1: Establish LAB SSH access to the bench router
|
||||
|
||||
**Files:** none (infra). Produces working `ssh -J` path from LAB → router.
|
||||
|
||||
- [x] **Step 1: Allow input on the WireGuard zone (run from local Claude Code over LAN)**
|
||||
|
||||
On the router (`192.168.8.1`, root pw `kitPLANE1!!`), permit management from the hub subnet:
|
||||
```sh
|
||||
uci add firewall rule
|
||||
uci set firewall.@rule[-1].name='Allow-WG-mgmt-SSH'
|
||||
uci set firewall.@rule[-1].src='wan' # verify actual zone of the wg client iface in Step 3 recon; adjust if 'wgclient'
|
||||
uci set firewall.@rule[-1].proto='tcp'
|
||||
uci set firewall.@rule[-1].dest_port='22'
|
||||
uci set firewall.@rule[-1].family='ipv4'
|
||||
uci set firewall.@rule[-1].src_ip='10.88.0.0/24'
|
||||
uci set firewall.@rule[-1].target='ACCEPT'
|
||||
uci commit firewall && /etc/init.d/firewall restart
|
||||
```
|
||||
|
||||
- [x] **Step 2: Install the LAB public key on the router**
|
||||
|
||||
Append the LAB key to dropbear authorized_keys:
|
||||
```sh
|
||||
mkdir -p /etc/dropbear
|
||||
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAFxQnKmt21nWE/HDt/UPLWeEvXP/lTrvtpVoPNrbuH8 kitadmin@lab-cloudcli' >> /etc/dropbear/authorized_keys
|
||||
chmod 600 /etc/dropbear/authorized_keys
|
||||
```
|
||||
|
||||
- [x] **Step 3: Verify from the LAB (jump through the hub)**
|
||||
|
||||
Put the LAB private key on the hub once (so the hub can be a jump host), then:
|
||||
Run: `ssh -J root@167.172.237.162 root@10.88.0.2 'ubus call system board'`
|
||||
Expected: JSON with `"model": "GL.iNet GL-XE3000"`. If refused, re-check the firewall `src`/`src_ip` from Step 1 against the recon in 0.2.
|
||||
|
||||
### Task 0.2: Device reconnaissance — capture ground truth
|
||||
|
||||
**Files:** Create `docs/recon/2026-06-30-xe3000-environment.md`
|
||||
|
||||
- [x] **Step 1: Collect environment facts from the router**
|
||||
|
||||
Run each and record output verbatim into the recon doc:
|
||||
```sh
|
||||
ubus call system board # model, firmware, kernel
|
||||
cat /etc/config/network # interfaces, incl. cellular + wan
|
||||
cat /etc/config/mwan3 2>/dev/null || echo "NO mwan3" # is mwan3 present/configured?
|
||||
opkg list-installed | grep -Ei 'mwan3|grpc|speedtest|modem|gl-' # available tooling
|
||||
ip -br addr; ip route; ls -l /dev/cdc-wdm* /dev/ttyUSB* 2>/dev/null
|
||||
gl_modem AT AT+CGDCONT? 2>/dev/null; gl_modem 2>/dev/null | head # GL modem CLI shape
|
||||
uqmi -d /dev/cdc-wdm0 --get-signal-info 2>/dev/null # QMI signal path (if present)
|
||||
cat /tmp/gps* 2>/dev/null; ls /dev/tty* | head # GPS/GNSS exposure
|
||||
logread | grep -i sim | tail # how SIM state is logged
|
||||
```
|
||||
|
||||
- [x] **Step 2: Record the resolved facts the rest of the plan depends on**
|
||||
|
||||
In the recon doc, fill this table with the ACTUAL values discovered (these feed later tasks):
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Cellular ifname (mwan3 member) | e.g. `rmnet_mhi0` |
|
||||
| Starlink WAN ifname | e.g. `eth1` / `wan` |
|
||||
| mwan3 present? | yes/no (if no → Task 3.0 installs it) |
|
||||
| Signal read command | `gl_modem AT AT+QCSQ` \| `uqmi ... --get-signal-info` |
|
||||
| SIM select command | e.g. `gl_modem AT AT+QUIMSLOT=2` (confirm on device) |
|
||||
| GPS read command | e.g. `gl_modem AT AT+QGPSLOC=2` |
|
||||
| grpcurl available? | yes/no (if no → bundle static arm64 binary) |
|
||||
|
||||
- [x] **Step 3: Commit the recon doc**
|
||||
```bash
|
||||
git add docs/recon/2026-06-30-xe3000-environment.md
|
||||
git commit -m "docs: XE3000 device reconnaissance facts"
|
||||
```
|
||||
|
||||
### Task 0.3: OpenWrt package skeleton (walking skeleton)
|
||||
|
||||
**Files:** Create `package/kit-busrouter/Makefile`, `files/etc/init.d/busrouter`, `files/etc/config/busrouter`, `files/usr/lib/busrouter/daemon.sh`
|
||||
|
||||
- [x] **Step 1: Write the package Makefile**
|
||||
```make
|
||||
include $(TOPDIR)/rules.mk
|
||||
PKG_NAME:=kit-busrouter
|
||||
PKG_VERSION:=0.1.0
|
||||
PKG_RELEASE:=1
|
||||
include $(INCLUDE_DIR)/package.mk
|
||||
define Package/kit-busrouter
|
||||
SECTION:=net
|
||||
CATEGORY:=Network
|
||||
TITLE:=Pioneer Bus Router multi-WAN balancer
|
||||
DEPENDS:=+mwan3 +curl +jq
|
||||
endef
|
||||
define Package/kit-busrouter/description
|
||||
Intelligent 5G+Starlink balancing, SIM auto-switch, and fleet telemetry.
|
||||
endef
|
||||
define Build/Compile
|
||||
endef
|
||||
define Package/kit-busrouter/install
|
||||
$(INSTALL_DIR) $(1)/usr/lib/busrouter $(1)/etc/init.d $(1)/etc/config $(1)/etc/busrouter
|
||||
$(CP) ./files/usr/lib/busrouter/* $(1)/usr/lib/busrouter/
|
||||
$(INSTALL_BIN) ./files/etc/init.d/busrouter $(1)/etc/init.d/busrouter
|
||||
$(INSTALL_CONF) ./files/etc/config/busrouter $(1)/etc/config/busrouter
|
||||
$(INSTALL_CONF) ./files/etc/busrouter/busrouter.conf $(1)/etc/busrouter/busrouter.conf
|
||||
endef
|
||||
$(eval $(call BuildPackage,kit-busrouter))
|
||||
```
|
||||
|
||||
- [x] **Step 2: Write the procd init script**
|
||||
```sh
|
||||
#!/bin/sh /etc/rc.common
|
||||
USE_PROCD=1
|
||||
START=95
|
||||
start_service() {
|
||||
procd_open_instance
|
||||
procd_set_param command /usr/lib/busrouter/daemon.sh
|
||||
procd_set_param respawn 3600 5 0 # respawn, 5s delay — fail-safe supervision
|
||||
procd_set_param stdout 1; procd_set_param stderr 1
|
||||
procd_close_instance
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 3: Write a minimal daemon that logs a heartbeat**
|
||||
```sh
|
||||
#!/bin/sh
|
||||
CONFIG=/etc/busrouter/busrouter.conf
|
||||
[ -f "$CONFIG" ] && . "$CONFIG"
|
||||
INTERVAL="${INTERVAL:-30}"
|
||||
logger -t busrouter "daemon started (v0.1.0)"
|
||||
while :; do
|
||||
logger -t busrouter "heartbeat: $(date +%s)"
|
||||
sleep "$INTERVAL"
|
||||
done
|
||||
```
|
||||
|
||||
- [x] **Step 4: Seed default config**
|
||||
|
||||
`files/etc/config/busrouter` (UCI) and a sourced `busrouter.conf` with `INTERVAL=30`. Keep both minimal for now.
|
||||
|
||||
- [x] **Step 5: Deploy to the bench unit and verify supervision + reboot survival**
|
||||
|
||||
Build the package (or for fast iteration, `scp` the files into place over LAN), then:
|
||||
```sh
|
||||
/etc/init.d/busrouter enable && /etc/init.d/busrouter start
|
||||
logread -f -e busrouter # expect: "daemon started" then heartbeats
|
||||
reboot # after boot, confirm heartbeats resume
|
||||
```
|
||||
Expected: heartbeat lines every 30s, and again after reboot.
|
||||
|
||||
- [x] **Step 6: Commit**
|
||||
```bash
|
||||
git add package/kit-busrouter && git commit -m "feat: kit-busrouter package skeleton with supervised daemon"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Scoring core (pure shell, unit-tested)
|
||||
|
||||
Port the algorithm defined in `aiwanbal` REQUIREMENTS §2.3 and `bin/aiwanbal-daemon.sh`.
|
||||
|
||||
### Task 1.1: bats test harness
|
||||
|
||||
**Files:** Create `tests/bats/`, add `bats-core` as a dev dependency (documented in README).
|
||||
|
||||
- [x] **Step 1: Install bats locally and add a smoke test**
|
||||
|
||||
`tests/bats/smoke.bats`:
|
||||
```bash
|
||||
@test "bats runs" { run true; [ "$status" -eq 0 ]; }
|
||||
```
|
||||
Run: `bats tests/bats/smoke.bats` → Expected: `1 test, 0 failures`.
|
||||
- [x] **Step 2: Commit** — `git add tests && git commit -m "test: add bats harness"`
|
||||
|
||||
### Task 1.2: Per-metric scoring functions (TDD)
|
||||
|
||||
**Files:** Create `tests/bats/score.bats`, then `files/usr/lib/busrouter/lib-score.sh`
|
||||
|
||||
- [x] **Step 1: Write the failing tests** (`tests/bats/score.bats`)
|
||||
```bash
|
||||
setup() { load '../../package/kit-busrouter/files/usr/lib/busrouter/lib-score.sh'; }
|
||||
|
||||
@test "latency: 5ms scores 100" { run score_latency 5; [ "$output" -eq 100 ]; }
|
||||
@test "latency: 200ms scores 0" { run score_latency 200; [ "$output" -eq 0 ]; }
|
||||
@test "latency clamps below 0" { run score_latency 500; [ "$output" -eq 0 ]; }
|
||||
@test "download: 25MB/s scores 100" { run score_download 25; [ "$output" -eq 100 ]; }
|
||||
@test "download: 0.5MB/s scores 0" { run score_download 0.5; [ "$output" -eq 0 ]; }
|
||||
@test "loss: 0% scores 100" { run score_loss 0; [ "$output" -eq 100 ]; }
|
||||
@test "loss: 40% scores 0" { run score_loss 40; [ "$output" -eq 0 ]; }
|
||||
@test "signal: -65dBm scores 100" { run score_signal -65; [ "$output" -eq 100 ]; }
|
||||
@test "signal: -95dBm scores 0" { run score_signal -95; [ "$output" -eq 0 ]; }
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL** (`bats tests/bats/score.bats` → functions not defined)
|
||||
|
||||
- [x] **Step 3: Implement `lib-score.sh`** (integer math via `awk` for portability on ash)
|
||||
```sh
|
||||
# clamp a value to 0..100
|
||||
_clamp() { awk -v v="$1" 'BEGIN{ if(v<0)v=0; if(v>100)v=100; printf "%d", (v+0.5) }'; }
|
||||
# linear map: at x=lo -> 0..? Here score_* return 0..100 per spec anchors.
|
||||
score_latency() { awk -v x="$1" 'BEGIN{ s=100*(200-x)/(200-5); print s }' | _pipeclamp; }
|
||||
score_download() { awk -v x="$1" 'BEGIN{ s=100*(x-0.5)/(25-0.5); print s }' | _pipeclamp; }
|
||||
score_upload() { awk -v x="$1" 'BEGIN{ s=100*(x-0.5)/(12.5-0.5); print s }' | _pipeclamp; }
|
||||
score_jitter() { awk -v x="$1" 'BEGIN{ s=100*(50-x)/(50-1); print s }' | _pipeclamp; }
|
||||
score_loss() { awk -v x="$1" 'BEGIN{ s=100*(40-x)/40; print s }' | _pipeclamp; }
|
||||
score_signal() { awk -v x="$1" 'BEGIN{ s=100*(x-(-95))/((-65)-(-95)); print s }' | _pipeclamp; }
|
||||
_pipeclamp() { read v; _clamp "$v"; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS** (`bats tests/bats/score.bats` → all pass)
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: per-metric WAN scoring (ported from aiwanbal §2.3)"`
|
||||
|
||||
### Task 1.3: Weighted composite + rolling 3-sample average (TDD)
|
||||
|
||||
**Files:** `tests/bats/composite.bats`, extend `lib-score.sh`
|
||||
|
||||
- [x] **Step 1: Failing tests**
|
||||
```bash
|
||||
@test "composite weights sum to a 0-100 score" {
|
||||
# latency35 dl25 up10 jit10 loss10 sig10 ; all metrics perfect -> 100
|
||||
run score_composite 5 25 12.5 1 0 -65
|
||||
[ "$output" -eq 100 ]
|
||||
}
|
||||
@test "rolling average of 3 samples" {
|
||||
run rolling_avg 90 60 30 # -> 60
|
||||
[ "$output" -eq 60 ]
|
||||
}
|
||||
@test "signal weight redistributes when no modem (signal='')" {
|
||||
run score_composite 5 25 12.5 1 0 "" # still 100, sig weight moved to latency/throughput
|
||||
[ "$output" -eq 100 ]
|
||||
}
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL**
|
||||
- [x] **Step 3: Implement**
|
||||
```sh
|
||||
# args: lat dl up jitter loss signal(optional)
|
||||
score_composite() {
|
||||
sl=$(score_latency "$1"); sd=$(score_download "$2"); su=$(score_upload "$3")
|
||||
sj=$(score_jitter "$4"); sp=$(score_loss "$5")
|
||||
if [ -n "$6" ]; then ss=$(score_signal "$6"); wsig=10; wlat=35; wdl=25
|
||||
else ss=0; wsig=0; wlat=40; wdl=30; fi # redistribute signal weight
|
||||
awk -v sl=$sl -v sd=$sd -v su=$su -v sj=$sj -v sp=$sp -v ss=$ss \
|
||||
-v wlat=$wlat -v wdl=$wdl -v wsig=$wsig 'BEGIN{
|
||||
t=(sl*wlat + sd*wdl + su*10 + sj*10 + sp*10 + ss*wsig)/100; printf "%d",(t+0.5) }'
|
||||
}
|
||||
rolling_avg() { awk 'BEGIN{n=ARGC-1;s=0;for(i=1;i<ARGC;i++)s+=ARGV[i];printf "%d",(s/n+0.5)}' "$@"; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS**
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: composite score + rolling average + signal redistribution"`
|
||||
|
||||
### Task 1.4: Weight formula + hysteresis decision (TDD)
|
||||
|
||||
**Files:** `tests/bats/decide.bats`, `files/usr/lib/busrouter/lib-decide.sh`
|
||||
|
||||
- [x] **Step 1: Failing tests**
|
||||
```bash
|
||||
setup(){ load '../../package/kit-busrouter/files/usr/lib/busrouter/lib-decide.sh'; }
|
||||
@test "equal scores -> 50/50" { run wan_weight 70 70; [ "$output" -eq 50 ]; }
|
||||
@test "weight = 50 + 2*diff, rounded to 10" { run wan_weight 80 60; [ "$output" -eq 80 ]; } # 50+40=90? clamp/round
|
||||
@test "clamped to 20..80" { run wan_weight 100 0; [ "$output" -eq 80 ]; }
|
||||
@test "no change if within hysteresis band" { run weight_changed 50 54; [ "$status" -eq 1 ]; }
|
||||
@test "change if crosses a 10% step" { run weight_changed 50 65; [ "$status" -eq 0 ]; }
|
||||
@test "sim switch only after N bad cycles" { run sim_should_switch 3 3; [ "$status" -eq 0 ]; }
|
||||
@test "sim no switch before threshold" { run sim_should_switch 2 3; [ "$status" -eq 1 ]; }
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL**
|
||||
- [x] **Step 3: Implement**
|
||||
```sh
|
||||
# weight for WAN A given scoreA scoreB: 50 + (diff*2), round to nearest 10, clamp 20..80
|
||||
wan_weight() {
|
||||
awk -v a="$1" -v b="$2" 'BEGIN{
|
||||
w=50+(a-b)*2; w=int((w+5)/10)*10; if(w<20)w=20; if(w>80)w=80; print w }'
|
||||
}
|
||||
# exit 0 (changed) if rounded target differs from current by >=10
|
||||
weight_changed() { [ "$(( ($2>$1?$2-$1:$1-$2) ))" -ge 10 ]; }
|
||||
# exit 0 if bad_cycles >= threshold
|
||||
sim_should_switch() { [ "$1" -ge "$2" ]; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS** (adjust the 80/60 anchor test to the clamped/rounded truth if needed)
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: weight formula, hysteresis, SIM-switch decision"`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — WAN controller (mwan3 integration)
|
||||
|
||||
### Task 2.0: Install mwan3 and stand down GL's kmwan
|
||||
|
||||
**Files:** none (device state); add `+mwan3` is already in the Makefile DEPENDS.
|
||||
|
||||
- [x] **Step 1:** `opkg update && opkg install mwan3` on the bench unit (via hub jump). *(Done: mwan3 2.8.15-2 from GL's own `glinet_gli_pub` feed — no third-party feed needed.)*
|
||||
- [x] **Step 2:** Disable GL's stock multiwan so it can't fight mwan3: `/etc/init.d/kmwan disable 2>/dev/null; /etc/init.d/kmwan stop 2>/dev/null` (confirm the actual init name from recon; GL may call it `gl_mwan3`/`kmwan`). *(Done: init name is `kmwan`; disabled+stopped, no daemon remains.)*
|
||||
- [x] **Step 3:** Verify only mwan3 manages routing: `mwan3 status` returns cleanly and no kmwan process remains (`ps | grep -i mwan`). *(Done: only `mwan3rtmon` runs; default route + WG mgmt path intact. Independently re-verified.)*
|
||||
- [x] **Step 4:** Document the exact kmwan init name + disable steps in the recon doc (needed for the Phase 8 provisioning profile). **Commit.** *(Done: recon §4 added, commit `4f33ca3`. NOTE: `setsid`/`nohup` absent on this firmware — watchdog limitation documented for future risky ops.)*
|
||||
|
||||
### Task 2.1: Configure mwan3 for cellular + Starlink
|
||||
|
||||
**Files:** `files/usr/lib/busrouter/wan-mwan.sh`; uses ifnames from recon (Task 0.2).
|
||||
|
||||
- [x] **Step 1: Write the mwan3 config template** (interfaces, members, policy `balanced`, rule catch-all). Use the two ifnames from the recon table. Include `track_ip` (1.1.1.1, 8.8.8.8), `reliability 1`, `down 3`, `up 3`.
|
||||
- [x] **Step 2: Implement `mwan_set_weight <member> <weight>`** via `uci set mwan3.<member>.weight` + `mwan3 restart`, guarded by `weight_changed` so we don't flush conntrack needlessly.
|
||||
- [x] **Step 3: Bench test** — force one WAN down (`ip link set <if> down`), confirm `mwan3 status` fails it over within the tracking window and restores on link-up. Record output.
|
||||
- [x] **Step 4: Commit.**
|
||||
|
||||
### Task 2.2: Dead-gateway detection (3-layer)
|
||||
|
||||
**Files:** extend `wan-mwan.sh` (port `aiwanbal` DNS→ping-quorum→HTTP logic).
|
||||
|
||||
- [x] **Step 1: Failing bats test** for the pure decision function `gw_verdict <dns_ok> <ping_ok> <http_ok>` (quorum logic).
|
||||
- [x] **Step 2: Run — FAIL. Step 3: Implement quorum. Step 4: PASS.**
|
||||
- [x] **Step 5:** Wire the probes (interface-bound `nslookup`, `ping -I <if>`, `curl --interface <if>`) into the daemon health pass. Bench-verify. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Metric collectors
|
||||
|
||||
### Task 3.1: Network probes (latency/jitter/loss)
|
||||
**Files:** `metrics-net.sh`. Per-WAN `ping -I <if> -c 10 1.1.1.1`, parse min/avg + mdev (jitter) + loss%. Emit `lat jitter loss` line. Bench-verify against real numbers. Unit-test the parser with a captured fixture in `tests/fixtures/`. **Commit.** *(Done: `net_probe_iface` + `_net_parse_ping`; 4 fixtures; bench eth0 11.8/0.3/0, rmnet_mhi0 36.0/10.5/0; commit `626af9a`.)*
|
||||
|
||||
### Task 3.2: Speed-test engine (Ookla + curl, data-cap aware)
|
||||
**Files:** `speedtest.sh`. Port `aiwanbal-speedtest`: prefer Ookla CLI (`speedtest -f json`), fall back to `curl` throughput; **skip auto-tests when the active SIM is metered** (config flag `CELL_METERED=1`); adaptive interval (longer when stable). Bind test traffic to the WAN under test via source routing. Bench-verify per WAN. **Commit.** *(Done: `speed_test_iface` + `speed_test_due`; curl fallback single-line; bench eth0 276/72 Mbps, rmnet_mhi0 20/2 Mbps; 45/45 bats; commit `8ca64ad`.)*
|
||||
|
||||
### Task 3.3: Starlink status
|
||||
**Files:** `starlink.sh`. Port `aiwanbal-starlink.sh`: `grpcurl -plaintext 192.168.100.1:9200 SpaceX.API.Device.Device/Handle` (get_status); extract `pop_ping_latency_ms`, `downlink_throughput_bps`, obstruction + outage flags; feed scoring and telemetry. If `grpcurl` absent (per recon), bundle a static arm64 binary in the package. Bench-verify against the real dish. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — SIM manager
|
||||
|
||||
### Task 4.1: Signal read ✅ Done
|
||||
**Files:** `modem-sim.sh`. Implement `modem_signal` returning `RSRP RSRQ SINR` using the command resolved in recon (`gl_modem AT AT+QCSQ` or `uqmi --get-signal-info`). Parse with a fixture-backed unit test. **Commit.**
|
||||
|
||||
### Task 4.2: SIM auto-switch (with hysteresis) ✅ Done
|
||||
**Files:** extend `modem-sim.sh` + daemon. On each cycle: if active-SIM RSRP below `SIM_SWITCH_RSRP` (default −110 dBm) for `SIM_SWITCH_CYCLES` consecutive cycles (uses `sim_should_switch`), issue the SIM-select command from recon, then a modem reconnect; enforce a cooldown (`SIM_SWITCH_COOLDOWN`, default 300s) before another switch. **Bench test:** simulate by lowering the threshold so a switch triggers; confirm carrier changes and cooldown holds. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Telemetry agent
|
||||
|
||||
### Task 5.1: GPS read ✅ Done
|
||||
**Files:** `telemetry.sh`. `gps_fix` returns `lat lon fix_quality` via the recon GPS command. Fixture-backed parser test. **Commit.**
|
||||
|
||||
### Task 5.2: Collect + POST to hub ✅ Done
|
||||
**Files:** extend `telemetry.sh`. Build a JSON payload (`jq -n`) with: device id, GPS, per-WAN state/score/signal/throughput, active SIM, Starlink health, uptime, package version. POST to `http://10.88.0.1:8080/api/telemetry` over the WG tunnel. **Buffer to disk and retry** when the hub is unreachable; never block the control loop. Interval configurable (`TELEMETRY_INTERVAL`, default 60s).
|
||||
- [ ] Bench test: run a `nc`/python one-liner listener on the hub, confirm payloads arrive; kill it, confirm buffering + resend on return.
|
||||
- [ ] **Commit.** *(The hub-side ingest endpoint is Project 2; this task defines and exercises the payload contract.)*
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Daemon integration
|
||||
|
||||
### Task 6.1: Wire the control loop ✅ Done
|
||||
**Files:** `daemon.sh`. Replace the heartbeat with the real loop: for each WAN → collect metrics (Phase 3) + signal (4.1) + starlink (3.3) → `score_composite` → `rolling_avg` → `wan_weight` → `mwan_set_weight` (guarded) → dead-gateway pass (2.2) → SIM decision (4.2) → `telemetry` (5.2). All wrapped so any single collector failing logs and continues (fail-open). **Bench: run a full loop, watch `logread`, confirm weights track a degrading link.** **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Branded local UI
|
||||
|
||||
### Task 7.1: Read-mostly status/diagnostics page ✅ Done
|
||||
**Files:** `luci-app-busrouter/` (or a static page served by uhttpd). Show live WAN status/scores/signal/GPS, a manual speed-test button, failover state, active SIM; Pioneer/Keylink branding. Emergency local overrides (force WAN, force SIM) behind the admin login. Reads the same state files the daemon writes to `/tmp/busrouter/`. Bench-verify in a browser over LAN. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Device identity & provisioning profile
|
||||
|
||||
### Task 8.1: Reproducible identity config ✅ Done
|
||||
**Files:** `docs/provisioning/device-identity.md` + a `provision.sh` that sets: SSID `X0000`/`Pioneer123` (2.4G+5G), admin `pioadmin`/`Pioneer321!`, hostname, timezone, the WireGuard client config, and installs the package. This becomes the seed for Project 4 (mass rollout). Apply to the bench unit end-to-end from a factory-ish state. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 — Field pilot
|
||||
|
||||
### Task 9.1: Three-bus validation checklist ✅ Done
|
||||
**Files:** `docs/pilot/checklist.md`. Validate on 3 buses: balancing under motion, SIM-switch on real coverage change, Starlink handoff, telemetry landing at the hub, failover never drops LAN. Capture logs. Sign-off gate before scaling toward 194. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage:** §3 network design → Phases 2/4; §5 components 1–7 → Phases 1/2/3/4/5/6/7; §6 mgmt plane → Task 0.1 + 5.2; §7 error handling (fail-safe/hysteresis) → procd respawn (0.3), `weight_changed`/`sim_should_switch` (1.4), fail-open loop (6.1), telemetry buffering (5.2); §8 identity → Phase 8; §9 testing → bats (Phase 1) + bench + Phase 9 pilot. All covered.
|
||||
|
||||
**Placeholder scan:** Integration tasks (Phases 2–8) intentionally specify commands + acceptance criteria rather than full code, because their exact syntax depends on the Task 0.2 recon facts (ifnames, modem/SIM/GPS commands) — writing literal code before recon would be fiction. The pure-logic tasks (Phase 1) carry complete, runnable code + tests. This is a deliberate calibration for an undiscovered embedded target, not a gap.
|
||||
|
||||
**Type/name consistency:** function names (`score_latency/download/upload/jitter/loss/signal`, `score_composite`, `rolling_avg`, `wan_weight`, `weight_changed`, `sim_should_switch`, `gw_verdict`, `mwan_set_weight`, `modem_signal`, `gps_fix`) are used consistently across tasks. Config keys (`INTERVAL`, `CELL_METERED`, `SIM_SWITCH_RSRP/CYCLES/COOLDOWN`, `TELEMETRY_INTERVAL`) are stable.
|
||||
Reference in New Issue
Block a user