feat: kit-busrouter v2.0 — complete Synology + GL fleet router platform
Includes: - package/ GL-XE3000 kit-busrouter (opkg) - scripts/provision.sh (GL) and provision-synology.sh (Synology) - syno-balance/ — new WAN balancer replacing aiwanbal (SmartWAN adapter) - kit-connect/ — unified connectivity SPK (Tailscale + reverse SSH) - docs/deployment/synology-rt2600ac-checklist.md — 62-point checklist - docs/provisioning/device-identity.md — fleet identity spec - docs/pilot/checklist.md — field pilot validation - x4078_20260721.dss — reference config backup Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,158 @@
|
||||
# Three-Bus Field Pilot Validation Checklist
|
||||
|
||||
**Plan ref:** Task 9.1 / Phase 9 — Field pilot sign-off gate
|
||||
**Units:** bus01 (bench), bus02, bus03 (first three production units)
|
||||
**Gate:** All three buses must pass before scaling to the 194-unit fleet
|
||||
|
||||
---
|
||||
|
||||
## Pre-flight (per unit, before boarding)
|
||||
|
||||
- [ ] `ssh pioadmin@192.168.8.1` → password `Pioneer321!` succeeds
|
||||
- [ ] `cat /etc/busrouter/device-id` returns correct ID (e.g. `B0001`)
|
||||
- [ ] `cat /etc/busrouter/version` returns current package version
|
||||
- [ ] WG tunnel up: `wg show` — peer handshake < 60 s ago
|
||||
- [ ] `mwan3 status` — at least cellular online; wan offline is OK if no dish yet
|
||||
- [ ] `ps | grep daemon.sh` — busrouter daemon running
|
||||
- [ ] `ls /tmp/busrouter/` — state files being written (metric_*, score_*, gps_last, etc.)
|
||||
- [ ] Status page: `http://192.168.8.1/busrouter/` loads in browser, shows live data
|
||||
- [ ] SSID visible on both bands as `<DEVICE_ID>`, connects with `Pioneer123`
|
||||
|
||||
---
|
||||
|
||||
## Section A — Dual-WAN balancing under motion
|
||||
|
||||
**Scenario:** Bus drives its route for ≥ 30 minutes with both cellular and Starlink active.
|
||||
|
||||
- [ ] A1: Both mwan3 members show "online" during route: `mwan3 status` from hub via WG
|
||||
- [ ] A2: Weights shift based on scores (not stuck 50/50 throughout entire run)
|
||||
- Capture: `cat /tmp/busrouter/score_modem_0001` and `score_wan` at start, midpoint, end
|
||||
- [ ] A3: No LAN interruption detected by a ping flood from a laptop on bus WiFi
|
||||
- Command: `ping -c 1800 8.8.8.8` (30 min, 1 s interval) — expect ≤ 2% loss total
|
||||
- [ ] A4: `/tmp/busrouter/metric_modem_0001` and `metric_wan` both updated each cycle
|
||||
- [ ] A5: No busrouter daemon crash: `logread | grep busrouter | grep -i error` — 0 results
|
||||
|
||||
---
|
||||
|
||||
## Section B — SIM auto-switch on real coverage change
|
||||
|
||||
**Scenario:** Drive through a known T-Mobile coverage gap or weak-signal zone (RSRP < −110 dBm
|
||||
for ≥ 3 consecutive cycles) with a second SIM in slot 2.
|
||||
|
||||
- [ ] B1: SIM slot switches from 1 → 2 when RSRP drops below threshold for SIM_SWITCH_CYCLES
|
||||
- Check: `cat /tmp/busrouter/sim_slot` changes from `1` to `2`
|
||||
- [ ] B2: Cooldown (300 s) is observed — no rapid oscillation between slots
|
||||
- Check: `logread | grep sim_switch` — only one event per coverage-gap pass
|
||||
- [ ] B3: Cellular reconnects after slot switch
|
||||
- Check: `mwan3 status` shows modem_0001 returns online within ~30 s of switch
|
||||
- [ ] B4: SIM slot correctly reported in telemetry (hub receives correct slot number)
|
||||
- [ ] B5: Capture mid-test snapshot — `logread | grep sim_switch` and `cat /tmp/busrouter/sim_slot` immediately after slot change
|
||||
|
||||
---
|
||||
|
||||
## Section C — Starlink handoff
|
||||
|
||||
**Scenario:** Trigger a Starlink→cellular handoff. Preferred: natural signal loss / obstruction
|
||||
event during the route. Acceptable fallback (bench unit or when no dish): `ifdown wan` / `ifup wan`
|
||||
— document which method was used in the sign-off record. If using fallback, note that this does
|
||||
not validate natural Starlink obstruction detection (requires grpcurl + real dish).
|
||||
|
||||
- [ ] C1: After `ifdown wan`, mwan3 shifts to `modem_0001 (100%)` within ≤ 15 s
|
||||
- Check: `mwan3 status`, `logread | grep mwan3`
|
||||
- [ ] C2: LAN ping survives the failover (continue A3 ping through this test)
|
||||
- [ ] C3: WG mgmt tunnel stays alive throughout — all `ssh` commands during failover succeed
|
||||
- **Critical:** WG must ride cellular when eth0 is down; verify ip rule still routes correctly
|
||||
- [ ] C4: After `ifup wan`, mwan3 declares wan online and restores balanced policy within ≤ 30 s
|
||||
- [ ] C5: Weight bar on status page `http://192.168.8.1/busrouter/` updates correctly
|
||||
|
||||
---
|
||||
|
||||
## Section D — Telemetry pipeline
|
||||
|
||||
**Scenario:** Run for ≥ 10 cycles with hub reachable, then simulate hub unreachable and
|
||||
confirm buffering, then restore hub and confirm flush.
|
||||
|
||||
- [ ] D1: Hub receives ≥ 5 telemetry payloads per bus while online
|
||||
- Check hub: `journalctl -u busrouter-hub` or inspect telemetry store for device_id entries
|
||||
- [ ] D2: JSON payloads are valid (no parse errors at hub) — correct device_id, version fields
|
||||
- [ ] D3: Buffer fills when hub unreachable
|
||||
- Simulate: set `TELEMETRY_HUB=http://192.168.99.99` in daemon env and cycle 5 times
|
||||
- Check: `ls /tmp/busrouter/telemetry-buf/*.json | wc -l` → 5 files
|
||||
- [ ] D4: Buffer flushes on hub restore
|
||||
- Restore TELEMETRY_HUB, trigger `telemetry_send` (or wait next cycle)
|
||||
- Check: `ls /tmp/busrouter/telemetry-buf/` → empty; hub received the 5 buffered payloads
|
||||
- [ ] D5: Buffer cap enforced — at 61st file, oldest is evicted (not new item rejected)
|
||||
- Check: create 60 dummy files in `telemetry-buf/`; run one cycle; count stays at 60
|
||||
|
||||
---
|
||||
|
||||
## Section E — Failover must never drop LAN
|
||||
|
||||
**Scenario:** Stress test — bounce both uplinks repeatedly and confirm LAN clients never lose
|
||||
internet for more than the mwan3 detection window (~15 s).
|
||||
|
||||
- [ ] E1: Bounce cellular 3x (`ifdown modem_0001; sleep 5; ifup modem_0001`): LAN gap ≤ 15 s each
|
||||
- [ ] E2: Bounce Starlink 3x (`ifdown wan; sleep 5; ifup wan`): LAN gap ≤ 15 s each
|
||||
- [ ] E3: Simultaneously down both for 10 s, then restore: graceful "unreachable" state, no crash
|
||||
- [ ] E4: After E3, both interfaces return online and balanced policy resumes automatically
|
||||
- [ ] E5: After E1–E4, busrouter daemon still running — no segfault or restart in syslog
|
||||
|
||||
---
|
||||
|
||||
## Section F — WireGuard mgmt plane integrity
|
||||
|
||||
**Scenario:** Verify the WG mgmt tunnel (10.88.0.X) survives every link-state change tested
|
||||
in Sections A–E. This is the "never lose the management plane" requirement.
|
||||
|
||||
- [ ] F1: Hub can SSH to router at `root@10.88.0.X` throughout all A–E tests
|
||||
- [ ] F2: `wg show wgclient1` latest handshake stays < 60 s during all tests
|
||||
- [ ] F3: After cellular→eth0 underlay switch (eth0 comes up), WG remains live on new underlay
|
||||
- Check endpoint change at hub: `wg show wg0 endpoints`
|
||||
- [ ] F4: After eth0→cellular underlay switch (eth0 goes down), WG stays live
|
||||
- **Note:** if eth0 is the WG underlay, do NOT down it without cellular as backup
|
||||
|
||||
---
|
||||
|
||||
## Section G — Log capture + sign-off
|
||||
|
||||
For each bus, capture the following at end of pilot run and attach to the sign-off record:
|
||||
|
||||
```sh
|
||||
# Run on router (or via hub WG tunnel):
|
||||
logread > /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
mwan3 status >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
ip rule show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
wg show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
cat /tmp/busrouter/score_modem_0001 /tmp/busrouter/score_wan \
|
||||
/tmp/busrouter/sim_slot /tmp/busrouter/gps_last \
|
||||
>> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
# Copy log off the router:
|
||||
scp root@192.168.8.1:/tmp/pilot-log-*.txt ./logs/
|
||||
```
|
||||
|
||||
### Sign-off record
|
||||
|
||||
**Log archive:** copy logs to `docs/pilot/logs/<DEVICE_ID>/` in this repo and commit after each bus.
|
||||
|
||||
| Unit | Pilot date | Sections passed | C method | Issues found | Signed off by |
|
||||
|------|-----------|-----------------|----------|--------------|---------------|
|
||||
| bus01 (bench) | — | A B C D E F | synthetic | — | — |
|
||||
| bus02 | — | A B C D E F | natural/synthetic | — | — |
|
||||
| bus03 | — | A B C D E F | natural/synthetic | — | — |
|
||||
|
||||
**Fleet scale gate:** all three rows have a signer in "Signed off by" → proceed to full 194-unit
|
||||
rollout using `provision.sh` + fleet automation (Project 4).
|
||||
|
||||
---
|
||||
|
||||
## Known issues going into pilot
|
||||
|
||||
1. **No Starlink dish on bench unit (bus01)** — Section A/C Starlink tests require a dish on eth0
|
||||
or an ATT Fiber substitute. Document which uplink was used for each Starlink-targeted test.
|
||||
2. **grpcurl absent** — Starlink gRPC metrics (`starlink.sh`) are currently stubbed; Starlink
|
||||
score is based on echo/ping only. `grpcurl` binary must be bundled before Section C tests
|
||||
use `starlink_last` data for scoring.
|
||||
3. **GPS fix lag** — First GPS fix after `AT+QGPS=1` can take 1–5 minutes cold-start. Section A
|
||||
GPS checks should allow time for first fix before asserting `gps_last` is populated.
|
||||
4. **RSRP threshold tuning** — `SIM_SWITCH_RSRP=-110` is the default. Real coverage-gap zones
|
||||
may need adjustment; tune per route if Section B tests don't trigger naturally.
|
||||
Reference in New Issue
Block a user