kit-busrouter/docs/pilot/checklist.md

# Three-Bus Field Pilot Validation Checklist

**Plan ref:** Task 9.1 / Phase 9 — Field pilot sign-off gate
**Units:** bus01 (bench), bus02, bus03 (first three production units)
**Gate:** All three buses must pass before scaling to the 194-unit fleet

---

## Pre-flight (per unit, before boarding)

- [ ] `ssh pioadmin@192.168.8.1` → password `Pioneer321!` succeeds
- [ ] `cat /etc/busrouter/device-id` returns correct ID (e.g. `B0001`)
- [ ] `cat /etc/busrouter/version` returns current package version
- [ ] WG tunnel up: `wg show` — peer handshake < 60 s ago
- [ ] `mwan3 status` — at least cellular online; wan offline is OK if no dish yet
- [ ] `ps | grep daemon.sh` — busrouter daemon running
- [ ] `ls /tmp/busrouter/` — state files being written (metric_*, score_*, gps_last, etc.)
- [ ] Status page: `http://192.168.8.1/busrouter/` loads in browser, shows live data
- [ ] SSID visible on both bands as `<DEVICE_ID>`, connects with `Pioneer123`

---

## Section A — Dual-WAN balancing under motion

**Scenario:** Bus drives its route for ≥ 30 minutes with both cellular and Starlink active.

- [ ] A1: Both mwan3 members show "online" during route: `mwan3 status` from hub via WG
- [ ] A2: Weights shift based on scores (not stuck 50/50 throughout entire run)
  - Capture: `cat /tmp/busrouter/score_modem_0001` and `score_wan` at start, midpoint, end
- [ ] A3: No LAN interruption detected by a ping flood from a laptop on bus WiFi
  - Command: `ping -c 1800 8.8.8.8` (30 min, 1 s interval) — expect ≤ 2% loss total
- [ ] A4: `/tmp/busrouter/metric_modem_0001` and `metric_wan` both updated each cycle
- [ ] A5: No busrouter daemon crash: `logread | grep busrouter | grep -i error` — 0 results

---

## Section B — SIM auto-switch on real coverage change

**Scenario:** Drive through a known T-Mobile coverage gap or weak-signal zone (RSRP < −110 dBm
for ≥ 3 consecutive cycles) with a second SIM in slot 2.

- [ ] B1: SIM slot switches from 1 → 2 when RSRP drops below threshold for SIM_SWITCH_CYCLES
  - Check: `cat /tmp/busrouter/sim_slot` changes from `1` to `2`
- [ ] B2: Cooldown (300 s) is observed — no rapid oscillation between slots
  - Check: `logread | grep sim_switch` — only one event per coverage-gap pass
- [ ] B3: Cellular reconnects after slot switch
  - Check: `mwan3 status` shows modem_0001 returns online within ~30 s of switch
- [ ] B4: SIM slot correctly reported in telemetry (hub receives correct slot number)
- [ ] B5: Capture mid-test snapshot — `logread | grep sim_switch` and `cat /tmp/busrouter/sim_slot` immediately after slot change

---

## Section C — Starlink handoff

**Scenario:** Trigger a Starlink→cellular handoff. Preferred: natural signal loss / obstruction
event during the route. Acceptable fallback (bench unit or when no dish): `ifdown wan` / `ifup wan`
— document which method was used in the sign-off record. If using fallback, note that this does
not validate natural Starlink obstruction detection (requires grpcurl + real dish).

- [ ] C1: After `ifdown wan`, mwan3 shifts to `modem_0001 (100%)` within ≤ 15 s
  - Check: `mwan3 status`, `logread | grep mwan3`
- [ ] C2: LAN ping survives the failover (continue A3 ping through this test)
- [ ] C3: WG mgmt tunnel stays alive throughout — all `ssh` commands during failover succeed
  - **Critical:** WG must ride cellular when eth0 is down; verify ip rule still routes correctly
- [ ] C4: After `ifup wan`, mwan3 declares wan online and restores balanced policy within ≤ 30 s
- [ ] C5: Weight bar on status page `http://192.168.8.1/busrouter/` updates correctly

---

## Section D — Telemetry pipeline

**Scenario:** Run for ≥ 10 cycles with hub reachable, then simulate hub unreachable and
confirm buffering, then restore hub and confirm flush.

- [ ] D1: Hub receives ≥ 5 telemetry payloads per bus while online
  - Check hub: `journalctl -u busrouter-hub` or inspect telemetry store for device_id entries
- [ ] D2: JSON payloads are valid (no parse errors at hub) — correct device_id, version fields
- [ ] D3: Buffer fills when hub unreachable
  - Simulate: set `TELEMETRY_HUB=http://192.168.99.99` in daemon env and cycle 5 times
  - Check: `ls /tmp/busrouter/telemetry-buf/*.json | wc -l` → 5 files
- [ ] D4: Buffer flushes on hub restore
  - Restore TELEMETRY_HUB, trigger `telemetry_send` (or wait next cycle)
  - Check: `ls /tmp/busrouter/telemetry-buf/` → empty; hub received the 5 buffered payloads
- [ ] D5: Buffer cap enforced — at 61st file, oldest is evicted (not new item rejected)
  - Check: create 60 dummy files in `telemetry-buf/`; run one cycle; count stays at 60

---

## Section E — Failover must never drop LAN

**Scenario:** Stress test — bounce both uplinks repeatedly and confirm LAN clients never lose
internet for more than the mwan3 detection window (~15 s).

- [ ] E1: Bounce cellular 3x (`ifdown modem_0001; sleep 5; ifup modem_0001`): LAN gap ≤ 15 s each
- [ ] E2: Bounce Starlink 3x (`ifdown wan; sleep 5; ifup wan`): LAN gap ≤ 15 s each
- [ ] E3: Simultaneously down both for 10 s, then restore: graceful "unreachable" state, no crash
- [ ] E4: After E3, both interfaces return online and balanced policy resumes automatically
- [ ] E5: After E1–E4, busrouter daemon still running — no segfault or restart in syslog

---

## Section F — WireGuard mgmt plane integrity

**Scenario:** Verify the WG mgmt tunnel (10.88.0.X) survives every link-state change tested
in Sections A–E. This is the "never lose the management plane" requirement.

- [ ] F1: Hub can SSH to router at `root@10.88.0.X` throughout all A–E tests
- [ ] F2: `wg show wgclient1` latest handshake stays < 60 s during all tests
- [ ] F3: After cellular→eth0 underlay switch (eth0 comes up), WG remains live on new underlay
  - Check endpoint change at hub: `wg show wg0 endpoints`
- [ ] F4: After eth0→cellular underlay switch (eth0 goes down), WG stays live
  - **Note:** if eth0 is the WG underlay, do NOT down it without cellular as backup

---

## Section G — Log capture + sign-off

For each bus, capture the following at end of pilot run and attach to the sign-off record:

```sh
# Run on router (or via hub WG tunnel):
logread > /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
mwan3 status >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
ip rule show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
wg show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
cat /tmp/busrouter/score_modem_0001 /tmp/busrouter/score_wan \
    /tmp/busrouter/sim_slot /tmp/busrouter/gps_last \
    >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
# Copy log off the router:
scp root@192.168.8.1:/tmp/pilot-log-*.txt ./logs/
```

### Sign-off record

**Log archive:** copy logs to `docs/pilot/logs/<DEVICE_ID>/` in this repo and commit after each bus.

| Unit | Pilot date | Sections passed | C method | Issues found | Signed off by |
|------|-----------|-----------------|----------|--------------|---------------|
| bus01 (bench) | — | A B C D E F | synthetic | — | — |
| bus02 | — | A B C D E F | natural/synthetic | — | — |
| bus03 | — | A B C D E F | natural/synthetic | — | — |

**Fleet scale gate:** all three rows have a signer in "Signed off by" → proceed to full 194-unit
rollout using `provision.sh` + fleet automation (Project 4).

---

## Known issues going into pilot

1. **No Starlink dish on bench unit (bus01)** — Section A/C Starlink tests require a dish on eth0
   or an ATT Fiber substitute. Document which uplink was used for each Starlink-targeted test.
2. **grpcurl absent** — Starlink gRPC metrics (`starlink.sh`) are currently stubbed; Starlink
   score is based on echo/ping only. `grpcurl` binary must be bundled before Section C tests
   use `starlink_last` data for scoring.
3. **GPS fix lag** — First GPS fix after `AT+QGPS=1` can take 1–5 minutes cold-start. Section A
   GPS checks should allow time for first fix before asserting `gps_last` is populated.
4. **RSRP threshold tuning** — `SIM_SWITCH_RSRP=-110` is the default. Real coverage-gap zones
   may need adjustment; tune per route if Section B tests don't trigger naturally.