kit-busrouter/docs/pilot/checklist.md

Three-Bus Field Pilot Validation Checklist

Plan ref: Task 9.1 / Phase 9 — Field pilot sign-off gate Units: bus01 (bench), bus02, bus03 (first three production units) Gate: All three buses must pass before scaling to the 194-unit fleet

---

Pre-flight (per unit, before boarding)

• ssh pioadmin@192.168.8.1 → password Pioneer321! succeeds
• cat /etc/busrouter/device-id returns correct ID (e.g. B0001)
• cat /etc/busrouter/version returns current package version
• WG tunnel up: wg show — peer handshake < 60 s ago
• mwan3 status — at least cellular online; wan offline is OK if no dish yet
• ps | grep daemon.sh — busrouter daemon running
• ls /tmp/busrouter/ — state files being written (metric_, score_, gps_last, etc.)
• Status page: http://192.168.8.1/busrouter/ loads in browser, shows live data
• SSID visible on both bands as <DEVICE_ID>, connects with Pioneer123

---

Section A — Dual-WAN balancing under motion

Scenario: Bus drives its route for ≥ 30 minutes with both cellular and Starlink active.

• A1: Both mwan3 members show "online" during route: mwan3 status from hub via WG
• A2: Weights shift based on scores (not stuck 50/50 throughout entire run)
  • Capture: cat /tmp/busrouter/score_modem_0001 and score_wan at start, midpoint, end
• A3: No LAN interruption detected by a ping flood from a laptop on bus WiFi
  • Command: ping -c 1800 8.8.8.8 (30 min, 1 s interval) — expect ≤ 2% loss total
• A4: /tmp/busrouter/metric_modem_0001 and metric_wan both updated each cycle
• A5: No busrouter daemon crash: logread | grep busrouter | grep -i error — 0 results

---

Section B — SIM auto-switch on real coverage change

Scenario: Drive through a known T-Mobile coverage gap or weak-signal zone (RSRP < −110 dBm for ≥ 3 consecutive cycles) with a second SIM in slot 2.

• B1: SIM slot switches from 1 → 2 when RSRP drops below threshold for SIM_SWITCH_CYCLES
  • Check: cat /tmp/busrouter/sim_slot changes from 1 to 2
• B2: Cooldown (300 s) is observed — no rapid oscillation between slots
  • Check: logread | grep sim_switch — only one event per coverage-gap pass
• B3: Cellular reconnects after slot switch
  • Check: mwan3 status shows modem_0001 returns online within ~30 s of switch
• B4: SIM slot correctly reported in telemetry (hub receives correct slot number)
• B5: Capture mid-test snapshot — logread | grep sim_switch and cat /tmp/busrouter/sim_slot immediately after slot change

---

Section C — Starlink handoff

Scenario: Trigger a Starlink→cellular handoff. Preferred: natural signal loss / obstruction event during the route. Acceptable fallback (bench unit or when no dish): ifdown wan / ifup wan — document which method was used in the sign-off record. If using fallback, note that this does not validate natural Starlink obstruction detection (requires grpcurl + real dish).

• C1: After ifdown wan, mwan3 shifts to modem_0001 (100%) within ≤ 15 s
  • Check: mwan3 status, logread | grep mwan3
• C2: LAN ping survives the failover (continue A3 ping through this test)
• C3: WG mgmt tunnel stays alive throughout — all ssh commands during failover succeed
  • Critical: WG must ride cellular when eth0 is down; verify ip rule still routes correctly
• C4: After ifup wan, mwan3 declares wan online and restores balanced policy within ≤ 30 s
• C5: Weight bar on status page http://192.168.8.1/busrouter/ updates correctly

---

Section D — Telemetry pipeline

Scenario: Run for ≥ 10 cycles with hub reachable, then simulate hub unreachable and confirm buffering, then restore hub and confirm flush.

• D1: Hub receives ≥ 5 telemetry payloads per bus while online
  • Check hub: journalctl -u busrouter-hub or inspect telemetry store for device_id entries
• D2: JSON payloads are valid (no parse errors at hub) — correct device_id, version fields
• D3: Buffer fills when hub unreachable
  • Simulate: set TELEMETRY_HUB=http://192.168.99.99 in daemon env and cycle 5 times
  • Check: ls /tmp/busrouter/telemetry-buf/*.json | wc -l → 5 files
• D4: Buffer flushes on hub restore
  • Restore TELEMETRY_HUB, trigger telemetry_send (or wait next cycle)
  • Check: ls /tmp/busrouter/telemetry-buf/ → empty; hub received the 5 buffered payloads
• D5: Buffer cap enforced — at 61st file, oldest is evicted (not new item rejected)
  • Check: create 60 dummy files in telemetry-buf/; run one cycle; count stays at 60

---

Section E — Failover must never drop LAN

Scenario: Stress test — bounce both uplinks repeatedly and confirm LAN clients never lose internet for more than the mwan3 detection window (~15 s).

• E1: Bounce cellular 3x (ifdown modem_0001; sleep 5; ifup modem_0001): LAN gap ≤ 15 s each
• E2: Bounce Starlink 3x (ifdown wan; sleep 5; ifup wan): LAN gap ≤ 15 s each
• E3: Simultaneously down both for 10 s, then restore: graceful "unreachable" state, no crash
• E4: After E3, both interfaces return online and balanced policy resumes automatically
• E5: After E1–E4, busrouter daemon still running — no segfault or restart in syslog

---

Section F — WireGuard mgmt plane integrity

Scenario: Verify the WG mgmt tunnel (10.88.0.X) survives every link-state change tested in Sections A–E. This is the "never lose the management plane" requirement.

• F1: Hub can SSH to router at root@10.88.0.X throughout all A–E tests
• F2: wg show wgclient1 latest handshake stays < 60 s during all tests
• F3: After cellular→eth0 underlay switch (eth0 comes up), WG remains live on new underlay
  • Check endpoint change at hub: wg show wg0 endpoints
• F4: After eth0→cellular underlay switch (eth0 goes down), WG stays live
  • Note: if eth0 is the WG underlay, do NOT down it without cellular as backup

---

Section G — Log capture + sign-off

For each bus, capture the following at end of pilot run and attach to the sign-off record:

# Run on router (or via hub WG tunnel):
logread > /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
mwan3 status >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
ip rule show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
wg show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
cat /tmp/busrouter/score_modem_0001 /tmp/busrouter/score_wan \
    /tmp/busrouter/sim_slot /tmp/busrouter/gps_last \
    >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
# Copy log off the router:
scp root@192.168.8.1:/tmp/pilot-log-*.txt ./logs/

Sign-off record

Log archive: copy logs to docs/pilot/logs/<DEVICE_ID>/ in this repo and commit after each bus.

Unit	Pilot date	Sections passed	C method	Issues found	Signed off by
bus01 (bench)	—	A B C D E F	synthetic	—	—
bus02	—	A B C D E F	natural/synthetic	—	—
bus03	—	A B C D E F	natural/synthetic	—	—

Fleet scale gate: all three rows have a signer in "Signed off by" → proceed to full 194-unit rollout using provision.sh + fleet automation (Project 4).

---

Known issues going into pilot

1. No Starlink dish on bench unit (bus01) — Section A/C Starlink tests require a dish on eth0 or an ATT Fiber substitute. Document which uplink was used for each Starlink-targeted test.
2. grpcurl absent — Starlink gRPC metrics (starlink.sh) are currently stubbed; Starlink score is based on echo/ping only. grpcurl binary must be bundled before Section C tests use starlink_last data for scoring.
3. GPS fix lag — First GPS fix after AT+QGPS=1 can take 1–5 minutes cold-start. Section A GPS checks should allow time for first fix before asserting gps_last is populated.
4. RSRP threshold tuning — SIM_SWITCH_RSRP=-110 is the default. Real coverage-gap zones may need adjustment; tune per route if Section B tests don't trigger naturally.