feat: kit-busrouter v2.0 — complete Synology + GL fleet router platform
Includes: - package/ GL-XE3000 kit-busrouter (opkg) - scripts/provision.sh (GL) and provision-synology.sh (Synology) - syno-balance/ — new WAN balancer replacing aiwanbal (SmartWAN adapter) - kit-connect/ — unified connectivity SPK (Tailscale + reverse SSH) - docs/deployment/synology-rt2600ac-checklist.md — 62-point checklist - docs/provisioning/device-identity.md — fleet identity spec - docs/pilot/checklist.md — field pilot validation - x4078_20260721.dss — reference config backup Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,238 @@
|
||||
# Synology RT2600ac — Fleet Deployment Checklist
|
||||
|
||||
**Version:** 1.1 · **Last updated:** 2026-07-21 · **Audited against:** x4078 DSS config backup
|
||||
|
||||
Every Synology RT2600ac bus router MUST pass ALL sections before departing the yard. Items flagged 🔴 were missed on x4078 and caused remote-access emergencies.
|
||||
|
||||
---
|
||||
|
||||
## 0. Pre-flight (before touching SRM)
|
||||
|
||||
- [ ] **0.1** Insert SD card into router (required for Package Center and third-party packages)
|
||||
- [ ] **0.2** Power on router; connect LAN port to a PC for setup
|
||||
- [ ] **0.3** Verify default LAN IP `192.168.1.1` is reachable
|
||||
|
||||
---
|
||||
|
||||
## 1. SRM First-Time Setup Wizard
|
||||
|
||||
- [ ] **1.1** 🔴 **Admin password:** `kitPLANE1!!` (user: `kitadmin`)
|
||||
- [ ] **1.2** 🔴 **Timezone:** `(GMT-06:00) Central Time (US & Canada)` — Chicago, CST/CDT
|
||||
- NOT UTC, NOT Central America (Mexico City)
|
||||
- [ ] **1.3** 🔴 **WiFi Main SSID:** `<DEVICE_ID> Pioneer Admin` (e.g. `x4078 Pioneer Admin`)
|
||||
- Security: WPA2-PSK · Password: `Pioneer321!`
|
||||
- Smart Connect: **ON** (single SSID for both bands)
|
||||
- Do NOT add `_2.4G` or `_5G` suffixes to the SSID
|
||||
- [ ] **1.4** 🔴 **WiFi Guest SSID:** `<DEVICE_ID> Pioneer` (e.g. `x4078 Pioneer`)
|
||||
- Security: WPA2-PSK · Password: `Pioneer123`
|
||||
- Client isolation: ON
|
||||
- Smart Connect: ON
|
||||
- Delete any default `SynologyGuest_*` networks
|
||||
- [ ] **1.5** **LAN subnet:** Keep default `192.168.1.0/24` (lbr0)
|
||||
- [ ] **1.6** **Guest subnet:** `192.168.2.0/24` (gbr0, SRM default)
|
||||
- [ ] **1.7** **Hostname:** Set to `<DEVICE_ID>` (e.g. `x4078`)
|
||||
- [ ] **1.8** **Router login port:** Keep default HTTP 8000 / HTTPS 8001
|
||||
|
||||
---
|
||||
|
||||
## 2. SRM Control Panel — System Settings
|
||||
|
||||
- [ ] **2.1** 🔴 **QuickConnect:** Enable · ID = `<DEVICE_ID>` (e.g. `x4078`)
|
||||
- Sign in with Synology Account (create `pioneercoach@keylinkit.net` if needed)
|
||||
- Verify: `https://<DEVICE_ID>.us2.quickconnect.to/webman/index.cgi` loads
|
||||
- [ ] **2.2** 🔴 **DSM Auto-Update:** Control Panel → System → Update & Restore → Update Settings
|
||||
- **Automatically check for updates:** ON
|
||||
- **Automatically install important updates:** ON
|
||||
- **Update schedule:** Daily, off-peak (e.g. 03:00)
|
||||
- [ ] **2.3** 🔴 **Database auto-update:** Security Advisor → Settings → Auto-update definitions
|
||||
- [ ] **2.4** 🔴 **Notifications:** Control Panel → Notification → Enable push/email notifications
|
||||
- Enable: System health, WAN status changes, package events
|
||||
- Email: configure SMTP or use SRM notification relay
|
||||
- [ ] **2.5** **SSH:** Control Panel → Terminal & SNMP → Enable SSH on port 2223
|
||||
- [ ] **2.6** **Regional Options:** Verify language, time format, and date format
|
||||
|
||||
---
|
||||
|
||||
## 3. SmartWAN Configuration
|
||||
|
||||
- [ ] **3.1** Network Center → Smart WAN → **Load Balancing with Failover** mode
|
||||
- [ ] **3.2** 🔴 **WAN1 (eth0):** Primary — this will be Eyeride Eyenet
|
||||
- [ ] **3.3** 🔴 **WAN2 (eth2):** Secondary — this will be Starlink
|
||||
- [ ] **3.4** **Health check targets:** `1.1.1.2`, `9.9.9.9`, `8.8.8.8`
|
||||
- Cloudflare malware-filtering (1.1.1.2) + Quad9 (9.9.9.9) + Google (8.8.8.8 fallback)
|
||||
- [ ] **3.5** **Policy routing:** All traffic balanced; no manual pinning yet
|
||||
|
||||
---
|
||||
|
||||
## 4. Package Center Setup
|
||||
|
||||
- [ ] **4.1** Open Package Center → Accept terms of service when prompted
|
||||
- [ ] **4.2** Package Center → Settings → Trust Level → **"Any publisher"**
|
||||
- [ ] **4.3** Verify SD card is recognized (Package Center → Storage)
|
||||
|
||||
---
|
||||
|
||||
## 5. aiwanbal Package Install
|
||||
|
||||
- [ ] **5.1** Download `aiwanbal-1.3-0013.spk` from Gitea:
|
||||
```
|
||||
https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer/releases/download/v1.3-0013/aiwanbal-1.3-0013.spk
|
||||
```
|
||||
- [ ] **5.2** SCP to router: `scp -O -P 2223 aiwanbal-*.spk kitadmin@192.168.1.1:/tmp/`
|
||||
- [ ] **5.3** Install via SSH: `echo "kitPLANE1!!" | sudo -S synopkg install /tmp/aiwanbal-*.spk`
|
||||
- [ ] **5.4** Start: `echo "kitPLANE1!!" | sudo -S synopkg start aiwanbal`
|
||||
- [ ] **5.5** 🔴 **Configure for mobile/bus environment:**
|
||||
```sh
|
||||
# In /etc/aiwanbal/aiwanbal.conf, set:
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf ENVIRONMENT mobile
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf SPEEDTEST_AUTO_ENABLED 0
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf DEVICE_NAME x4078
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf CHECK_INTERVAL 30
|
||||
```
|
||||
- [ ] **5.6** 🔴 **Set Eyeride GPS config** (if Eyeride password is available):
|
||||
```sh
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_ENABLE 1
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_IP 192.168.10.1
|
||||
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_PASSWORD <eyeride-password>
|
||||
```
|
||||
- [ ] **5.7** 🔴 **Verify ISP detection:** Check the aiwanbal UI at `/webman/3rdparty/aiwanbal/index.html`
|
||||
- WAN1 should show carrier/ISP (Eyeride → T-Mobile/Verizon depending on SIM)
|
||||
- WAN2 should show Starlink
|
||||
- If not detecting, run auto-detection: `sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth0; sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth2`
|
||||
|
||||
---
|
||||
|
||||
## 6. kittunnel Package Install (Reverse SSH Tunnel)
|
||||
|
||||
- [ ] **6.1** Assign a unique port for this bus (see fleet port registry)
|
||||
- [ ] **6.2** Download hardened kittunnel SPK (v1.1+) from Gitea
|
||||
- [ ] **6.3** SCP to router and install via synopkg
|
||||
- [ ] **6.4** Verify tunnel.conf has correct `REMOTE_PORT`
|
||||
- [ ] **6.5** Start: `sudo synopkg start kittunnel`
|
||||
- [ ] **6.6** Verify tunnel is connected: check port is LISTENING on VPS (162.243.83.36)
|
||||
- [ ] **6.7** Verify end-to-end: `ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"`
|
||||
|
||||
---
|
||||
|
||||
## 7. Tailscale Install (Management Tailnet)
|
||||
|
||||
- [ ] **7.1** Download Tailscale static `linux_arm` binary
|
||||
- [ ] **7.2** Install to `/usr/local/bin/tailscale` and `/usr/local/bin/tailscaled`
|
||||
- [ ] **7.3** Start tailscaled in userspace mode:
|
||||
```sh
|
||||
/usr/bin/setsid /usr/local/bin/tailscaled \
|
||||
--statedir=/var/packages/Tailscale/var/state \
|
||||
--tun=userspace-networking \
|
||||
--socket=/var/packages/Tailscale/var/run/tailscaled.sock &
|
||||
```
|
||||
- [ ] **7.4** 🔴 Join bus fleet tailnet:
|
||||
```sh
|
||||
tailscale up --auth-key tskey-auth-... --hostname <DEVICE_ID> --accept-routes=false --accept-dns=false
|
||||
```
|
||||
- [ ] **7.5** 🔴 Install rc.local with **correct paths** (`/usr/bin/setsid`, `/bin/mkdir`):
|
||||
```sh
|
||||
echo '#!/bin/sh' | sudo tee /usr/syno/etc/rc.d/rc.local
|
||||
echo '/bin/mkdir -p /var/packages/Tailscale/var/state /var/packages/Tailscale/var/run' | sudo tee -a /usr/syno/etc/rc.d/rc.local
|
||||
echo '/usr/bin/setsid /usr/local/bin/tailscaled --statedir=/var/packages/Tailscale/var/state --tun=userspace-networking --socket=/var/packages/Tailscale/var/run/tailscaled.sock &' | sudo tee -a /usr/syno/etc/rc.d/rc.local
|
||||
sudo chmod +x /usr/syno/etc/rc.d/rc.local
|
||||
```
|
||||
- [ ] **7.6** 🔴 **Reboot test:** `sudo reboot` → wait 2 min → verify tailscaled is running
|
||||
- [ ] **7.7** Verify Tailscale status shows router online at `pioneercoach@`
|
||||
|
||||
---
|
||||
|
||||
## 8. Device Identity & Hostname (Persistent)
|
||||
|
||||
- [ ] **8.1** 🔴 Create `/etc/busrouter/device-id` with `<DEVICE_ID>`:
|
||||
```sh
|
||||
echo "<DEVICE_ID>" | sudo tee /etc/busrouter/device-id
|
||||
```
|
||||
- [ ] **8.2** 🔴 Create `/etc/busrouter/version` with `0.0`:
|
||||
```sh
|
||||
echo "0.0" | sudo tee /etc/busrouter/version
|
||||
```
|
||||
- [ ] **8.3** 🔴 **Persistent hostname** — set in ALL locations:
|
||||
```sh
|
||||
sudo hostname <DEVICE_ID>
|
||||
echo "<DEVICE_ID>" | sudo tee /etc/hostname
|
||||
```
|
||||
Also check SRM UI: Control Panel → Network → Router Name
|
||||
- [ ] **8.4** 🔴 **DHCP hostname:** SRM → Network Center → Internet → WAN → edit → Hostname = `<DEVICE_ID>`
|
||||
(Without this, DHCP sends `SynologyRouter` to upstream)
|
||||
|
||||
---
|
||||
|
||||
## 9. Telemetry Agent
|
||||
|
||||
- [ ] **9.1** SCP `telemetry-synology.sh` to `/usr/lib/busrouter/telemetry-synology.sh`
|
||||
- [ ] **9.2** Configure `TELEMETRY_HUB` to fleet hub endpoint:
|
||||
```sh
|
||||
# In the script or via environment: http://167.172.237.162:8080/api/telemetry
|
||||
```
|
||||
- [ ] **9.3** 🔴 Add cron job (every 60s):
|
||||
```sh
|
||||
echo '* * * * * root TELEMETRY_HUB=http://167.172.237.162:8080/api/telemetry /usr/lib/busrouter/telemetry-synology.sh 2>/dev/null' | sudo tee -a /etc/crontab
|
||||
```
|
||||
- [ ] **9.4** Dry-run test: `sudo /usr/lib/busrouter/telemetry-synology.sh`
|
||||
|
||||
---
|
||||
|
||||
## 10. Reboot & Final Verification
|
||||
|
||||
- [ ] **10.1** Reboot: `sudo reboot`
|
||||
- [ ] **10.2** After reboot, verify ALL of these survive:
|
||||
|
||||
| Check | Command | Expected |
|
||||
|-------|---------|----------|
|
||||
| Hostname | `hostname` | `<DEVICE_ID>` |
|
||||
| Device ID | `cat /etc/busrouter/device-id` | `<DEVICE_ID>` |
|
||||
| aiwanbal | `sudo synopkg status aiwanbal` | started |
|
||||
| kittunnel | `sudo synopkg status kittunnel` | started |
|
||||
| Tailscale | `ps \| grep tailscaled` | running |
|
||||
| Tailscale status | `tailscale status` | online, pioneercoach@ |
|
||||
| WiFi SSID | Check phone WiFi scan | `<DEVICE_ID> Pioneer Admin` |
|
||||
| Guest WiFi | Check phone WiFi scan | `<DEVICE_ID> Pioneer` |
|
||||
| QuickConnect | Browser → `https://<DEVICE_ID>.us2.quickconnect.to` | SRM login loads |
|
||||
| SSH tunnel | From lab: `ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"` | `<DEVICE_ID>` |
|
||||
| Tailscale path | From lab: `ssh -J root@172.18.0.1 -p 2223 kitadmin@<tailscale-ip> "hostname"` | `<DEVICE_ID>` |
|
||||
| WAN1 carrier | aiwanbal UI | Shows ISP |
|
||||
| WAN2 carrier | aiwanbal UI | Shows Starlink |
|
||||
| Telemetry cron | `grep telemetry /etc/crontab` | present |
|
||||
| Auto-update | SRM UI → Update & Restore | Auto-check ON |
|
||||
| Notifications | SRM UI → Notification | Enabled |
|
||||
|
||||
- [ ] **10.3** 🔴 **Run the reboot test before the bus leaves the yard.** There is no second chance.
|
||||
- [ ] **10.4** Store credentials in vault under `busrouter/<DEVICE_ID>`
|
||||
- [ ] **10.5** Record device in fleet registry with IPs, port, keys
|
||||
|
||||
---
|
||||
|
||||
## Fleet Port Registry
|
||||
|
||||
| Device | VPS Port | Tailscale IP | QuickConnect URL |
|
||||
|--------|----------|-------------|-------------------|
|
||||
| x4078 | 2226 | `100.116.71.43` | `https://x4078.us2.quickconnect.to` |
|
||||
| x5925 | 2230 | TBD | `https://x5925.us2.quickconnect.to` |
|
||||
| *(next)* | 2231 | TBD | TBD |
|
||||
| *(next)* | 2231 | TBD | TBD |
|
||||
|
||||
---
|
||||
|
||||
## x4078 Audit (2026-07-21) — Findings from DSS backup
|
||||
|
||||
🔴 **Critical — must fix remotely or on next physical access:**
|
||||
|
||||
| Gap | Severity | Fix |
|
||||
|-----|----------|-----|
|
||||
| SSID: `x4078 Admin` → should be `x4078 Pioneer Admin` | Medium | SRM Wi-Fi settings |
|
||||
| Guest SSID: `x4078 Pioneer Guest` → should be `x4078 Pioneer` | Medium | SRM Wi-Fi settings |
|
||||
| Band SSIDs have `_2.4G`/`_5G` suffixes | Low | Disable per-band SSIDs, use SmartConnect only |
|
||||
| Old `SynologyGuest_*` networks still present | Low | Delete in SRM Wi-Fi settings |
|
||||
| Timezone was Central America, not CST/CDT (Chicago) | High | Fixed via QuickConnect SRM UI |
|
||||
| Database auto-update OFF | High | Fixed via QuickConnect SRM UI |
|
||||
| Notifications not enabled | High | Fixed via QuickConnect SRM UI |
|
||||
| DHCP hostname sends `SynologyRouter` | Medium | SRM Network Center → WAN → Hostname |
|
||||
| Hostname persistence failing on reboot | High | Verify `/etc/hostname` + SRM Network Center |
|
||||
| ISP/carrier not detecting per WAN | Medium | Run `aiwanbal-detect.sh` on each WAN |
|
||||
| rc.local has wrong `/bin/setsid` path → `/usr/bin/setsid` | High | Fixed already |
|
||||
| Tailscale not boot-tested | Critical | Must test on next physical access |
|
||||
@@ -0,0 +1,171 @@
|
||||
# Pioneer Bus Router — Device Identity & Provisioning (Synology SRM)
|
||||
|
||||
**Last updated:** 2026-07-21
|
||||
**Target hardware:** Synology RT2600ac, SRM 1.3.1+
|
||||
|
||||
---
|
||||
|
||||
## 1. Fleet-wide constants
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| WiFi SSID (2.4 + 5 GHz) | `<DEVICE_ID>` — SSID = device ID (e.g. `B0042`) |
|
||||
| WiFi password | `Pioneer123` |
|
||||
| Admin SSH user | `root` (enabled via SRM Control Panel → Terminal & SNMP) |
|
||||
| Admin password | `Pioneer321!` |
|
||||
| Timezone | `UTC` |
|
||||
| Cellular modem | Eyeride Eyenet (OpenWrt) at `192.168.10.1:8080` |
|
||||
| WAN balancer | aiwanbal (SPK package), environment=**mobile** |
|
||||
| VPN transport | **Tailscale** (not WireGuard) |
|
||||
| Fleet hub Tailscale IP | `100.76.128.1` (the hub's Tailscale address) |
|
||||
| Telemetry endpoint | `http://<HUB_TAILSCALE_IP>:8080/api/telemetry` |
|
||||
| Telemetry interval | 60 s (via cron) |
|
||||
| Telemetry buffer dir | `/tmp/busrouter/telemetry-buf/` (60-file cap) |
|
||||
| SSH jump key | `busfleet-hub-jump` (ed25519) |
|
||||
|
||||
**Platform identifier:** `"platform":"synology"` in telemetry JSON payloads.
|
||||
|
||||
---
|
||||
|
||||
## 2. Per-device parameters
|
||||
|
||||
| DEVICE_ID | Tailscale IP | Eyeride IP | Notes |
|
||||
|-----------|-------------|------------|-------|
|
||||
| `bus02` | *(auto)* | `192.168.10.1` | bench unit |
|
||||
| `B0002` | *(auto)* | `192.168.10.1` | first production Synology unit |
|
||||
|
||||
Tailscale IPs are auto-assigned by the coordination server (100.x.y.z range).
|
||||
|
||||
---
|
||||
|
||||
## 3. Provisioning procedure
|
||||
|
||||
### Prerequisites
|
||||
- SRM 1.3.1+ with SSH enabled (Control Panel → Terminal & SNMP → Enable SSH)
|
||||
- SD card inserted
|
||||
- LAN access at `192.168.1.1`
|
||||
- `aiwanbal-latest.spk` at `/tmp/` on the router
|
||||
- Tailscale SPK installed
|
||||
- Tailscale auth key generated
|
||||
- Eyeride Eyenet reachable at `192.168.10.1`
|
||||
|
||||
### Steps
|
||||
|
||||
```sh
|
||||
# 1. Build + copy files
|
||||
cd /home/node/workspace/aiwanbal && bash build.sh
|
||||
scp -O scripts/provision-synology.sh \
|
||||
package/kit-busrouter/files/usr/lib/busrouter/telemetry-synology.sh \
|
||||
kitadmin@192.168.1.1:/tmp/
|
||||
scp -O aiwanbal-*.spk kitadmin@192.168.1.1:/tmp/aiwanbal-latest.spk
|
||||
|
||||
# 2. SSH in and run
|
||||
ssh kitadmin@192.168.1.1
|
||||
sudo -i
|
||||
chmod +x /tmp/provision-synology.sh /tmp/telemetry-synology.sh
|
||||
/tmp/provision-synology.sh B0042 "tskey-auth-xxxx"
|
||||
# Reboots after 5 s.
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
ssh root@<router-ip>
|
||||
cat /etc/busrouter/device-id # B0042
|
||||
synopkg status aiwanbal # "started"
|
||||
tailscale status # "active"
|
||||
grep telemetry /etc/crontab # cron job present
|
||||
/usr/lib/busrouter/telemetry-synology.sh # dry-run
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture differences: GL vs Synology
|
||||
|
||||
| Component | GL-XE3000 (OpenWrt) | Synology RT2600ac (SRM) |
|
||||
|-----------|--------------------|--------------------------|
|
||||
| WAN balancer | kit-busrouter + mwan3 | aiwanbal + SRM SmartWAN |
|
||||
| State dir | `/tmp/busrouter/` | `/tmp/aiwanbal/` |
|
||||
| VPN tunnel | WireGuard (`wgclient1`) | Tailscale (`tailscale0`) |
|
||||
| Hub IP | `10.88.0.1` (WG) | `100.76.128.1` (Tailscale) |
|
||||
| Telemetry trigger | daemon loop (30s) | cron (60s) |
|
||||
| GPS source | Quectel AT commands | Eyeride ubus JSON-RPC |
|
||||
| Modem signal | Quectel ubus | aiwanbal-modem.sh drivers |
|
||||
| Starlink | gRPC to dish | aiwanbal-starlink.sh |
|
||||
| Package mgmt | opkg | synopkg |
|
||||
| Init system | procd | start-stop-status |
|
||||
| Identity file | `/etc/busrouter/device-id` | `/etc/busrouter/device-id` *(same)* |
|
||||
|
||||
---
|
||||
|
||||
## 5. Telemetry JSON schema
|
||||
|
||||
```json
|
||||
{
|
||||
"device_id": "B0042",
|
||||
"timestamp": 1737500000,
|
||||
"uptime": 86400,
|
||||
"version": "1.0",
|
||||
"platform": "synology",
|
||||
"sel_primary": "wan1",
|
||||
"gps": {"lat": 40.7128, "lon": -74.0060, "fix": 2},
|
||||
"wan": {
|
||||
"wan1": {"score": 85, "latency_ms": 15, "dl_mbps": 50.0, "ul_mbps": 10.0,
|
||||
"jitter_ms": 2, "loss_pct": 0, "rsrp_dbm": -80, "rsrq_db": -12,
|
||||
"sinr_db": 8, "technology": "5G-NR", "carrier": "Eyeride"},
|
||||
"wan2": {"score": 70, "latency_ms": 25, "dl_mbps": 30.0, "ul_mbps": 5.0,
|
||||
"jitter_ms": 5, "loss_pct": 1, "rsrp_dbm": -90, "rsrq_db": -15,
|
||||
"sinr_db": 3, "technology": "LTE", "carrier": "Peplink"}
|
||||
},
|
||||
"starlink": {"quality": 0, "latency_ms": null, "obstructed": 0, "outage": 0}
|
||||
}
|
||||
```
|
||||
|
||||
### Field mapping: aiwanbal state → telemetry JSON
|
||||
|
||||
| Telemetry field | aiwanbal state file | Notes |
|
||||
|----------------|--------------------|-------|
|
||||
| `wan.*.score` | `wanN_score_avg` | 3-sample rolling average |
|
||||
| `wan.*.dl_mbps` | `wanN_throughput` | B/s → Mbps |
|
||||
| `wan.*.ul_mbps` | `wanN_upload` | B/s → Mbps |
|
||||
| `wan.*.latency_ms` | `wanN_latency` | ping probe result |
|
||||
| `wan.*.jitter_ms` | `wanN_jitter` | ping jitter |
|
||||
| `wan.*.loss_pct` | `wanN_packet_loss` | % |
|
||||
| `wan.*.rsrp_dbm` | `wanN_rsrp` | cellular signal |
|
||||
| `wan.*.technology` | `wanN_technology` | "5G-NR", "LTE", etc. |
|
||||
| `wan.*.carrier` | `wanN_modem_type` | mapped to name |
|
||||
| `sel_primary` | `wan0_mode` + states | "wan1" or "wan2" |
|
||||
| `gps.*` | Eyeride ubus | null if unavailable |
|
||||
| `starlink.*` | `wan0_starlink_*` | null if no dish |
|
||||
|
||||
### Hub-side: GL vs Synology payloads
|
||||
|
||||
| Field | GL value | Synology value |
|
||||
|-------|---------|---------------|
|
||||
| `platform` | absent or `"openwrt"` | `"synology"` |
|
||||
| `wan` keys | `modem_0001`, `wan` | `wan1`, `wan2` |
|
||||
| `sim_slot` | present (1 or 2) | absent |
|
||||
|
||||
The hub should handle both schemas.
|
||||
|
||||
---
|
||||
|
||||
## 6. Checklist — new Synology device before field deployment
|
||||
|
||||
- [ ] Device ID assigned and recorded in fleet registry
|
||||
- [ ] SD card inserted and mounted
|
||||
- [ ] SRM updated to latest 1.3.1+
|
||||
- [ ] SSH enabled
|
||||
- [ ] Tailscale SPK installed and authenticated
|
||||
- [ ] aiwanbal SPK built and installed
|
||||
- [ ] aiwanbal configured: ENVIRONMENT=mobile, SPEEDTEST_AUTO=0
|
||||
- [ ] Eyeride Eyenet reachable at `192.168.10.1:8080`
|
||||
- [ ] `provision-synology.sh` run end-to-end; device rebooted cleanly
|
||||
- [ ] WiFi visible as `<DEVICE_ID>`, pw `Pioneer123`
|
||||
- [ ] SSH to `root@<router-ip>` succeeds
|
||||
- [ ] `synopkg status aiwanbal` shows "started"
|
||||
- [ ] `tailscale status` shows "active"
|
||||
- [ ] `/etc/busrouter/device-id` correct
|
||||
- [ ] Telemetry cron job present
|
||||
- [ ] Telemetry dry-run produces valid JSON
|
||||
- [ ] Hub receiving telemetry payloads with `"platform":"synology"`
|
||||
@@ -0,0 +1,165 @@
|
||||
# Handoff — WG mgmt incident + resume Task 2.1 (2026-07-01)
|
||||
|
||||
**Audience:** local Claude Code running on a machine with **LAN access** to the bench router.
|
||||
**Why this exists:** the cloud session hit the router only over the WireGuard mgmt tunnel, which is
|
||||
currently DOWN. The remaining work must be driven from the LAN side. This doc + the two scripts in
|
||||
`scripts/diag/` contain everything needed to (1) restore WG mgmt, (2) make the tunnel robust across
|
||||
either WAN, then (3) resume Task 2.1.
|
||||
|
||||
---
|
||||
|
||||
## 1. Current project state
|
||||
|
||||
- **Phase 0** (0.1 SSH, 0.2 recon, 0.3 package skeleton) — DONE, committed, plan boxes checked.
|
||||
- **Phase 1** (scoring/decision pure-shell libs + bats) — DONE, committed.
|
||||
- **Task 2.0** (install `mwan3 2.8.15-2` from GL's own feed; disable GL `kmwan`) — DONE + independently
|
||||
verified. Commit `4f33ca3`. Recon doc §4. Default route + WG mgmt survived that change.
|
||||
- **BLOCKER (this doc):** WG mgmt tunnel to the bench router (`10.88.0.2`) went dead the moment ATT
|
||||
fiber was plugged into `eth0`. On-device Task 2.1 work is blocked until mgmt is restored.
|
||||
|
||||
Plan of record: `docs/superpowers/plans/2026-06-30-pioneer-bus-router-balancer.md`
|
||||
Recon / ground truth: `docs/recon/2026-06-30-xe3000-environment.md` (see §5 for the fiber pre-check).
|
||||
|
||||
---
|
||||
|
||||
## 2. Environment & access
|
||||
|
||||
| Thing | Value |
|
||||
|------|-------|
|
||||
| Router | GL.iNet **GL-XE3000**, OpenWrt 21.02-SNAPSHOT, busybox/ash, kernel 5.4 |
|
||||
| Router LAN | **`192.168.8.1`** — `ssh root@192.168.8.1`, password **`kitPLANE1!!`** (dropbear) |
|
||||
| Router WG mgmt IP | `10.88.0.2` (iface `wgclient1`, route table `1001`) |
|
||||
| Uplink 1 (cellular) | `rmnet_mhi0` — T-Mobile CGNAT, default-route metric **40** |
|
||||
| Uplink 2 (fiber, = "Starlink" stand-in) | `eth0` / UCI iface `wan` — ATT fiber, metric **10** (now the preferred default). Public IP seen by hub: `99.163.124.171` |
|
||||
| Fleet hub | DO droplet **`kit-fleet-hub`**, public **`167.172.237.162`**, WG **`10.88.0.1`**, wg iface `wg0`. Cloud session reaches it via `mcp__lab-do__do_ssh_command droplet="kit-fleet-hub"`. |
|
||||
| NOT the hub | `162.243.83.36` is a *different* lab server — ignore it for this work. |
|
||||
|
||||
**Reachability rule:** while WG is down, the **LAN is the only way in**. The hub has NO route to
|
||||
`192.168.8.1` (router LAN is private); and no interactive shell survives over the broken tunnel.
|
||||
|
||||
---
|
||||
|
||||
## 3. The incident — evidence gathered so far
|
||||
|
||||
**Symptom:** fiber plugged into `eth0` → fiber (metric 10) became the default route → the WG *underlay*
|
||||
roamed onto fiber (hub now sees the peer at `99.163.124.171:44881`, was T-Mobile CGNAT `100.x`).
|
||||
Handshakes succeed **both** directions and are always "fresh", but **no data flows** — ICMP/SSH over
|
||||
the tunnel get zero replies in either direction.
|
||||
|
||||
**Verified CLEAN on the hub** (`167.172.237.162`, checked directly via DO API):
|
||||
- `wg show` peer allowed-ips = **`10.88.0.2/32`** ✓
|
||||
- `ip route get 10.88.0.2` → **`dev wg0 src 10.88.0.1`** ✓; `net.ipv4.ip_forward = 1` ✓
|
||||
- Firewall: `nft list ruleset` **empty**; iptables INPUT/FORWARD/OUTPUT policy **ACCEPT, 0 packets** ✓
|
||||
- Endpoint roam **succeeded** (hub has the fiber endpoint, fresh handshake).
|
||||
- → The three "usual hub culprits" (allowed-ips, firewall, return route) are **all ruled out**.
|
||||
|
||||
**Verified on the router** (LAN side):
|
||||
- `rp_filter` = **0** on all interfaces → **NOT** the cause (an earlier hypothesis; discard it).
|
||||
- Routing correct: `ip route get 10.88.0.1 from 10.88.0.2` → **`dev wgclient1 table 1001`** ✓
|
||||
- `wgclient1` MTU = 1420 (WG default).
|
||||
- Source-bound ping (`ping -I 10.88.0.2 10.88.0.1`) **fails**, but `wg` transfer incremented only ~32
|
||||
bytes — suspiciously small for 3 ICMP packets (~430B expected), so it's unclear the data packets
|
||||
actually hit the wire.
|
||||
|
||||
**Measurement caveat:** hub-side and router-side counters were sampled at *different moments*, so
|
||||
cross-side counter comparisons are unreliable. The §B capture in the diagnose script fixes this by
|
||||
capturing on the router itself while it generates traffic.
|
||||
|
||||
**Ruled out:** rp_filter (=0), hub firewall (none), hub return route (correct), hub allowed-ips
|
||||
(correct), MTU-for-tiny-pings (a 144B encapsulated ping ≈ the 148B handshake that succeeds — MTU only
|
||||
kills *large* packets like SSH data, not pings).
|
||||
|
||||
**Leading hypotheses, in priority order:**
|
||||
1. **`persistent-keepalive` missing** on the router's `wgclient1` peer → behind ATT fiber NAT the
|
||||
return mapping expires between handshakes (T-Mobile CGNAT likely masked this). *Not yet confirmed —
|
||||
we never captured the router's `wg show`.* **Get `wg show` first.**
|
||||
2. **Router-side cryptokey routing / peer AllowedIPs** doesn't include `10.88.0.1` → inbound replies
|
||||
silently dropped by WG. Also revealed by `wg show`.
|
||||
3. **MTU/PMTU black-hole** for large packets — unlikely to block pings but WILL break SSH data; worth
|
||||
pre-empting with `wgclient1 mtu 1280` + MSS clamp.
|
||||
|
||||
---
|
||||
|
||||
## 4. DO THIS FIRST — restore mgmt (LAN, `root@192.168.8.1`)
|
||||
|
||||
### Step 1 — diagnose (read-only, safe)
|
||||
```sh
|
||||
sh scripts/diag/wg-mgmt-diagnose.sh # or paste it; it's plain ash
|
||||
```
|
||||
The single most important output is **`wg show`** (§A) — we have never seen the router's peer config.
|
||||
Look for:
|
||||
- `persistent keepalive: (none)` → hypothesis #1 is live.
|
||||
- `allowed ips:` on the hub peer — must include `10.88.0.1` (else hypothesis #2).
|
||||
- §B capture: does `[eth0]` show **inbound** `167.172.237.162.51820 →` packets during the ping?
|
||||
- inbound present → replies arrive on fiber → decrypt/inner issue (chase AllowedIPs/keys).
|
||||
- outbound only → router's packets aren't leaving, or hub isn't returning → coordinate a hub capture.
|
||||
|
||||
### Step 2 — apply the fix (mutating; `wg-mgmt-fix.sh` is gated, read its header)
|
||||
Default fix order (safe, reversible, and each independently supports "works over either WAN"):
|
||||
```sh
|
||||
sh scripts/diag/wg-mgmt-fix.sh keepalive # set persistent_keepalive=25 (try FIRST)
|
||||
# re-test: ping -c3 -I 10.88.0.2 10.88.0.1 → expect replies
|
||||
sh scripts/diag/wg-mgmt-fix.sh mtu # set wgclient1 mtu=1280 (if keepalive alone didn't fix)
|
||||
sh scripts/diag/wg-mgmt-fix.sh rpfilter # persist loose rp_filter (already 0 at runtime; makes it durable)
|
||||
```
|
||||
Each is `uci set … ; uci commit ; ifup wgclient1` — the tunnel bounces (~5s) but nothing else is touched.
|
||||
|
||||
### Step 3 — verify restored
|
||||
```sh
|
||||
ping -c3 -W2 -I 10.88.0.2 10.88.0.1 # replies from the hub = tunnel data path healthy
|
||||
```
|
||||
When that replies, mgmt over WG is back. (The cloud session can then re-confirm from the hub with
|
||||
`ssh -i /root/.ssh/busfleet root@10.88.0.2 'ubus call system board'` — but you can also just keep
|
||||
driving from the LAN.)
|
||||
|
||||
---
|
||||
|
||||
## 5. Robustness goal — WG mgmt must survive on EITHER WAN
|
||||
|
||||
Bake these in (they're the durable fix, not just incident cleanup), then fold into the Phase 8
|
||||
provisioning profile:
|
||||
- `persistent_keepalive=25` — holds the NAT mapping open on whichever WAN is active.
|
||||
- `wgclient1 mtu=1280` — fits under cellular **and** ATT-fiber/PPPoE path MTUs; kills PMTU black-holes.
|
||||
- rp_filter loose (`=2`) persisted via `/etc/sysctl.conf`; if GL re-tightens on WAN flap, add a
|
||||
`/etc/hotplug.d/iface/` hook that re-asserts it (the fix script writes one under `rpfilter`).
|
||||
- The mgmt return rule `from 10.88.0.2 lookup 1001` is already default-route-independent — leave it.
|
||||
- After the fix, **prove failover**: `ifdown wan` (fiber) → mgmt should ride cellular; `ifup wan` →
|
||||
rides fiber again. Do this only once mgmt is confirmed healthy, and see the guardrail below.
|
||||
|
||||
---
|
||||
|
||||
## 6. THEN resume Task 2.1 (per the plan)
|
||||
|
||||
Follow `superpowers:subagent-driven-development` (implement → spec review → code-quality review →
|
||||
commit → check the plan box), same as Tasks 0.3 / 2.0.
|
||||
|
||||
Task 2.1 files & specifics (from the plan):
|
||||
- `package/kit-busrouter/files/usr/lib/busrouter/wan-mwan.sh`
|
||||
- mwan3 config: members **cellular** (`modem_0001` / `rmnet_mhi0`) + **fiber-as-WAN2** (`wan` / `eth0`);
|
||||
policy `balanced`; `track_ip` 1.1.1.1 & 8.8.8.8; `reliability 1`; `down 3`; `up 3`.
|
||||
- `mwan_set_weight <member> <weight>` = `uci set mwan3.<member>.weight` + `mwan3 restart`, **guarded by
|
||||
`weight_changed`** (already implemented in `lib-decide.sh`) so we don't flush conntrack needlessly.
|
||||
- Rewrite mwan3's stock config — it ships referencing `wan/wwan/tethering`, none of which match our
|
||||
live ifaces (noted in Task 2.0).
|
||||
|
||||
**Failover drill — SAFE direction (critical):**
|
||||
- Fiber currently carries the WG mgmt underlay, so it is **SAFE to bounce CELLULAR** (`rmnet_mhi0`) for
|
||||
the failover test, and **NEVER down `eth0`/fiber** while it carries mgmt.
|
||||
- **Always re-check first:** `ip route get 167.172.237.162` tells you which uplink the WG underlay is
|
||||
on *right now*; down the OTHER one. (After you set persistent_keepalive + prove either-WAN failover,
|
||||
the mgmt-bearing uplink can shift — re-check every time.)
|
||||
|
||||
---
|
||||
|
||||
## 7. Guardrails (learned this session)
|
||||
|
||||
- **No dead-man watchdog on this firmware:** `setsid` and `nohup` are both absent, and
|
||||
`start-stop-daemon -x /bin/sh` self-matches the session shell. So you cannot arm a reliable
|
||||
auto-rollback over SSH. Mitigation: do risky routing changes **from the LAN side** (you're physically
|
||||
present) so a mistake doesn't lock you out; verify mgmt after every mutation.
|
||||
- **No `base64` on the router** — transfer files by decoding on the hub/LAN box and piping cleartext
|
||||
into `cat >` on the router, or just paste scripts.
|
||||
- **busybox `ping` has no `-M do`** — can't do a clean DF/PMTU probe with it; infer MTU issues from
|
||||
large-vs-small behavior or `tracepath` if present.
|
||||
- Verify mgmt access (a WG-sourced ping or SSH) after **every** routing/firewall/mwan3 change.
|
||||
```
|
||||
@@ -0,0 +1,85 @@
|
||||
# Inbound management SSH over the WireGuard tunnel
|
||||
|
||||
How the busfleet hub reaches a GL-XE3000 router for management, and how to fix it when
|
||||
inbound SSH over the tunnel is broken.
|
||||
|
||||
## How it works
|
||||
|
||||
The router runs a WireGuard **client** (`wgclient1`, `proto=wgclient`) that dials the hub
|
||||
at `167.172.237.162:51820`. Once the tunnel is up the router holds `10.88.0.2/32` and the
|
||||
hub is `10.88.0.1`. The hub SSHes *inbound* to `10.88.0.2:22` over that tunnel using the
|
||||
`busfleet-hub-jump` key.
|
||||
|
||||
The key detail: on OpenWrt the WG client interface is in its **own firewall zone**
|
||||
(`wgclient1`), which defaults to `input='DROP'`. Traffic arriving over the tunnel is
|
||||
evaluated by the `zone_wgclient1_input` iptables chain — **not** `zone_wan_input`. So the
|
||||
mgmt-SSH accept rule must have its source zone set to `wgclient1`, or it lands in the wrong
|
||||
chain and never matches (tunnel SSH silently hits the zone DROP).
|
||||
|
||||
## Provisioning a router
|
||||
|
||||
1. **Confirm the WG client zone name** (may differ per unit):
|
||||
|
||||
```sh
|
||||
uci show network | grep -i wgclient # interface, e.g. network.wgclient1
|
||||
uci show firewall | grep -iE 'name=|\.network=|input=' # find its zone + input policy
|
||||
```
|
||||
|
||||
2. **Allow inbound mgmt SSH over the tunnel.** Bind the rule's `src` to the WG client zone
|
||||
(not `wan`), restricted to the hub subnet:
|
||||
|
||||
```sh
|
||||
uci add firewall rule
|
||||
uci set firewall.@rule[-1].name='Allow-WG-mgmt-SSH'
|
||||
uci set firewall.@rule[-1].src='wgclient1' # the ACTUAL WG zone from step 1
|
||||
uci set firewall.@rule[-1].proto='tcp'
|
||||
uci set firewall.@rule[-1].dest_port='22'
|
||||
uci set firewall.@rule[-1].family='ipv4'
|
||||
uci set firewall.@rule[-1].src_ip='10.88.0.0/24'
|
||||
uci set firewall.@rule[-1].target='ACCEPT'
|
||||
uci commit firewall && /etc/init.d/firewall restart
|
||||
```
|
||||
|
||||
3. **Install the hub jump key:**
|
||||
|
||||
```sh
|
||||
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMS842ZbXPXHbVd9IjyVGubq7M/yQA/cn/vHgB5R0/ZE busfleet-hub-jump' >> /etc/dropbear/authorized_keys
|
||||
chmod 600 /etc/dropbear/authorized_keys
|
||||
```
|
||||
|
||||
## Verifying
|
||||
|
||||
Use `wg show` as the source of truth for tunnel health — a recent `latest handshake` plus
|
||||
non-zero transfer counters means the data path is live:
|
||||
|
||||
```sh
|
||||
wg show # endpoint, latest handshake, transfer
|
||||
iptables -S zone_wgclient1_input | grep 'dport 22' # exactly one ACCEPT from 10.88.0.0/24
|
||||
```
|
||||
|
||||
Gotchas:
|
||||
|
||||
- **BusyBox `nc` has no `-v`/`-z` flags** (`nc [IPADDR PORT]` only), and `51820` is UDP —
|
||||
don't rely on `nc -vz` to test the tunnel. Use `wg show`.
|
||||
- **The hub does not answer ICMP over the tunnel**, so `ping 10.88.0.1` shows 100% loss even
|
||||
when the tunnel is healthy. The definitive end-to-end test is the hub SSHing in.
|
||||
|
||||
## Troubleshooting: hub can't SSH in
|
||||
|
||||
1. `wg show` — is there a recent handshake? If not, the tunnel is down (check WAN/5G, the
|
||||
hub endpoint, and keys) before looking at the firewall.
|
||||
2. `uci show firewall | grep Allow-WG-mgmt-SSH` up through the rule — confirm `src` is the
|
||||
WG client zone (`wgclient1`), **not** `wan`. This is the most common misconfiguration:
|
||||
|
||||
```sh
|
||||
uci set firewall.<section>.src='wgclient1'
|
||||
uci commit firewall && /etc/init.d/firewall restart
|
||||
```
|
||||
|
||||
3. `grep busfleet-hub-jump /etc/dropbear/authorized_keys` — key present, file perms `600`.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-07-01** — Fixed inbound mgmt SSH on a unit whose `Allow-WG-mgmt-SSH` rule was bound
|
||||
to `src='wan'` instead of `src='wgclient1'`, so tunnel SSH hit the `wgclient1` zone DROP.
|
||||
Repointed the rule to `wgclient1` and installed the `busfleet-hub-jump` key.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Three-Bus Field Pilot Validation Checklist
|
||||
|
||||
**Plan ref:** Task 9.1 / Phase 9 — Field pilot sign-off gate
|
||||
**Units:** bus01 (bench), bus02, bus03 (first three production units)
|
||||
**Gate:** All three buses must pass before scaling to the 194-unit fleet
|
||||
|
||||
---
|
||||
|
||||
## Pre-flight (per unit, before boarding)
|
||||
|
||||
- [ ] `ssh pioadmin@192.168.8.1` → password `Pioneer321!` succeeds
|
||||
- [ ] `cat /etc/busrouter/device-id` returns correct ID (e.g. `B0001`)
|
||||
- [ ] `cat /etc/busrouter/version` returns current package version
|
||||
- [ ] WG tunnel up: `wg show` — peer handshake < 60 s ago
|
||||
- [ ] `mwan3 status` — at least cellular online; wan offline is OK if no dish yet
|
||||
- [ ] `ps | grep daemon.sh` — busrouter daemon running
|
||||
- [ ] `ls /tmp/busrouter/` — state files being written (metric_*, score_*, gps_last, etc.)
|
||||
- [ ] Status page: `http://192.168.8.1/busrouter/` loads in browser, shows live data
|
||||
- [ ] SSID visible on both bands as `<DEVICE_ID>`, connects with `Pioneer123`
|
||||
|
||||
---
|
||||
|
||||
## Section A — Dual-WAN balancing under motion
|
||||
|
||||
**Scenario:** Bus drives its route for ≥ 30 minutes with both cellular and Starlink active.
|
||||
|
||||
- [ ] A1: Both mwan3 members show "online" during route: `mwan3 status` from hub via WG
|
||||
- [ ] A2: Weights shift based on scores (not stuck 50/50 throughout entire run)
|
||||
- Capture: `cat /tmp/busrouter/score_modem_0001` and `score_wan` at start, midpoint, end
|
||||
- [ ] A3: No LAN interruption detected by a ping flood from a laptop on bus WiFi
|
||||
- Command: `ping -c 1800 8.8.8.8` (30 min, 1 s interval) — expect ≤ 2% loss total
|
||||
- [ ] A4: `/tmp/busrouter/metric_modem_0001` and `metric_wan` both updated each cycle
|
||||
- [ ] A5: No busrouter daemon crash: `logread | grep busrouter | grep -i error` — 0 results
|
||||
|
||||
---
|
||||
|
||||
## Section B — SIM auto-switch on real coverage change
|
||||
|
||||
**Scenario:** Drive through a known T-Mobile coverage gap or weak-signal zone (RSRP < −110 dBm
|
||||
for ≥ 3 consecutive cycles) with a second SIM in slot 2.
|
||||
|
||||
- [ ] B1: SIM slot switches from 1 → 2 when RSRP drops below threshold for SIM_SWITCH_CYCLES
|
||||
- Check: `cat /tmp/busrouter/sim_slot` changes from `1` to `2`
|
||||
- [ ] B2: Cooldown (300 s) is observed — no rapid oscillation between slots
|
||||
- Check: `logread | grep sim_switch` — only one event per coverage-gap pass
|
||||
- [ ] B3: Cellular reconnects after slot switch
|
||||
- Check: `mwan3 status` shows modem_0001 returns online within ~30 s of switch
|
||||
- [ ] B4: SIM slot correctly reported in telemetry (hub receives correct slot number)
|
||||
- [ ] B5: Capture mid-test snapshot — `logread | grep sim_switch` and `cat /tmp/busrouter/sim_slot` immediately after slot change
|
||||
|
||||
---
|
||||
|
||||
## Section C — Starlink handoff
|
||||
|
||||
**Scenario:** Trigger a Starlink→cellular handoff. Preferred: natural signal loss / obstruction
|
||||
event during the route. Acceptable fallback (bench unit or when no dish): `ifdown wan` / `ifup wan`
|
||||
— document which method was used in the sign-off record. If using fallback, note that this does
|
||||
not validate natural Starlink obstruction detection (requires grpcurl + real dish).
|
||||
|
||||
- [ ] C1: After `ifdown wan`, mwan3 shifts to `modem_0001 (100%)` within ≤ 15 s
|
||||
- Check: `mwan3 status`, `logread | grep mwan3`
|
||||
- [ ] C2: LAN ping survives the failover (continue A3 ping through this test)
|
||||
- [ ] C3: WG mgmt tunnel stays alive throughout — all `ssh` commands during failover succeed
|
||||
- **Critical:** WG must ride cellular when eth0 is down; verify ip rule still routes correctly
|
||||
- [ ] C4: After `ifup wan`, mwan3 declares wan online and restores balanced policy within ≤ 30 s
|
||||
- [ ] C5: Weight bar on status page `http://192.168.8.1/busrouter/` updates correctly
|
||||
|
||||
---
|
||||
|
||||
## Section D — Telemetry pipeline
|
||||
|
||||
**Scenario:** Run for ≥ 10 cycles with hub reachable, then simulate hub unreachable and
|
||||
confirm buffering, then restore hub and confirm flush.
|
||||
|
||||
- [ ] D1: Hub receives ≥ 5 telemetry payloads per bus while online
|
||||
- Check hub: `journalctl -u busrouter-hub` or inspect telemetry store for device_id entries
|
||||
- [ ] D2: JSON payloads are valid (no parse errors at hub) — correct device_id, version fields
|
||||
- [ ] D3: Buffer fills when hub unreachable
|
||||
- Simulate: set `TELEMETRY_HUB=http://192.168.99.99` in daemon env and cycle 5 times
|
||||
- Check: `ls /tmp/busrouter/telemetry-buf/*.json | wc -l` → 5 files
|
||||
- [ ] D4: Buffer flushes on hub restore
|
||||
- Restore TELEMETRY_HUB, trigger `telemetry_send` (or wait next cycle)
|
||||
- Check: `ls /tmp/busrouter/telemetry-buf/` → empty; hub received the 5 buffered payloads
|
||||
- [ ] D5: Buffer cap enforced — at 61st file, oldest is evicted (not new item rejected)
|
||||
- Check: create 60 dummy files in `telemetry-buf/`; run one cycle; count stays at 60
|
||||
|
||||
---
|
||||
|
||||
## Section E — Failover must never drop LAN
|
||||
|
||||
**Scenario:** Stress test — bounce both uplinks repeatedly and confirm LAN clients never lose
|
||||
internet for more than the mwan3 detection window (~15 s).
|
||||
|
||||
- [ ] E1: Bounce cellular 3x (`ifdown modem_0001; sleep 5; ifup modem_0001`): LAN gap ≤ 15 s each
|
||||
- [ ] E2: Bounce Starlink 3x (`ifdown wan; sleep 5; ifup wan`): LAN gap ≤ 15 s each
|
||||
- [ ] E3: Simultaneously down both for 10 s, then restore: graceful "unreachable" state, no crash
|
||||
- [ ] E4: After E3, both interfaces return online and balanced policy resumes automatically
|
||||
- [ ] E5: After E1–E4, busrouter daemon still running — no segfault or restart in syslog
|
||||
|
||||
---
|
||||
|
||||
## Section F — WireGuard mgmt plane integrity
|
||||
|
||||
**Scenario:** Verify the WG mgmt tunnel (10.88.0.X) survives every link-state change tested
|
||||
in Sections A–E. This is the "never lose the management plane" requirement.
|
||||
|
||||
- [ ] F1: Hub can SSH to router at `root@10.88.0.X` throughout all A–E tests
|
||||
- [ ] F2: `wg show wgclient1` latest handshake stays < 60 s during all tests
|
||||
- [ ] F3: After cellular→eth0 underlay switch (eth0 comes up), WG remains live on new underlay
|
||||
- Check endpoint change at hub: `wg show wg0 endpoints`
|
||||
- [ ] F4: After eth0→cellular underlay switch (eth0 goes down), WG stays live
|
||||
- **Note:** if eth0 is the WG underlay, do NOT down it without cellular as backup
|
||||
|
||||
---
|
||||
|
||||
## Section G — Log capture + sign-off
|
||||
|
||||
For each bus, capture the following at end of pilot run and attach to the sign-off record:
|
||||
|
||||
```sh
|
||||
# Run on router (or via hub WG tunnel):
|
||||
logread > /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
mwan3 status >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
ip rule show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
wg show >> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
cat /tmp/busrouter/score_modem_0001 /tmp/busrouter/score_wan \
|
||||
/tmp/busrouter/sim_slot /tmp/busrouter/gps_last \
|
||||
>> /tmp/pilot-log-$(cat /etc/busrouter/device-id)-$(date +%Y%m%d).txt
|
||||
# Copy log off the router:
|
||||
scp root@192.168.8.1:/tmp/pilot-log-*.txt ./logs/
|
||||
```
|
||||
|
||||
### Sign-off record
|
||||
|
||||
**Log archive:** copy logs to `docs/pilot/logs/<DEVICE_ID>/` in this repo and commit after each bus.
|
||||
|
||||
| Unit | Pilot date | Sections passed | C method | Issues found | Signed off by |
|
||||
|------|-----------|-----------------|----------|--------------|---------------|
|
||||
| bus01 (bench) | — | A B C D E F | synthetic | — | — |
|
||||
| bus02 | — | A B C D E F | natural/synthetic | — | — |
|
||||
| bus03 | — | A B C D E F | natural/synthetic | — | — |
|
||||
|
||||
**Fleet scale gate:** all three rows have a signer in "Signed off by" → proceed to full 194-unit
|
||||
rollout using `provision.sh` + fleet automation (Project 4).
|
||||
|
||||
---
|
||||
|
||||
## Known issues going into pilot
|
||||
|
||||
1. **No Starlink dish on bench unit (bus01)** — Section A/C Starlink tests require a dish on eth0
|
||||
or an ATT Fiber substitute. Document which uplink was used for each Starlink-targeted test.
|
||||
2. **grpcurl absent** — Starlink gRPC metrics (`starlink.sh`) are currently stubbed; Starlink
|
||||
score is based on echo/ping only. `grpcurl` binary must be bundled before Section C tests
|
||||
use `starlink_last` data for scoring.
|
||||
3. **GPS fix lag** — First GPS fix after `AT+QGPS=1` can take 1–5 minutes cold-start. Section A
|
||||
GPS checks should allow time for first fix before asserting `gps_last` is populated.
|
||||
4. **RSRP threshold tuning** — `SIM_SWITCH_RSRP=-110` is the default. Real coverage-gap zones
|
||||
may need adjustment; tune per route if Section B tests don't trigger naturally.
|
||||
@@ -0,0 +1,183 @@
|
||||
# Pioneer Bus Router — Device Identity & Provisioning
|
||||
|
||||
**Last updated:** 2026-07-01
|
||||
**Target hardware:** GL.iNet GL-XE3000 (Puli AX), OpenWrt 21.02-SNAPSHOT, ARMv8
|
||||
|
||||
---
|
||||
|
||||
## 1. Fleet-wide constants
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| WiFi SSID (2.4 + 5 GHz) | `<DEVICE_ID>` — the plan spec's `X0000` shorthand means SSID = device ID (e.g. `B0042`) |
|
||||
| WiFi password | `Pioneer123` |
|
||||
| Admin username (logical) | `pioadmin` |
|
||||
| Admin password | `Pioneer321!` |
|
||||
| Admin SSH user (OpenWrt) | `root` (GL-XE3000 uses root for all admin; pioadmin is the logical name) |
|
||||
| Timezone | `UTC` |
|
||||
| Cellular APN | `fast.t-mobile.com` (T-Mobile 5G, Quectel RM520N-GL, bus `1-1.2`) |
|
||||
| WireGuard hub endpoint | `167.172.237.162:51820` |
|
||||
| WireGuard hub IP | `10.88.0.1/24` |
|
||||
| WireGuard tunnel subnet | `10.88.0.0/24` |
|
||||
| WireGuard keepalive | `25 s` |
|
||||
| WireGuard MTU | `1280` |
|
||||
| WG mwan3 fix rule | priority `99`, dest `10.88.0.0/24`, lookup `1001` (required — see §4) |
|
||||
| mwan3 balance mode | `balanced` (weight-based, 50/50 default; daemon adjusts live weights) |
|
||||
|
||||
**Hub WireGuard public key:** `REPLACE_WITH_HUB_WG_PUBKEY`
|
||||
(Retrieve with `wg pubkey < /etc/wireguard/hub.key` on the hub at `167.172.237.162`)
|
||||
|
||||
---
|
||||
|
||||
## 2. Per-device parameters
|
||||
|
||||
Each bus unit gets a unique identity. Fill in this table for each device:
|
||||
|
||||
| DEVICE_ID | WG tunnel IP | WG public key | SIM 1 ICCID | SIM 2 ICCID | Notes |
|
||||
|-----------|-------------|---------------|-------------|-------------|-------|
|
||||
| `bus01` | `10.88.0.2` | *(see recon)* | `8901260417776090740F` | — | bench unit; ATT Fiber on eth0 in lab |
|
||||
| `B0001` | `10.88.0.3` | *(generate)* | — | — | *(first production unit)* |
|
||||
|
||||
**Column format:**
|
||||
- `DEVICE_ID`: `B` + 4-digit zero-padded bus number (e.g. `B0042`)
|
||||
- `WG tunnel IP`: `10.88.0.X` (assigned by hub admin, no DHCP; range 10.88.0.2–10.88.0.254)
|
||||
- `WG private key`: base64, generated per device, never stored in this doc (keep in secrets store)
|
||||
- `SIM ICCID`: printed on SIM card tray
|
||||
|
||||
### Generating a keypair for a new device
|
||||
|
||||
```sh
|
||||
# On any Linux machine with wireguard-tools:
|
||||
PRIVKEY=$(wg genkey)
|
||||
PUBKEY=$(echo "$PRIVKEY" | wg pubkey)
|
||||
echo "Private: $PRIVKEY"
|
||||
echo "Public: $PUBKEY"
|
||||
```
|
||||
|
||||
Register the public key + assigned IP at the hub:
|
||||
```sh
|
||||
# On hub (167.172.237.162):
|
||||
wg set wg0 peer <DEVICE_PUBKEY> allowed-ips 10.88.0.X/32 persistent-keepalive 25
|
||||
# Persist (choose ONE based on how the hub manages wg0):
|
||||
wg-quick save wg0 # if hub uses wg-quick
|
||||
# OR: edit /etc/wireguard/wg0.conf manually and run: wg addconf wg0 <(diff)
|
||||
# Verify hub persistence method before using wg-quick save — it overwrites the file.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Provisioning procedure
|
||||
|
||||
### Prerequisites
|
||||
- Physical or LAN access to the router at `192.168.8.1`
|
||||
- Cellular SIM installed in slot 1 (provides internet for opkg)
|
||||
- `kit-busrouter.ipk` copied to `/tmp/kit-busrouter.ipk` on the router
|
||||
- Device keypair generated and hub-side peer registered (§2)
|
||||
|
||||
### Steps
|
||||
|
||||
```sh
|
||||
# 1. Copy provision.sh and package to the router
|
||||
scp scripts/provision.sh root@192.168.8.1:/tmp/provision.sh
|
||||
scp kit-busrouter_*.ipk root@192.168.8.1:/tmp/kit-busrouter.ipk
|
||||
|
||||
# 2. SSH in and run
|
||||
ssh root@192.168.8.1
|
||||
chmod +x /tmp/provision.sh
|
||||
/tmp/provision.sh B0042 "<WG_PRIVKEY>" "10.88.0.42"
|
||||
# Device reboots automatically after ~5 s.
|
||||
|
||||
# 3. Post-reboot verification (from hub 10.88.0.1 over WireGuard)
|
||||
ssh root@10.88.0.42
|
||||
mwan3 status # both modem_0001 and wan should appear (wan offline if no dish)
|
||||
cat /etc/busrouter/device-id # B0042
|
||||
wg show # wgclient1 peer handshake recent
|
||||
```
|
||||
|
||||
### Expected WiFi after provisioning
|
||||
- SSID: `B0042` (same on 2.4 + 5 GHz)
|
||||
- Password: `Pioneer123`
|
||||
- LAN IP: `192.168.8.1`
|
||||
|
||||
### Expected admin access
|
||||
- SSH: `ssh root@192.168.8.1` → password `Pioneer321!`
|
||||
- Web UI: `http://192.168.8.1` → root / `Pioneer321!`
|
||||
- WG path: `ssh root@10.88.0.42` (from hub, or via WG client)
|
||||
|
||||
---
|
||||
|
||||
## 4. mwan3 + WireGuard tunnel coexistence fix
|
||||
|
||||
**Critical — must be applied or in-tunnel SSH drops silently.**
|
||||
|
||||
Root cause: mwan3 adds all directly-connected networks (including wgclient1's 10.88.0.0/24) to
|
||||
its `mwan3_connected` ipset and marks matching packets with `0x3f00`. The ip rule
|
||||
`9000: not from all fwmark 0/0xf000 lookup main` then routes replies via main table, which has
|
||||
no route for 10.88.0.0/24 (wgclient1 routes live in table 1001). Packets are silently dropped.
|
||||
|
||||
Fix: add an ip rule at priority 99 (before mwan3's mark-based rules) routing to 10.88.0.0/24
|
||||
via table 1001. `provision.sh` does this automatically:
|
||||
|
||||
```sh
|
||||
# Runtime:
|
||||
ip rule add to 10.88.0.0/24 priority 99 lookup 1001
|
||||
# Persisted via UCI (recreated by netifd on boot):
|
||||
uci add network rule
|
||||
uci set network.@rule[-1].lookup='1001'
|
||||
uci set network.@rule[-1].dest='10.88.0.0/24'
|
||||
uci set network.@rule[-1].priority='99'
|
||||
uci commit network
|
||||
```
|
||||
|
||||
Also required: `net.ipv4.conf.wgclient1.rp_filter=2` (written to /etc/sysctl.conf by provision.sh
|
||||
and re-asserted on WAN flap via /etc/hotplug.d/iface/99-busrouter-rpfilter).
|
||||
|
||||
---
|
||||
|
||||
## 5. GL-XE3000 hardware notes
|
||||
|
||||
| Detail | Value |
|
||||
|--------|-------|
|
||||
| Modem | Quectel RM520N-GL (5G NR SA) |
|
||||
| Modem transport | MHI/PCIe (NOT USB-CDC; no `/dev/cdc-wdm*`) |
|
||||
| Modem bus for gl_modem | `1-1.2` (verify per unit: `ls /sys/bus/usb/devices/ | grep -v ':' | sort`) |
|
||||
| Signal read | `ubus call modem.signal get_signals '{"time":1}'` (primary); `gl_modem -B 1-1.2 AT AT+QCSQ` (fallback) |
|
||||
| GPS enable | `gl_modem -B 1-1.2 AT AT+QGPS=1` (OFF by default; provision.sh writes to /etc/rc.local) |
|
||||
| GPS read | `gl_modem -B 1-1.2 AT AT+QGPSLOC=2` (returns error 505/516 until first fix after GNSS enable) |
|
||||
| SIM switch | `gl_modem -B 1-1.2 AT AT+QUIMSLOT=<1|2>` |
|
||||
| SIM query | `gl_modem -B 1-1.2 AT AT+QUIMSLOT?` |
|
||||
| kmwan (GL multiwan) | Disabled by provision.sh — do NOT re-enable; mwan3 owns routing |
|
||||
| grpcurl | NOT installed; Phase 3 (Starlink) must bundle static arm64 binary in the .ipk |
|
||||
| setsid / nohup | ABSENT on this firmware; use `start-stop-daemon -S -b -x <prog>` for background procs |
|
||||
| WG firewall zone | `wgclient1` (NOT `wan`); SSH allow rule: `src=wgclient1 src_ip=10.88.0.0/24 dest_port=22` |
|
||||
|
||||
---
|
||||
|
||||
## 6. Bench unit `bus01` — specific config
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| DEVICE_ID | `bus01` (rename to fleet format `B0001` for production) |
|
||||
| LAN IP | `192.168.8.1` |
|
||||
| WG tunnel IP | `10.88.0.2` |
|
||||
| Cellular (WAN1) | `rmnet_mhi0`, UCI `modem_0001`, T-Mobile CGNAT |
|
||||
| Starlink (WAN2) | `eth0`, UCI `wan` — no dish on bench unit; ATT Fiber used as stand-in |
|
||||
| WG underlay | Rides `eth0` when fiber is attached (do NOT down eth0 for bench tests) |
|
||||
| Access via hub | `ssh root@10.88.0.2` (from hub `10.88.0.1`) |
|
||||
|
||||
---
|
||||
|
||||
## 7. Checklist — new device before field deployment
|
||||
|
||||
- [ ] Device ID assigned and recorded in fleet registry
|
||||
- [ ] WG keypair generated; public key registered at hub with assigned IP
|
||||
- [ ] SIM installed; APN = `fast.t-mobile.com`; cellular data confirmed
|
||||
- [ ] `provision.sh` run end-to-end; device rebooted cleanly
|
||||
- [ ] WiFi visible as `<DEVICE_ID>`, password `Pioneer123`
|
||||
- [ ] SSH to `root@192.168.8.1` succeeds with `Pioneer321!`
|
||||
- [ ] `mwan3 status` shows both interfaces (wan offline is OK if no dish yet)
|
||||
- [ ] WG tunnel up: `wg show` shows handshake < 60 s ago
|
||||
- [ ] Hub can SSH to `root@10.88.0.WG_IP`
|
||||
- [ ] `/etc/busrouter/device-id` contains correct DEVICE_ID
|
||||
- [ ] `busrouter` daemon running: `ps | grep daemon.sh`
|
||||
- [ ] Status page accessible: `http://192.168.8.1/busrouter/`
|
||||
@@ -0,0 +1,384 @@
|
||||
# GL-XE3000 Device Reconnaissance (Task 0.2)
|
||||
|
||||
**Date:** 2026-06-30
|
||||
**Unit:** bench `bus01` — GL.iNet GL-XE3000 (Puli AX)
|
||||
**Access path:** LAB → `kit-fleet-hub` (DO droplet 167.172.237.162, WG 10.88.0.1) → jump SSH
|
||||
`ssh -i /root/.ssh/busfleet root@10.88.0.2` (router WG IP 10.88.0.2).
|
||||
**Access confirmed:** yes — `ubus call system board` returns `"model": "GL.iNet GL-XE3000"`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Resolved facts table (feeds later plan tasks)
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Cellular ifname (WAN1 member) | **`rmnet_mhi0`** (QMI proto `qcm`, control node `/dev/mhi_QMI0`, UCI iface `modem_0001`, APN `fast.t-mobile.com`) |
|
||||
| Starlink WAN ifname (WAN2) | **`eth0`** (UCI iface `wan`, proto `dhcp`). NOTE: currently `DOWN`/`NO-CARRIER` — no dish attached to bench unit. `eth1` = LAN (`br-lan`). |
|
||||
| mwan3 present? | **NO** — `/etc/config/mwan3` absent, `mwan3` package not installed → Task 3.0 / Phase 2 must `opkg install mwan3`. GL ships its own multiwan (`gl-sdk4-ui-multiwan`, `kmod-gl-sdk4-kmwan`, ubus `hotplug.kmwan`) which we are NOT using. |
|
||||
| Signal read command | **Preferred:** `ubus call modem.signal get_signals '{"time":1}'` → JSON `{rsrp,rsrq,sinr,slot,network_type,strength,mode}`. **AT fallback:** `gl_modem -B 1-1.2 AT AT+QCSQ` → `+QCSQ: "NR5G",<rssi>,<sinr>,<rsrp>`. **`uqmi` is NOT usable** (see Implications). |
|
||||
| SIM select command | Read: `gl_modem -B 1-1.2 AT AT+QUIMSLOT?` → `+QUIMSLOT: 1`. Switch: `gl_modem -B 1-1.2 AT AT+QUIMSLOT=2` (confirmed slot-query works; switch command not exercised to avoid dropping the live link). |
|
||||
| GPS read command | `gl_modem -B 1-1.2 AT AT+QGPSLOC=2` — but GNSS must be enabled first: `AT+QGPS=1`. Currently `AT+QGPS?` = `0` (disabled), so `QGPSLOC` returns `+CME ERROR: 505` (no fix). |
|
||||
| grpcurl available? | **NO** — `command -v grpcurl` = MISSING → Phase 3 (Starlink) must bundle a static arm64 `grpcurl` binary in the opkg package. |
|
||||
| Firewall zone of WG client iface | **`wgclient1`** — its own dedicated zone named `wgclient1` (network `wgclient1`, input DROP). NOT `wan`. The access-fix rule is `firewall.@rule[22]` name `Allow-WG-mgmt-SSH`, `src='wgclient1'`, `src_ip='10.88.0.0/24'`, dest_port 22, ACCEPT. |
|
||||
| Modem USB bus (for `gl_modem -B`) | **`1-1.2`** (Quectel RM520N-GL, 4 USB interfaces at `1-1.2:1.0-1.3`) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Raw command outputs
|
||||
|
||||
### `ubus call system board`
|
||||
```json
|
||||
{
|
||||
"kernel": "5.4.211",
|
||||
"hostname": "GL-XE3000",
|
||||
"system": "ARMv8 Processor rev 4",
|
||||
"model": "GL.iNet GL-XE3000",
|
||||
"board_name": "glinet,xe3000-emmc",
|
||||
"release": {
|
||||
"distribution": "OpenWrt",
|
||||
"version": "21.02-SNAPSHOT",
|
||||
"target": "mediatek/mt7981"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `/etc/config/network` (relevant excerpts)
|
||||
```
|
||||
config device # LAN bridge
|
||||
option name 'br-lan'
|
||||
list ports 'eth1'
|
||||
config interface 'lan'
|
||||
option device 'br-lan'
|
||||
option ipaddr '192.168.8.1'
|
||||
config device
|
||||
option name 'eth0'
|
||||
config interface 'wan' # Starlink WAN port
|
||||
option device 'eth0'
|
||||
option proto 'dhcp'
|
||||
option metric '10'
|
||||
config interface 'modem_0001' # Cellular / 5G
|
||||
option apn 'fast.t-mobile.com'
|
||||
option ip_type 'IPV4V6'
|
||||
option proto 'qcm'
|
||||
option device '/dev/mhi_QMI0'
|
||||
option metric '40'
|
||||
option iccid '8901260417776090740F'
|
||||
config interface 'wgclient1' # WireGuard out-of-band mgmt
|
||||
option proto 'wgclient'
|
||||
option config 'peer_2001'
|
||||
option ip4table '1001'
|
||||
```
|
||||
(Also present but unused/disabled: `wwan`, `secondwan`, `guest`, various GL vpn policy rules.)
|
||||
|
||||
### `/etc/config/mwan3`
|
||||
```
|
||||
NO mwan3
|
||||
```
|
||||
|
||||
### `opkg list-installed` (filtered: mwan3|grpc|speedtest|modem|gl-|uqmi|quectel)
|
||||
Notable entries (full list captured in session):
|
||||
```
|
||||
gl-sdk4-modem git-2025.290.37125-7c3efcc-1
|
||||
gl-sdk4-ui-multiwan git-2025.259.31350-8a2a5c6-1
|
||||
gl-sdk4-ui-modemsignallog git-2024.303.14204-4241be7-1
|
||||
kmod-gl-pcie-modem 5.4.211+20221009-1
|
||||
kmod-gl-sdk4-kmwan 5.4.211+git-2025.304.07170-bb470c0-1
|
||||
uqmi 2020-11-22-0a19b5b7-3
|
||||
```
|
||||
No `mwan3`, no `grpcurl`, no `speedtest` (Ookla) package installed.
|
||||
|
||||
### `ip -br addr`
|
||||
```
|
||||
lo UNKNOWN 127.0.0.1/8 ::1/128
|
||||
eth0 DOWN # Starlink WAN — no carrier
|
||||
eth1 UP # LAN member
|
||||
ra0 UP # 2.4G wifi
|
||||
rmnet_mhi0 UP 100.112.80.45/30 # Cellular (CGNAT)
|
||||
br-lan UP 192.168.8.1/24
|
||||
wgclient1 UNKNOWN 10.88.0.2/32 # WG mgmt tunnel
|
||||
```
|
||||
|
||||
### `ip route`
|
||||
```
|
||||
default via 100.112.80.46 dev rmnet_mhi0 proto static src 100.112.80.45 metric 40
|
||||
100.112.80.44/30 dev rmnet_mhi0 proto static scope link metric 40
|
||||
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
|
||||
```
|
||||
|
||||
### Modem device nodes
|
||||
```
|
||||
/dev/mhi_BHI /dev/mhi_DIAG /dev/mhi_DUN /dev/mhi_LOOPBACK /dev/mhi_QMI0
|
||||
/dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2 /dev/ttyUSB3
|
||||
```
|
||||
No `/dev/cdc-wdm*` — modem is on the **MHI/PCIe** transport, not USB-CDC.
|
||||
|
||||
### Tool availability
|
||||
```
|
||||
gl_modem: /usr/bin/gl_modem
|
||||
uqmi: /sbin/uqmi
|
||||
grpcurl: MISSING
|
||||
mmcli: MISSING
|
||||
gpsd: MISSING
|
||||
```
|
||||
|
||||
### `gl_modem` usage
|
||||
```
|
||||
gl_modem -B <bus> AT <AT_CMD> # send AT command; bus for this unit = 1-1.2
|
||||
```
|
||||
|
||||
### Signal — ubus (preferred)
|
||||
```json
|
||||
// ubus call modem.signal get_signals '{"time":1}'
|
||||
{
|
||||
"signals": [{
|
||||
"sinr": 13, "slot": 1, "network_type": "NR5G-SA",
|
||||
"timestamp": 1782917019, "strength": 4,
|
||||
"rsrp": -96, "rsrq": -10, "mode": 5
|
||||
}]
|
||||
}
|
||||
```
|
||||
`ubus -v list modem.signal` → methods `get_signals {time:Integer}`, `upload_cloud_signals {action:String}`.
|
||||
|
||||
### Signal — AT fallback
|
||||
```
|
||||
gl_modem -B 1-1.2 AT AT+QCSQ
|
||||
+QCSQ: "NR5G",-97,14,-10 # <sysmode>,<rssi>,<sinr>,<rsrp>
|
||||
```
|
||||
|
||||
### SIM slot
|
||||
```
|
||||
gl_modem -B 1-1.2 AT AT+QUIMSLOT?
|
||||
+QUIMSLOT: 1 # active SIM = slot 1
|
||||
```
|
||||
|
||||
### GPS / GNSS
|
||||
```
|
||||
gl_modem -B 1-1.2 AT AT+QGPS?
|
||||
+QGPS: 0 # GNSS engine currently OFF
|
||||
gl_modem -B 1-1.2 AT AT+QGPSLOC=2
|
||||
+CME ERROR: 505 # no fix (engine not enabled)
|
||||
```
|
||||
|
||||
### Starlink WAN port link
|
||||
```
|
||||
eth0 DOWN <NO-CARRIER,BROADCAST,MULTICAST,UP>
|
||||
```
|
||||
|
||||
### Firewall (WG mgmt zone + access-fix rule)
|
||||
```
|
||||
firewall.wgclient1 = zone
|
||||
firewall.wgclient1.name='wgclient1'
|
||||
firewall.wgclient1.network='wgclient1'
|
||||
firewall.wgclient1.input='DROP'
|
||||
firewall.@rule[22].name='Allow-WG-mgmt-SSH'
|
||||
firewall.@rule[22].src='wgclient1'
|
||||
firewall.@rule[22].src_ip='10.88.0.0/24'
|
||||
```
|
||||
Zone[1] `wan` groups: `wan wan6 wwan secondwan modem_0001 modem_0001_4` (input DROP).
|
||||
|
||||
---
|
||||
|
||||
## 3. Implications for the plan
|
||||
|
||||
1. **mwan3 is NOT installed (plan flagged this as possible).** `/etc/config/mwan3` is absent and no `mwan3`
|
||||
opkg is present. Phase 2 / Task 3.0 must `opkg install mwan3` (verify it exists in GL's feed for
|
||||
`mediatek/mt7981` @ 21.02-SNAPSHOT; if not, add the upstream OpenWrt 21.02 feed or bundle it).
|
||||
GL ships its own `kmwan`/multiwan stack (`kmod-gl-sdk4-kmwan`, `gl-sdk4-ui-multiwan`, ubus
|
||||
`hotplug.kmwan`) — the spec locks on `mwan3`, so we intentionally do NOT use GL's kmwan. Watch for
|
||||
conflict/overlap between the two multiwan systems on the same box (routing marks, hotplug).
|
||||
|
||||
2. **Firewall zone assumption in Task 0.1 was WRONG.** Task 0.1 Step 1 guessed `src='wan'` (with a
|
||||
"verify / adjust if wgclient" note). The WG client interface is actually in its **own zone `wgclient1`**,
|
||||
and that is what the working fix rule uses (`@rule[22] src='wgclient1' src_ip='10.88.0.0/24'`). Any
|
||||
future mgmt/telemetry firewall rules must target zone **`wgclient1`**, not `wan`.
|
||||
|
||||
3. **`uqmi` is effectively unusable for signal reads.** `uqmi -d /dev/mhi_QMI0 --get-signal-info` **hangs
|
||||
(timeout)** — GL's modem manager holds the QMI control channel (single-client QMUX). Use the GL ubus
|
||||
path `modem.signal get_signals` (clean JSON, no device contention) as the primary signal source, with
|
||||
`gl_modem -B 1-1.2 AT AT+QCSQ` as the AT fallback. **Phase 4 Task 4.1 should target ubus first**, not
|
||||
uqmi as the plan's example suggested.
|
||||
|
||||
4. **`gl_modem` requires a `-B <bus>` argument** (the plan's examples `gl_modem AT AT+...` omit it and would
|
||||
fail). The bus on this unit is **`1-1.2`**. This must be discovered per-unit (parse from
|
||||
`/sys/bus/usb/devices` or GL config) rather than hard-coded, since it could differ across units —
|
||||
record for the provisioning profile (Phase 8).
|
||||
|
||||
5. **GPS/GNSS is present but OFF by default.** `AT+QGPS?` = 0, so `AT+QGPSLOC=2` errors `+CME ERROR: 505`.
|
||||
Phase 5 Task 5.1 (`gps_fix`) must **enable GNSS first** (`AT+QGPS=1`, one-time / on boot) and handle the
|
||||
"no fix yet" (error 505/516) state gracefully before location is available. GPS command form is
|
||||
confirmed as Quectel `AT+QGPSLOC=2` (decimal degrees).
|
||||
|
||||
6. **`grpcurl` absent (plan flagged this).** Phase 3 Task 3.3 must bundle a static **arm64 (ARMv8)**
|
||||
`grpcurl` binary in the opkg package for Starlink gRPC.
|
||||
|
||||
7. **No Starlink dish on the bench unit** (`eth0` NO-CARRIER). WAN2 (Starlink) integration and any
|
||||
two-WAN balancing test cannot be validated on this bench unit as-is — need a dish on `eth0`, or test
|
||||
WAN2 with a substitute uplink. The cellular WAN (`rmnet_mhi0`) is the only live uplink right now, and
|
||||
it carries the WG mgmt tunnel (CGNAT `100.112.80.45/30`, T-Mobile).
|
||||
|
||||
8. **Ookla speedtest CLI not installed** — Phase 3 Task 3.2 must bundle/install it (or rely on the curl
|
||||
fallback), consistent with the plan's fallback design.
|
||||
|
||||
---
|
||||
|
||||
## 4. Task 2.0 — mwan3 / kmwan changeover (2026-07-01)
|
||||
|
||||
Performed on bench `bus01` via the WG mgmt path. This section feeds the Phase 8 provisioning profile.
|
||||
|
||||
### GL multiwan (kmwan) — exact init name + stand-down
|
||||
- **GL multiwan init script name: `kmwan`** — `/etc/init.d/kmwan` (the ONLY `mwan|kmwan` init on the box;
|
||||
confirms recon §1 `kmod-gl-sdk4-kmwan`). It is a `/etc/rc.common` script, `START=21`, sourcing
|
||||
`/lib/functions/kmwan.sh`. It was **enabled** but had **no running daemon** (routing is managed via
|
||||
hotplug scripts / a single static default route, not a persistent process).
|
||||
- **Exact commands used to stand it down** (Phase 8 must run these):
|
||||
```
|
||||
/etc/init.d/kmwan disable # rc=0 → NOW_DISABLED (won't start on boot)
|
||||
/etc/init.d/kmwan stop # rc=0
|
||||
```
|
||||
Verify with `/etc/init.d/kmwan enabled` (returns non-zero = disabled) and
|
||||
`ps w | grep -i mwan | grep -v grep` (no kmwan process).
|
||||
|
||||
### mwan3 install
|
||||
- Installed from **GL's own feed** (no third-party/upstream feed needed):
|
||||
`opkg update && opkg install mwan3` → **`mwan3 2.8.15-2`** (`mwan3_2.8.15-2_all.ipk`,
|
||||
feed `glinet_gli_pub` @ v21.02.3). Binary at `/usr/sbin/mwan3`.
|
||||
- mwan3 ships a **stock default `/etc/config/mwan3`** (dated 2022) with interfaces
|
||||
`wan/wwan/tethering` (+v6). **NONE of these match our live cellular uplink** (`rmnet_mhi0`, UCI iface
|
||||
`modem_0001`), so mwan3 is currently **inert**: `mwan3 status` shows every interface "offline and
|
||||
tracking is down", **no active policies, no active user rules**. mwan3's `mwan3rtmon` route-monitor
|
||||
daemon runs but does not touch the default route (member config is a later task).
|
||||
|
||||
### Safety / recovery notes for Phase 8
|
||||
- **`setsid` and `nohup` are BOTH ABSENT** on this GL firmware (BusyBox build). For detached/background
|
||||
processes use **`start-stop-daemon -S -b -x <prog> -- <args>`** (present at `/sbin/start-stop-daemon`).
|
||||
Note: `start-stop-daemon -b -x /bin/sh` will refuse to launch if it matches an already-running
|
||||
`/bin/sh` ("… is already running") — give the watchdog a unique interpreter/exec name or wrap it so the
|
||||
`-x` match is unique.
|
||||
- **Post-change verification (mgmt access + default route SURVIVED):** after `kmwan disable/stop`, a fresh
|
||||
probe returned `"model": "GL.iNet GL-XE3000"`, default route unchanged
|
||||
(`default via 100.116.118.126 dev rmnet_mhi0 … metric 40` — CGNAT src rotated from the recon-day
|
||||
`100.112.80.45` to `100.116.118.125`, normal T-Mobile lease change), and `wgclient1 10.88.0.2/32` up.
|
||||
The WG mgmt routing rules (`from/to 10.88.0.2 lookup 1001`, table 1001) are intact. **Disabling kmwan
|
||||
did not disturb routing** — expected, since kmwan had no active daemon and the live path is a single
|
||||
static default on `rmnet_mhi0`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Task 2.1 pre-check — second uplink (ATT Fiber on eth0) added (2026-07-01)
|
||||
|
||||
**Status: BLOCKED (device shell unreachable) — but the load-bearing WG-underlay question is answered
|
||||
from hub-side WireGuard evidence.**
|
||||
|
||||
### What happened
|
||||
The operator plugged ATT Fiber into `eth0` (WAN2 stand-in for Starlink). Immediately afterward the
|
||||
router's SSH shell (via the WG mgmt tunnel `10.88.0.2:22`) became **unreachable** from the fleet hub,
|
||||
even though the WireGuard tunnel itself is **healthy**. All on-device commands (eth0 lease, `ip route`
|
||||
metrics, `ip route get`, `ip rule`, `mwan3 status`) therefore **could not be run**. No device state was
|
||||
mutated. The facts below are what the hub can observe about the tunnel from `10.88.0.1` (wg0).
|
||||
|
||||
### THE KEY ANSWER — WG mgmt path now rides eth0 (ATT Fiber)
|
||||
The WireGuard peer **endpoint changed** from the T-Mobile CGNAT `100.x.x.x` seen in §2/§4 to a **public,
|
||||
non-CGNAT address**, which is the ATT Fiber public IP:
|
||||
|
||||
```
|
||||
# hub: wg show wg0 endpoints
|
||||
0Xn7aGlyxHQ/X9uo8lsOeI3EFA7KO+zlO+qIhrzsAF8= 99.163.124.171:44881
|
||||
# latest-handshakes: 6 s ago (fresh); transfer: 335 KiB rx / 244 KiB sent (growing)
|
||||
```
|
||||
|
||||
`99.163.124.171` is a routable public IP (not `100.64.0.0/10` CGNAT), i.e. the router is now sourcing its
|
||||
WG handshakes out **eth0 / ATT Fiber**, not `rmnet_mhi0` / cellular. **The WG mgmt underlay rides eth0
|
||||
(ATT Fiber).** Corollary: **downing `eth0` would sever the WG mgmt path / SSH access.**
|
||||
|
||||
### Safe failover-test direction
|
||||
- **DO NOT bring down `eth0` (ATT Fiber)** for the failover drill — it carries the WG underlay; downing
|
||||
it cuts mgmt/SSH.
|
||||
- **`rmnet_mhi0` (cellular) is the SAFE interface to bring down** for the mwan3 failover test — it does
|
||||
NOT currently carry the WG underlay handshake. (This is the *inverse* of the pre-fiber state in §2/§4,
|
||||
where cellular carried mgmt and eth0 was NO-CARRIER.)
|
||||
|
||||
### Current mgmt-reachability fault (flag for operator — needs on-device follow-up)
|
||||
The WG control plane is up but **no in-tunnel application traffic returns from the router**:
|
||||
|
||||
```
|
||||
# hub, over the tunnel:
|
||||
ip route get 10.88.0.2 -> 10.88.0.2 dev wg0 src 10.88.0.1 (route correct)
|
||||
ping -c3 10.88.0.2 -> 100% loss (even 56-byte payloads)
|
||||
nc -vz 10.88.0.2 22 -> timed out ("Operation now in progress")
|
||||
# 4 s window: wg 'sent' climbed 244308 -> (SYN retransmits reach router), 'rx' FLAT
|
||||
```
|
||||
|
||||
Interpretation: the router **receives** hub traffic (encrypted `sent` counter grows) but **replies with
|
||||
nothing** — its in-tunnel return path for ICMP/TCP is broken. Most likely the policy-routing that steers
|
||||
tunnel replies (table `1001`, `from/to 10.88.0.2 lookup 1001`, per §4) or the WG interface's reverse-path
|
||||
was disturbed when the fiber default route (metric 10) came up and displaced cellular (metric 40) as the
|
||||
preferred default. This must be diagnosed/repaired **on-device via a console/LAN path** (the WG path is
|
||||
the very thing that's down) before Task 2.1 routing changes proceed. Not attempted here (read-only,
|
||||
no-mutate scope).
|
||||
|
||||
### Facts NOT captured (require device shell — blocked)
|
||||
- eth0/wan DHCP lease (ATT subnet / gateway / netmask), eth0 carrier flag from the device side.
|
||||
- Full `ip route` default table with metrics (expected eth0 metric 10 preferred over cellular metric 40 —
|
||||
strongly implied by the endpoint move to fiber, but not directly read).
|
||||
- `rmnet_mhi0` current CGNAT IP, `ip route get 167.172.237.162` / `ip route get 99.163.124.171`.
|
||||
- `mwan3 status` / `mwan3 interfaces` view of the two ifaces, `ip rule` table listing.
|
||||
|
||||
Re-run §5 in full once console/LAN access to `bus01` is restored and the WG mgmt path is repaired.
|
||||
|
||||
---
|
||||
|
||||
## 6. WG mgmt tunnel root cause + fix (2026-07-01)
|
||||
|
||||
**Symptom:** after ATT Fiber plugged into eth0 (metric 10 preferred default), WG handshakes were
|
||||
fresh but all in-tunnel data (ICMP/SSH from 10.88.0.1 to 10.88.0.2) returned 100% loss.
|
||||
|
||||
**Root cause:** `mwan3` adds all directly-connected networks to the `mwan3_connected` ipset.
|
||||
10.88.0.0/24 is a directly-connected route (wgclient1), so mwan3's iptables `mwan3_connected`
|
||||
chain marks packets destined for that prefix with `0x3f00`. The ip rule
|
||||
`9000: not from all fwmark 0/0xf000 lookup main` then routes them via the **main table**,
|
||||
which has **no route for 10.88.0.0/24** (wgclient1 routes are only in table 1001 due to
|
||||
`ip4table='1001'` in UCI). The packet is silently dropped — never reaches WG.
|
||||
|
||||
**Fix (applied 2026-07-01):**
|
||||
Add an ip rule at priority 99 (before any mwan3 fwmark rule) routing packets to 10.88.0.0/24
|
||||
via table 1001:
|
||||
|
||||
```
|
||||
ip rule add to 10.88.0.0/24 priority 99 lookup 1001
|
||||
```
|
||||
|
||||
Persisted as a UCI network rule so netifd recreates it on every boot:
|
||||
|
||||
```
|
||||
uci add network rule
|
||||
uci set network.@rule[-1].lookup='1001'
|
||||
uci set network.@rule[-1].dest='10.88.0.0/24'
|
||||
uci set network.@rule[-1].priority='99'
|
||||
uci commit network
|
||||
```
|
||||
|
||||
Also applied for robustness: `wgclient1 mtu=1280`, rp_filter=2 persisted via /etc/sysctl.conf
|
||||
and /etc/hotplug.d/iface/99-busrouter-rpfilter (re-asserts on WAN flap).
|
||||
|
||||
**Failover note:** `scripts/diag/wg-mgmt-diagnose.sh` and `scripts/diag/wg-mgmt-fix.sh` in
|
||||
this repo implement the fix. The underlying keepalive (`persistent_keepalive=25`) was already
|
||||
configured from a prior session; the new ip rule was the missing piece.
|
||||
|
||||
**Phase 8 provisioning profile must include** this UCI rule — without it, any device with
|
||||
mwan3 configured will have the WG mgmt tunnel silently drop in-tunnel data.
|
||||
|
||||
## 7. Task 2.1 bench-test findings (2026-07-01)
|
||||
|
||||
mwan3 configured with `modem_0001` (cellular) + `wan` (fiber/eth0), policy `balanced`.
|
||||
Both interfaces tracked online (1.1.1.1 + 8.8.8.8, reliability 1, down=3, up=3 x 5s interval).
|
||||
|
||||
**Failover drill result:**
|
||||
- `ip link set rmnet_mhi0 down` -> mwan3 declares `modem_0001 error` within ~15s (checked)
|
||||
- Policy shifts to `wan (100%)` automatically (checked)
|
||||
- WG mgmt tunnel (riding eth0) stays alive throughout failover (checked)
|
||||
- **Recovery gotcha:** `ip link set rmnet_mhi0 up` does NOT restore the QMI data session.
|
||||
Must use `ifdown modem_0001 && ifup modem_0001` to reconnect. After ~20s both interfaces
|
||||
return to `balanced: wan(50%) modem_0001(50%)` (checked)
|
||||
|
||||
**Phase 9 pilot:** fiber failover (bring fiber down, WG must ride cellular) should be tested
|
||||
once the provisioning robustness fix (section 6 ip rule) is confirmed to survive both uplink combos.
|
||||
@@ -0,0 +1,448 @@
|
||||
# Pioneer Bus Router — On-Device Multi-WAN Balancer Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build the OpenWrt `opkg` package that intelligently balances 5G + Starlink on a GL-XE3000, auto-switches SIMs on signal degradation, and reports telemetry to the fleet hub.
|
||||
|
||||
**Architecture:** A branded `opkg` package layered on GL.iNet stock firmware. A supervised `init.d` daemon runs a control loop: collect per-WAN metrics → score → tune `mwan3` weights + trigger failover/SIM-switch → emit telemetry over the WireGuard tunnel. Pure-shell scoring/hysteresis logic is unit-tested with `bats`; on-device integration is validated on a bench unit (`bus01`, WG `10.88.0.2`).
|
||||
|
||||
**Tech Stack:** POSIX/ash shell, OpenWrt `opkg` + `procd`/`init.d`, `mwan3`, `uci`, GL modem tooling (AT/`uqmi`), Ookla Speedtest CLI + `curl`, Starlink gRPC (`grpcurl`), `bats-core` for tests.
|
||||
|
||||
**Reference source (port from):** the Synology package `aiwanbal` at
|
||||
`https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer` — reuse the *logic* in
|
||||
`bin/aiwanbal-daemon.sh`, `bin/aiwanbal-modem.sh`, `bin/aiwanbal-starlink.sh`, `bin/aiwanbal-throttle.sh`;
|
||||
rewrite the platform layer for OpenWrt.
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-06-30-pioneer-bus-router-design.md`
|
||||
|
||||
---
|
||||
|
||||
## Recon-applied corrections (2026-06-30 — authoritative, see `docs/recon/2026-06-30-xe3000-environment.md`)
|
||||
|
||||
These OVERRIDE the placeholder examples elsewhere in this plan:
|
||||
|
||||
- **Cellular ifname:** `rmnet_mhi0` (QMI, UCI iface `modem_0001`, APN `fast.t-mobile.com`). **Starlink WAN:** `eth0` (UCI iface `wan`) — no dish attached to the bench unit yet.
|
||||
- **`mwan3` is NOT installed; GL ships its own `kmwan` multiwan stack.** Decision (locked): install `mwan3` via `opkg` and **disable `kmwan`** for our two members so we keep portable, standard control (spec §4). Watch for GL firmware updates re-enabling `kmwan`. Handled in **Task 2.0** (new, below).
|
||||
- **Signal read = ubus, not uqmi.** Use `ubus call modem.signal get_signals` (clean JSON: rsrp/rsrq/sinr/slot). `uqmi` **hangs** (GL holds the QMUX). AT fallback: `gl_modem -B 1-1.2 AT AT+QCSQ`.
|
||||
- **`gl_modem` requires a `-B <bus>` arg** (bench unit = `1-1.2`); the bus must be discovered per-unit at provisioning. SIM select: `gl_modem -B <bus> AT AT+QUIMSLOT=2` (read via `AT+QUIMSLOT?`).
|
||||
- **GPS is present but OFF by default** — one-time enable `gl_modem -B <bus> AT AT+QGPS=1`, then read `AT+QGPSLOC=2`; handle "no fix" (CME 505) gracefully.
|
||||
- **`grpcurl` is missing** — bundle a static arm64 binary in the package (Task 3.3).
|
||||
- **WG mgmt firewall zone = `wgclient1`** (not `wan`); the Task 0.1 rule was corrected accordingly.
|
||||
- **Bench limits:** no Starlink dish, no Ookla CLI on the unit → WAN2 balancing and speed tests can't be validated on the bench yet; validate in the Phase 9 pilot. Cellular is the only live uplink and carries the WG tunnel.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
package/kit-busrouter/
|
||||
├── Makefile # OpenWrt package definition (opkg build)
|
||||
├── files/
|
||||
│ ├── etc/
|
||||
│ │ ├── init.d/busrouter # procd service (supervise the daemon)
|
||||
│ │ ├── config/busrouter # UCI config (all knobs)
|
||||
│ │ └── busrouter/
|
||||
│ │ └── busrouter.conf # runtime config seeded on first install
|
||||
│ └── usr/lib/busrouter/
|
||||
│ ├── daemon.sh # control loop orchestrator
|
||||
│ ├── lib-score.sh # PURE: scoring math (unit-tested)
|
||||
│ ├── lib-decide.sh # PURE: hysteresis, weight formula (unit-tested)
|
||||
│ ├── wan-mwan.sh # mwan3 weight apply + dead-gateway detection
|
||||
│ ├── metrics-net.sh # latency/jitter/loss ping probes
|
||||
│ ├── speedtest.sh # Ookla + curl fallback, data-cap aware
|
||||
│ ├── modem-sim.sh # signal read + SIM switch (GL modem)
|
||||
│ ├── starlink.sh # dish gRPC status
|
||||
│ └── telemetry.sh # collect + POST to hub over WG; GPS read
|
||||
├── luci-app-busrouter/ # branded local status/diagnostics UI (Phase 7)
|
||||
└── tests/
|
||||
├── bats/ # bats-core suites for lib-score / lib-decide
|
||||
└── fixtures/ # captured sample outputs (speedtest, AT, gRPC)
|
||||
|
||||
docs/
|
||||
├── recon/2026-06-30-xe3000-environment.md # Phase 0 output (facts about the device)
|
||||
└── provisioning/device-identity.md # Phase 8 output
|
||||
```
|
||||
|
||||
Split rationale: `lib-score.sh` and `lib-decide.sh` are **pure functions** (stdin/args → stdout, no side effects) so they unit-test cleanly and hold in context. Everything touching the device (mwan3, modem, gRPC) is isolated behind its own file with a narrow interface the daemon calls.
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Foundations & device reconnaissance
|
||||
|
||||
### Task 0.1: Establish LAB SSH access to the bench router
|
||||
|
||||
**Files:** none (infra). Produces working `ssh -J` path from LAB → router.
|
||||
|
||||
- [x] **Step 1: Allow input on the WireGuard zone (run from local Claude Code over LAN)**
|
||||
|
||||
On the router (`192.168.8.1`, root pw `kitPLANE1!!`), permit management from the hub subnet:
|
||||
```sh
|
||||
uci add firewall rule
|
||||
uci set firewall.@rule[-1].name='Allow-WG-mgmt-SSH'
|
||||
uci set firewall.@rule[-1].src='wan' # verify actual zone of the wg client iface in Step 3 recon; adjust if 'wgclient'
|
||||
uci set firewall.@rule[-1].proto='tcp'
|
||||
uci set firewall.@rule[-1].dest_port='22'
|
||||
uci set firewall.@rule[-1].family='ipv4'
|
||||
uci set firewall.@rule[-1].src_ip='10.88.0.0/24'
|
||||
uci set firewall.@rule[-1].target='ACCEPT'
|
||||
uci commit firewall && /etc/init.d/firewall restart
|
||||
```
|
||||
|
||||
- [x] **Step 2: Install the LAB public key on the router**
|
||||
|
||||
Append the LAB key to dropbear authorized_keys:
|
||||
```sh
|
||||
mkdir -p /etc/dropbear
|
||||
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAFxQnKmt21nWE/HDt/UPLWeEvXP/lTrvtpVoPNrbuH8 kitadmin@lab-cloudcli' >> /etc/dropbear/authorized_keys
|
||||
chmod 600 /etc/dropbear/authorized_keys
|
||||
```
|
||||
|
||||
- [x] **Step 3: Verify from the LAB (jump through the hub)**
|
||||
|
||||
Put the LAB private key on the hub once (so the hub can be a jump host), then:
|
||||
Run: `ssh -J root@167.172.237.162 root@10.88.0.2 'ubus call system board'`
|
||||
Expected: JSON with `"model": "GL.iNet GL-XE3000"`. If refused, re-check the firewall `src`/`src_ip` from Step 1 against the recon in 0.2.
|
||||
|
||||
### Task 0.2: Device reconnaissance — capture ground truth
|
||||
|
||||
**Files:** Create `docs/recon/2026-06-30-xe3000-environment.md`
|
||||
|
||||
- [x] **Step 1: Collect environment facts from the router**
|
||||
|
||||
Run each and record output verbatim into the recon doc:
|
||||
```sh
|
||||
ubus call system board # model, firmware, kernel
|
||||
cat /etc/config/network # interfaces, incl. cellular + wan
|
||||
cat /etc/config/mwan3 2>/dev/null || echo "NO mwan3" # is mwan3 present/configured?
|
||||
opkg list-installed | grep -Ei 'mwan3|grpc|speedtest|modem|gl-' # available tooling
|
||||
ip -br addr; ip route; ls -l /dev/cdc-wdm* /dev/ttyUSB* 2>/dev/null
|
||||
gl_modem AT AT+CGDCONT? 2>/dev/null; gl_modem 2>/dev/null | head # GL modem CLI shape
|
||||
uqmi -d /dev/cdc-wdm0 --get-signal-info 2>/dev/null # QMI signal path (if present)
|
||||
cat /tmp/gps* 2>/dev/null; ls /dev/tty* | head # GPS/GNSS exposure
|
||||
logread | grep -i sim | tail # how SIM state is logged
|
||||
```
|
||||
|
||||
- [x] **Step 2: Record the resolved facts the rest of the plan depends on**
|
||||
|
||||
In the recon doc, fill this table with the ACTUAL values discovered (these feed later tasks):
|
||||
|
||||
| Fact | Value |
|
||||
|------|-------|
|
||||
| Cellular ifname (mwan3 member) | e.g. `rmnet_mhi0` |
|
||||
| Starlink WAN ifname | e.g. `eth1` / `wan` |
|
||||
| mwan3 present? | yes/no (if no → Task 3.0 installs it) |
|
||||
| Signal read command | `gl_modem AT AT+QCSQ` \| `uqmi ... --get-signal-info` |
|
||||
| SIM select command | e.g. `gl_modem AT AT+QUIMSLOT=2` (confirm on device) |
|
||||
| GPS read command | e.g. `gl_modem AT AT+QGPSLOC=2` |
|
||||
| grpcurl available? | yes/no (if no → bundle static arm64 binary) |
|
||||
|
||||
- [x] **Step 3: Commit the recon doc**
|
||||
```bash
|
||||
git add docs/recon/2026-06-30-xe3000-environment.md
|
||||
git commit -m "docs: XE3000 device reconnaissance facts"
|
||||
```
|
||||
|
||||
### Task 0.3: OpenWrt package skeleton (walking skeleton)
|
||||
|
||||
**Files:** Create `package/kit-busrouter/Makefile`, `files/etc/init.d/busrouter`, `files/etc/config/busrouter`, `files/usr/lib/busrouter/daemon.sh`
|
||||
|
||||
- [x] **Step 1: Write the package Makefile**
|
||||
```make
|
||||
include $(TOPDIR)/rules.mk
|
||||
PKG_NAME:=kit-busrouter
|
||||
PKG_VERSION:=0.1.0
|
||||
PKG_RELEASE:=1
|
||||
include $(INCLUDE_DIR)/package.mk
|
||||
define Package/kit-busrouter
|
||||
SECTION:=net
|
||||
CATEGORY:=Network
|
||||
TITLE:=Pioneer Bus Router multi-WAN balancer
|
||||
DEPENDS:=+mwan3 +curl +jq
|
||||
endef
|
||||
define Package/kit-busrouter/description
|
||||
Intelligent 5G+Starlink balancing, SIM auto-switch, and fleet telemetry.
|
||||
endef
|
||||
define Build/Compile
|
||||
endef
|
||||
define Package/kit-busrouter/install
|
||||
$(INSTALL_DIR) $(1)/usr/lib/busrouter $(1)/etc/init.d $(1)/etc/config $(1)/etc/busrouter
|
||||
$(CP) ./files/usr/lib/busrouter/* $(1)/usr/lib/busrouter/
|
||||
$(INSTALL_BIN) ./files/etc/init.d/busrouter $(1)/etc/init.d/busrouter
|
||||
$(INSTALL_CONF) ./files/etc/config/busrouter $(1)/etc/config/busrouter
|
||||
$(INSTALL_CONF) ./files/etc/busrouter/busrouter.conf $(1)/etc/busrouter/busrouter.conf
|
||||
endef
|
||||
$(eval $(call BuildPackage,kit-busrouter))
|
||||
```
|
||||
|
||||
- [x] **Step 2: Write the procd init script**
|
||||
```sh
|
||||
#!/bin/sh /etc/rc.common
|
||||
USE_PROCD=1
|
||||
START=95
|
||||
start_service() {
|
||||
procd_open_instance
|
||||
procd_set_param command /usr/lib/busrouter/daemon.sh
|
||||
procd_set_param respawn 3600 5 0 # respawn, 5s delay — fail-safe supervision
|
||||
procd_set_param stdout 1; procd_set_param stderr 1
|
||||
procd_close_instance
|
||||
}
|
||||
```
|
||||
|
||||
- [x] **Step 3: Write a minimal daemon that logs a heartbeat**
|
||||
```sh
|
||||
#!/bin/sh
|
||||
CONFIG=/etc/busrouter/busrouter.conf
|
||||
[ -f "$CONFIG" ] && . "$CONFIG"
|
||||
INTERVAL="${INTERVAL:-30}"
|
||||
logger -t busrouter "daemon started (v0.1.0)"
|
||||
while :; do
|
||||
logger -t busrouter "heartbeat: $(date +%s)"
|
||||
sleep "$INTERVAL"
|
||||
done
|
||||
```
|
||||
|
||||
- [x] **Step 4: Seed default config**
|
||||
|
||||
`files/etc/config/busrouter` (UCI) and a sourced `busrouter.conf` with `INTERVAL=30`. Keep both minimal for now.
|
||||
|
||||
- [x] **Step 5: Deploy to the bench unit and verify supervision + reboot survival**
|
||||
|
||||
Build the package (or for fast iteration, `scp` the files into place over LAN), then:
|
||||
```sh
|
||||
/etc/init.d/busrouter enable && /etc/init.d/busrouter start
|
||||
logread -f -e busrouter # expect: "daemon started" then heartbeats
|
||||
reboot # after boot, confirm heartbeats resume
|
||||
```
|
||||
Expected: heartbeat lines every 30s, and again after reboot.
|
||||
|
||||
- [x] **Step 6: Commit**
|
||||
```bash
|
||||
git add package/kit-busrouter && git commit -m "feat: kit-busrouter package skeleton with supervised daemon"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Scoring core (pure shell, unit-tested)
|
||||
|
||||
Port the algorithm defined in `aiwanbal` REQUIREMENTS §2.3 and `bin/aiwanbal-daemon.sh`.
|
||||
|
||||
### Task 1.1: bats test harness
|
||||
|
||||
**Files:** Create `tests/bats/`, add `bats-core` as a dev dependency (documented in README).
|
||||
|
||||
- [x] **Step 1: Install bats locally and add a smoke test**
|
||||
|
||||
`tests/bats/smoke.bats`:
|
||||
```bash
|
||||
@test "bats runs" { run true; [ "$status" -eq 0 ]; }
|
||||
```
|
||||
Run: `bats tests/bats/smoke.bats` → Expected: `1 test, 0 failures`.
|
||||
- [x] **Step 2: Commit** — `git add tests && git commit -m "test: add bats harness"`
|
||||
|
||||
### Task 1.2: Per-metric scoring functions (TDD)
|
||||
|
||||
**Files:** Create `tests/bats/score.bats`, then `files/usr/lib/busrouter/lib-score.sh`
|
||||
|
||||
- [x] **Step 1: Write the failing tests** (`tests/bats/score.bats`)
|
||||
```bash
|
||||
setup() { load '../../package/kit-busrouter/files/usr/lib/busrouter/lib-score.sh'; }
|
||||
|
||||
@test "latency: 5ms scores 100" { run score_latency 5; [ "$output" -eq 100 ]; }
|
||||
@test "latency: 200ms scores 0" { run score_latency 200; [ "$output" -eq 0 ]; }
|
||||
@test "latency clamps below 0" { run score_latency 500; [ "$output" -eq 0 ]; }
|
||||
@test "download: 25MB/s scores 100" { run score_download 25; [ "$output" -eq 100 ]; }
|
||||
@test "download: 0.5MB/s scores 0" { run score_download 0.5; [ "$output" -eq 0 ]; }
|
||||
@test "loss: 0% scores 100" { run score_loss 0; [ "$output" -eq 100 ]; }
|
||||
@test "loss: 40% scores 0" { run score_loss 40; [ "$output" -eq 0 ]; }
|
||||
@test "signal: -65dBm scores 100" { run score_signal -65; [ "$output" -eq 100 ]; }
|
||||
@test "signal: -95dBm scores 0" { run score_signal -95; [ "$output" -eq 0 ]; }
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL** (`bats tests/bats/score.bats` → functions not defined)
|
||||
|
||||
- [x] **Step 3: Implement `lib-score.sh`** (integer math via `awk` for portability on ash)
|
||||
```sh
|
||||
# clamp a value to 0..100
|
||||
_clamp() { awk -v v="$1" 'BEGIN{ if(v<0)v=0; if(v>100)v=100; printf "%d", (v+0.5) }'; }
|
||||
# linear map: at x=lo -> 0..? Here score_* return 0..100 per spec anchors.
|
||||
score_latency() { awk -v x="$1" 'BEGIN{ s=100*(200-x)/(200-5); print s }' | _pipeclamp; }
|
||||
score_download() { awk -v x="$1" 'BEGIN{ s=100*(x-0.5)/(25-0.5); print s }' | _pipeclamp; }
|
||||
score_upload() { awk -v x="$1" 'BEGIN{ s=100*(x-0.5)/(12.5-0.5); print s }' | _pipeclamp; }
|
||||
score_jitter() { awk -v x="$1" 'BEGIN{ s=100*(50-x)/(50-1); print s }' | _pipeclamp; }
|
||||
score_loss() { awk -v x="$1" 'BEGIN{ s=100*(40-x)/40; print s }' | _pipeclamp; }
|
||||
score_signal() { awk -v x="$1" 'BEGIN{ s=100*(x-(-95))/((-65)-(-95)); print s }' | _pipeclamp; }
|
||||
_pipeclamp() { read v; _clamp "$v"; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS** (`bats tests/bats/score.bats` → all pass)
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: per-metric WAN scoring (ported from aiwanbal §2.3)"`
|
||||
|
||||
### Task 1.3: Weighted composite + rolling 3-sample average (TDD)
|
||||
|
||||
**Files:** `tests/bats/composite.bats`, extend `lib-score.sh`
|
||||
|
||||
- [x] **Step 1: Failing tests**
|
||||
```bash
|
||||
@test "composite weights sum to a 0-100 score" {
|
||||
# latency35 dl25 up10 jit10 loss10 sig10 ; all metrics perfect -> 100
|
||||
run score_composite 5 25 12.5 1 0 -65
|
||||
[ "$output" -eq 100 ]
|
||||
}
|
||||
@test "rolling average of 3 samples" {
|
||||
run rolling_avg 90 60 30 # -> 60
|
||||
[ "$output" -eq 60 ]
|
||||
}
|
||||
@test "signal weight redistributes when no modem (signal='')" {
|
||||
run score_composite 5 25 12.5 1 0 "" # still 100, sig weight moved to latency/throughput
|
||||
[ "$output" -eq 100 ]
|
||||
}
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL**
|
||||
- [x] **Step 3: Implement**
|
||||
```sh
|
||||
# args: lat dl up jitter loss signal(optional)
|
||||
score_composite() {
|
||||
sl=$(score_latency "$1"); sd=$(score_download "$2"); su=$(score_upload "$3")
|
||||
sj=$(score_jitter "$4"); sp=$(score_loss "$5")
|
||||
if [ -n "$6" ]; then ss=$(score_signal "$6"); wsig=10; wlat=35; wdl=25
|
||||
else ss=0; wsig=0; wlat=40; wdl=30; fi # redistribute signal weight
|
||||
awk -v sl=$sl -v sd=$sd -v su=$su -v sj=$sj -v sp=$sp -v ss=$ss \
|
||||
-v wlat=$wlat -v wdl=$wdl -v wsig=$wsig 'BEGIN{
|
||||
t=(sl*wlat + sd*wdl + su*10 + sj*10 + sp*10 + ss*wsig)/100; printf "%d",(t+0.5) }'
|
||||
}
|
||||
rolling_avg() { awk 'BEGIN{n=ARGC-1;s=0;for(i=1;i<ARGC;i++)s+=ARGV[i];printf "%d",(s/n+0.5)}' "$@"; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS**
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: composite score + rolling average + signal redistribution"`
|
||||
|
||||
### Task 1.4: Weight formula + hysteresis decision (TDD)
|
||||
|
||||
**Files:** `tests/bats/decide.bats`, `files/usr/lib/busrouter/lib-decide.sh`
|
||||
|
||||
- [x] **Step 1: Failing tests**
|
||||
```bash
|
||||
setup(){ load '../../package/kit-busrouter/files/usr/lib/busrouter/lib-decide.sh'; }
|
||||
@test "equal scores -> 50/50" { run wan_weight 70 70; [ "$output" -eq 50 ]; }
|
||||
@test "weight = 50 + 2*diff, rounded to 10" { run wan_weight 80 60; [ "$output" -eq 80 ]; } # 50+40=90? clamp/round
|
||||
@test "clamped to 20..80" { run wan_weight 100 0; [ "$output" -eq 80 ]; }
|
||||
@test "no change if within hysteresis band" { run weight_changed 50 54; [ "$status" -eq 1 ]; }
|
||||
@test "change if crosses a 10% step" { run weight_changed 50 65; [ "$status" -eq 0 ]; }
|
||||
@test "sim switch only after N bad cycles" { run sim_should_switch 3 3; [ "$status" -eq 0 ]; }
|
||||
@test "sim no switch before threshold" { run sim_should_switch 2 3; [ "$status" -eq 1 ]; }
|
||||
```
|
||||
- [x] **Step 2: Run — expect FAIL**
|
||||
- [x] **Step 3: Implement**
|
||||
```sh
|
||||
# weight for WAN A given scoreA scoreB: 50 + (diff*2), round to nearest 10, clamp 20..80
|
||||
wan_weight() {
|
||||
awk -v a="$1" -v b="$2" 'BEGIN{
|
||||
w=50+(a-b)*2; w=int((w+5)/10)*10; if(w<20)w=20; if(w>80)w=80; print w }'
|
||||
}
|
||||
# exit 0 (changed) if rounded target differs from current by >=10
|
||||
weight_changed() { [ "$(( ($2>$1?$2-$1:$1-$2) ))" -ge 10 ]; }
|
||||
# exit 0 if bad_cycles >= threshold
|
||||
sim_should_switch() { [ "$1" -ge "$2" ]; }
|
||||
```
|
||||
- [x] **Step 4: Run — expect PASS** (adjust the 80/60 anchor test to the clamped/rounded truth if needed)
|
||||
- [x] **Step 5: Commit** — `git commit -am "feat: weight formula, hysteresis, SIM-switch decision"`
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — WAN controller (mwan3 integration)
|
||||
|
||||
### Task 2.0: Install mwan3 and stand down GL's kmwan
|
||||
|
||||
**Files:** none (device state); add `+mwan3` is already in the Makefile DEPENDS.
|
||||
|
||||
- [x] **Step 1:** `opkg update && opkg install mwan3` on the bench unit (via hub jump). *(Done: mwan3 2.8.15-2 from GL's own `glinet_gli_pub` feed — no third-party feed needed.)*
|
||||
- [x] **Step 2:** Disable GL's stock multiwan so it can't fight mwan3: `/etc/init.d/kmwan disable 2>/dev/null; /etc/init.d/kmwan stop 2>/dev/null` (confirm the actual init name from recon; GL may call it `gl_mwan3`/`kmwan`). *(Done: init name is `kmwan`; disabled+stopped, no daemon remains.)*
|
||||
- [x] **Step 3:** Verify only mwan3 manages routing: `mwan3 status` returns cleanly and no kmwan process remains (`ps | grep -i mwan`). *(Done: only `mwan3rtmon` runs; default route + WG mgmt path intact. Independently re-verified.)*
|
||||
- [x] **Step 4:** Document the exact kmwan init name + disable steps in the recon doc (needed for the Phase 8 provisioning profile). **Commit.** *(Done: recon §4 added, commit `4f33ca3`. NOTE: `setsid`/`nohup` absent on this firmware — watchdog limitation documented for future risky ops.)*
|
||||
|
||||
### Task 2.1: Configure mwan3 for cellular + Starlink
|
||||
|
||||
**Files:** `files/usr/lib/busrouter/wan-mwan.sh`; uses ifnames from recon (Task 0.2).
|
||||
|
||||
- [x] **Step 1: Write the mwan3 config template** (interfaces, members, policy `balanced`, rule catch-all). Use the two ifnames from the recon table. Include `track_ip` (1.1.1.1, 8.8.8.8), `reliability 1`, `down 3`, `up 3`.
|
||||
- [x] **Step 2: Implement `mwan_set_weight <member> <weight>`** via `uci set mwan3.<member>.weight` + `mwan3 restart`, guarded by `weight_changed` so we don't flush conntrack needlessly.
|
||||
- [x] **Step 3: Bench test** — force one WAN down (`ip link set <if> down`), confirm `mwan3 status` fails it over within the tracking window and restores on link-up. Record output.
|
||||
- [x] **Step 4: Commit.**
|
||||
|
||||
### Task 2.2: Dead-gateway detection (3-layer)
|
||||
|
||||
**Files:** extend `wan-mwan.sh` (port `aiwanbal` DNS→ping-quorum→HTTP logic).
|
||||
|
||||
- [x] **Step 1: Failing bats test** for the pure decision function `gw_verdict <dns_ok> <ping_ok> <http_ok>` (quorum logic).
|
||||
- [x] **Step 2: Run — FAIL. Step 3: Implement quorum. Step 4: PASS.**
|
||||
- [x] **Step 5:** Wire the probes (interface-bound `nslookup`, `ping -I <if>`, `curl --interface <if>`) into the daemon health pass. Bench-verify. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Metric collectors
|
||||
|
||||
### Task 3.1: Network probes (latency/jitter/loss)
|
||||
**Files:** `metrics-net.sh`. Per-WAN `ping -I <if> -c 10 1.1.1.1`, parse min/avg + mdev (jitter) + loss%. Emit `lat jitter loss` line. Bench-verify against real numbers. Unit-test the parser with a captured fixture in `tests/fixtures/`. **Commit.** *(Done: `net_probe_iface` + `_net_parse_ping`; 4 fixtures; bench eth0 11.8/0.3/0, rmnet_mhi0 36.0/10.5/0; commit `626af9a`.)*
|
||||
|
||||
### Task 3.2: Speed-test engine (Ookla + curl, data-cap aware)
|
||||
**Files:** `speedtest.sh`. Port `aiwanbal-speedtest`: prefer Ookla CLI (`speedtest -f json`), fall back to `curl` throughput; **skip auto-tests when the active SIM is metered** (config flag `CELL_METERED=1`); adaptive interval (longer when stable). Bind test traffic to the WAN under test via source routing. Bench-verify per WAN. **Commit.** *(Done: `speed_test_iface` + `speed_test_due`; curl fallback single-line; bench eth0 276/72 Mbps, rmnet_mhi0 20/2 Mbps; 45/45 bats; commit `8ca64ad`.)*
|
||||
|
||||
### Task 3.3: Starlink status
|
||||
**Files:** `starlink.sh`. Port `aiwanbal-starlink.sh`: `grpcurl -plaintext 192.168.100.1:9200 SpaceX.API.Device.Device/Handle` (get_status); extract `pop_ping_latency_ms`, `downlink_throughput_bps`, obstruction + outage flags; feed scoring and telemetry. If `grpcurl` absent (per recon), bundle a static arm64 binary in the package. Bench-verify against the real dish. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — SIM manager
|
||||
|
||||
### Task 4.1: Signal read ✅ Done
|
||||
**Files:** `modem-sim.sh`. Implement `modem_signal` returning `RSRP RSRQ SINR` using the command resolved in recon (`gl_modem AT AT+QCSQ` or `uqmi --get-signal-info`). Parse with a fixture-backed unit test. **Commit.**
|
||||
|
||||
### Task 4.2: SIM auto-switch (with hysteresis) ✅ Done
|
||||
**Files:** extend `modem-sim.sh` + daemon. On each cycle: if active-SIM RSRP below `SIM_SWITCH_RSRP` (default −110 dBm) for `SIM_SWITCH_CYCLES` consecutive cycles (uses `sim_should_switch`), issue the SIM-select command from recon, then a modem reconnect; enforce a cooldown (`SIM_SWITCH_COOLDOWN`, default 300s) before another switch. **Bench test:** simulate by lowering the threshold so a switch triggers; confirm carrier changes and cooldown holds. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Telemetry agent
|
||||
|
||||
### Task 5.1: GPS read ✅ Done
|
||||
**Files:** `telemetry.sh`. `gps_fix` returns `lat lon fix_quality` via the recon GPS command. Fixture-backed parser test. **Commit.**
|
||||
|
||||
### Task 5.2: Collect + POST to hub ✅ Done
|
||||
**Files:** extend `telemetry.sh`. Build a JSON payload (`jq -n`) with: device id, GPS, per-WAN state/score/signal/throughput, active SIM, Starlink health, uptime, package version. POST to `http://10.88.0.1:8080/api/telemetry` over the WG tunnel. **Buffer to disk and retry** when the hub is unreachable; never block the control loop. Interval configurable (`TELEMETRY_INTERVAL`, default 60s).
|
||||
- [ ] Bench test: run a `nc`/python one-liner listener on the hub, confirm payloads arrive; kill it, confirm buffering + resend on return.
|
||||
- [ ] **Commit.** *(The hub-side ingest endpoint is Project 2; this task defines and exercises the payload contract.)*
|
||||
|
||||
---
|
||||
|
||||
## Phase 6 — Daemon integration
|
||||
|
||||
### Task 6.1: Wire the control loop ✅ Done
|
||||
**Files:** `daemon.sh`. Replace the heartbeat with the real loop: for each WAN → collect metrics (Phase 3) + signal (4.1) + starlink (3.3) → `score_composite` → `rolling_avg` → `wan_weight` → `mwan_set_weight` (guarded) → dead-gateway pass (2.2) → SIM decision (4.2) → `telemetry` (5.2). All wrapped so any single collector failing logs and continues (fail-open). **Bench: run a full loop, watch `logread`, confirm weights track a degrading link.** **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 7 — Branded local UI
|
||||
|
||||
### Task 7.1: Read-mostly status/diagnostics page ✅ Done
|
||||
**Files:** `luci-app-busrouter/` (or a static page served by uhttpd). Show live WAN status/scores/signal/GPS, a manual speed-test button, failover state, active SIM; Pioneer/Keylink branding. Emergency local overrides (force WAN, force SIM) behind the admin login. Reads the same state files the daemon writes to `/tmp/busrouter/`. Bench-verify in a browser over LAN. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 8 — Device identity & provisioning profile
|
||||
|
||||
### Task 8.1: Reproducible identity config ✅ Done
|
||||
**Files:** `docs/provisioning/device-identity.md` + a `provision.sh` that sets: SSID `X0000`/`Pioneer123` (2.4G+5G), admin `pioadmin`/`Pioneer321!`, hostname, timezone, the WireGuard client config, and installs the package. This becomes the seed for Project 4 (mass rollout). Apply to the bench unit end-to-end from a factory-ish state. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 9 — Field pilot
|
||||
|
||||
### Task 9.1: Three-bus validation checklist ✅ Done
|
||||
**Files:** `docs/pilot/checklist.md`. Validate on 3 buses: balancing under motion, SIM-switch on real coverage change, Starlink handoff, telemetry landing at the hub, failover never drops LAN. Capture logs. Sign-off gate before scaling toward 194. **Commit.**
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
**Spec coverage:** §3 network design → Phases 2/4; §5 components 1–7 → Phases 1/2/3/4/5/6/7; §6 mgmt plane → Task 0.1 + 5.2; §7 error handling (fail-safe/hysteresis) → procd respawn (0.3), `weight_changed`/`sim_should_switch` (1.4), fail-open loop (6.1), telemetry buffering (5.2); §8 identity → Phase 8; §9 testing → bats (Phase 1) + bench + Phase 9 pilot. All covered.
|
||||
|
||||
**Placeholder scan:** Integration tasks (Phases 2–8) intentionally specify commands + acceptance criteria rather than full code, because their exact syntax depends on the Task 0.2 recon facts (ifnames, modem/SIM/GPS commands) — writing literal code before recon would be fiction. The pure-logic tasks (Phase 1) carry complete, runnable code + tests. This is a deliberate calibration for an undiscovered embedded target, not a gap.
|
||||
|
||||
**Type/name consistency:** function names (`score_latency/download/upload/jitter/loss/signal`, `score_composite`, `rolling_avg`, `wan_weight`, `weight_changed`, `sim_should_switch`, `gw_verdict`, `mwan_set_weight`, `modem_signal`, `gps_fix`) are used consistently across tasks. Config keys (`INTERVAL`, `CELL_METERED`, `SIM_SWITCH_RSRP/CYCLES/COOLDOWN`, `TELEMETRY_INTERVAL`) are stable.
|
||||
@@ -0,0 +1,165 @@
|
||||
# Pioneer Bus Router — Project 1: On-Device Multi-WAN Balancer
|
||||
|
||||
**Status:** Design approved (2026-06-30) · **Owner:** Keylink IT · **Client:** Pioneer Coach Buses
|
||||
**Target hardware:** GL.iNet GL-XE3000 (Puli AX) · OpenWrt 21.02-SNAPSHOT · MediaTek MT7981 (ARMv8) · kernel 5.4
|
||||
|
||||
---
|
||||
|
||||
## 1. Context & Goal
|
||||
|
||||
Pioneer Coach Buses is replacing its **eyeride** solution with a fleet of GL-XE3000 routers (3 in the
|
||||
next month, **194 eventually**). Each bus needs **high-quality, highly stable internet** from multiple
|
||||
uplinks, with automatic, intelligent use of whichever links are healthy.
|
||||
|
||||
This project (Project 1) is the **on-router software**: the multi-WAN balancing brain plus the telemetry
|
||||
agent that later feeds the central fleet console (Project 2). It is a **port** of the existing Synology
|
||||
SRM package [`aiwanbal`](https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer) to OpenWrt — we reuse
|
||||
the *logic*, rewrite the *platform layer*.
|
||||
|
||||
### Scope decomposition (whole program)
|
||||
1. **Project 1 — On-router multi-WAN balancer (this spec)**
|
||||
2. Project 2 — Central fleet console (map, status, reporting, remote config, OTA) — *separate spec*
|
||||
3. Project 3 — Starlink deep integration & reporting — *folded partly into P1, extended in P2*
|
||||
4. Project 4 — Fleet provisioning / mass rollout of 194 units — *separate spec*
|
||||
|
||||
---
|
||||
|
||||
## 2. Hardware constraints (load-bearing)
|
||||
|
||||
- **Single 5G modem, dual-SIM *single-standby*.** The Quectel RM520N-GL has two SIM slots but only **one
|
||||
SIM is active at a time**. Therefore a single XE3000 **cannot** simultaneously balance T-Mobile *and*
|
||||
Verizon. Decision: run **one active cellular WAN** and treat the second SIM as **failover**.
|
||||
- **Ports:** 2.5 GbE WAN, 1 GbE LAN, USB 2.0, microSD. GNSS/GPS available via the modem.
|
||||
- **Upstream observed in the field unit:** cellular `rmnet_mhi0` on a **carrier CGNAT address**
|
||||
(e.g. `21.188.231.63/25`) — no public inbound, which is why remote management uses an **outbound**
|
||||
WireGuard tunnel (see §6).
|
||||
|
||||
---
|
||||
|
||||
## 3. Per-bus network design
|
||||
|
||||
| WAN | Link | Role |
|
||||
|-----|------|------|
|
||||
| **WAN1 — Cellular 5G** | internal modem, dual-SIM single-standby | Balanced. Auto-switch to the other SIM when the active carrier degrades past threshold. |
|
||||
| **WAN2 — Starlink** | 2.5 GbE WAN port → Starlink | Balanced. |
|
||||
|
||||
- **Normal state:** 5G + Starlink **balanced together** (2 live WANs) via `mwan3`.
|
||||
- **Degradation paths:** bad SIM → switch SIM (hysteresis); bad cellular overall → lean on Starlink;
|
||||
bad Starlink → lean on cellular. Never strand a bus offline.
|
||||
- **Downstream:** LAN → Omada fully-managed switch. Roof antenna provides 5G + GPS + Wi-Fi.
|
||||
|
||||
---
|
||||
|
||||
## 4. Platform approach
|
||||
|
||||
- **Base OS:** **GL.iNet stock firmware** (OpenWrt 21.02). We do **not** build a custom firmware image —
|
||||
this keeps GL's security/stability updates flowing and the Quectel modem fully supported.
|
||||
- **Delivery:** our software ships as a **branded `opkg` package + `init.d` service**, installed on top.
|
||||
Persists across reboots, OTA-updatable, survives GL firmware updates when done carefully.
|
||||
- **Balancing engine:** **`mwan3`** (OpenWrt standard weighted multi-WAN + failover). Our daemon tunes
|
||||
`mwan3` member weights instead of Synology SmartWAN slots / `srm.json`.
|
||||
- **Branding:** applied to the layer customers see — local dashboard, SSID/login identity, hostname, and
|
||||
the fleet console — **not** a full firmware rebrand.
|
||||
|
||||
### Reuse map (from `aiwanbal`)
|
||||
| Reused logic | Rewritten platform glue |
|
||||
|---|---|
|
||||
| 6-factor scoring + rolling average | SmartWAN weights → `mwan3` member weights |
|
||||
| 3-layer dead-gateway detection (DNS→ping quorum→HTTP) | SRM `srm.json`/`tctool` → `mwan3`/`tc` |
|
||||
| Adaptive, data-cap-aware speed testing (Ookla + curl) | `synopkg` lifecycle → `opkg` + `init.d` |
|
||||
| Modem signal monitoring (RSRP/RSRQ/SINR) | SRM modem drivers → GL modem manager / QMI / AT |
|
||||
| Starlink gRPC helper | SRM-embedded UI → LuCI/GL plugin |
|
||||
|
||||
---
|
||||
|
||||
## 5. Components (each independently testable)
|
||||
|
||||
1. **Scoring daemon** (`busrouter-daemon`) — polls each WAN's latency / throughput / jitter / loss /
|
||||
signal, computes the 6-factor score with rolling 3-sample average. Adapted from 2 generic WANs to
|
||||
**cellular + Starlink**. Writes desired weights.
|
||||
2. **WAN controller** (`busrouter-mwan`) — translates scores → `mwan3` member weights + failover; ports
|
||||
the 3-layer dead-gateway check. **Hysteresis** so weights don't flap; minimizes conntrack-flush resets.
|
||||
3. **SIM manager** (`busrouter-modem`) — reads RSRP/RSRQ/SINR via the GL modem manager; **auto-switches
|
||||
the active SIM** when the live carrier degrades past threshold, with cooldown/hysteresis to prevent
|
||||
ping-pong. ("Another 5G when signal gets bad.")
|
||||
4. **Starlink integration** (`busrouter-starlink`) — gRPC to the dish (status, obstruction, outage),
|
||||
feeding both scoring and telemetry.
|
||||
5. **Speed-test engine** (`busrouter-speedtest`) — Ookla CLI + curl fallback, adaptive interval,
|
||||
**data-cap-aware on cellular** (metered SIM skips auto-tests).
|
||||
6. **Telemetry agent** (`busrouter-agent`) — collects **GPS (modem GNSS)**, WAN states/scores, active SIM,
|
||||
signal, throughput, Starlink health, uptime; posts to the fleet hub over the **WireGuard tunnel**, and
|
||||
pulls remote config / OTA triggers. Defines the **contract** Project 2 consumes.
|
||||
7. **Branded local UI** — lightweight LuCI / GL-plugin status + diagnostics page (live WAN status, scores,
|
||||
signal, GPS, manual speed test, failover state). **Read-mostly** — real control is centralized — plus
|
||||
emergency local overrides for field techs.
|
||||
|
||||
---
|
||||
|
||||
## 6. Management plane (provisioned 2026-06-30)
|
||||
|
||||
Remote access and telemetry ride a single **WireGuard hub**, which **doubles as the future fleet console
|
||||
host** (Project 2). Buses dial **out** as WG clients — CGNAT-safe.
|
||||
|
||||
```
|
||||
┌──────── WireGuard hub: kit-fleet-hub ────────┐
|
||||
│ DO droplet · 167.172.237.162 · nyc3 │
|
||||
│ s-1vcpu-1gb ($6/mo, resize as fleet grows) │
|
||||
│ wg0 10.88.0.1/24 · ListenPort 51820 │
|
||||
│ hub pubkey 4QAauZ9gS/wZ/u0Wf66OcJFGo4er... │
|
||||
└──────────────────────────────────────────────┘
|
||||
▲ ▲ ▲
|
||||
bus01 = 10.88.0.2 LAB (jump via hub) fleet console (later, same box)
|
||||
(GL-XE3000) ssh -J hub → router 10.88.0.1
|
||||
```
|
||||
|
||||
- **Per-bus WG client config** stored on the hub at `/root/busXX-wireguard.conf`, imported into GL's
|
||||
WireGuard client UI. `AllowedIPs = 10.88.0.0/24` only — **user internet stays on the balanced WANs**;
|
||||
the tunnel is pure out-of-band management.
|
||||
- **LAB access:** the LAB SSHes routers via the hub as a **jump host** (`ssh -J hub root@10.88.0.2`),
|
||||
key-based once the LAB public key is installed on the router.
|
||||
- **Local dev access:** local Claude Code on the operator PC deploys/tests over LAN (`192.168.8.1`) — no
|
||||
tunnel needed for the fast build/test loop.
|
||||
- **Division of labor:** local PC = build/deploy over LAN; LAB = remote SSH + management over WireGuard.
|
||||
|
||||
---
|
||||
|
||||
## 7. Config & error handling
|
||||
|
||||
- Master config under `/etc/busrouter/*.conf`, **persists across upgrades**.
|
||||
- **Fail-safe:** if our daemon dies, both WANs stay up via `mwan3` defaults — a software fault never takes
|
||||
a bus offline.
|
||||
- **Hysteresis** on both SIM-switching and weight changes; cooldown windows to damp oscillation.
|
||||
- Telemetry agent degrades gracefully when the hub is unreachable (buffer + retry); never blocks routing.
|
||||
|
||||
---
|
||||
|
||||
## 8. Device identity (provisioning defaults)
|
||||
|
||||
- SSID `X0000` / `Pioneer123` (2.4G + 5G), admin user `pioadmin` / `Pioneer321!`, branded hostname.
|
||||
- Captured as a **reproducible provisioning profile** so all units are identical (formalized in Project 4).
|
||||
- Router admin (current test unit): web `pioadmin` / `Pioneer321!`; SSH/root password `kitPLANE1!!`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Testing strategy
|
||||
|
||||
- **Unit:** shell logic (scoring, hysteresis, dead-gateway decision) tested in a VM / dev container.
|
||||
- **Bench:** single XE3000 test unit (`bus01`, WG `10.88.0.2`) — iterate via local LAN push (CGI/HTML hot,
|
||||
daemon needs service restart), mirroring `aiwanbal`'s live-push workflow.
|
||||
- **Field pilot:** 3 buses before scaling. Validate failover, SIM-switch, Starlink handoff, telemetry.
|
||||
|
||||
---
|
||||
|
||||
## 10. Out of scope (later projects)
|
||||
|
||||
Fleet console UI/map/reporting, OTA distribution infrastructure, mass provisioning of 194 units, Omada
|
||||
switch configuration, Starlink account/dish provisioning.
|
||||
|
||||
## 11. Decisions locked
|
||||
|
||||
- **Cellular:** one active SIM, the other as failover (no simultaneous dual-carrier). ✅
|
||||
- **Engine:** `mwan3`. ✅
|
||||
- **Base:** GL.iNet stock firmware + branded `opkg` layer (no custom firmware image). ✅
|
||||
- **Mgmt transport:** WireGuard, buses dial out to a shared hub that becomes the fleet console host. ✅
|
||||
- **Local UI:** branded, read-mostly; control centralized. ✅
|
||||
- **Hub sizing:** `s-1vcpu-1gb` now (~3 buses), resize up as the fleet grows. ✅
|
||||
Reference in New Issue
Block a user