allen/kit-busrouter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: kit-busrouter v2.0 — complete Synology + GL fleet router platform

Browse Source 
Includes:
- package/ GL-XE3000 kit-busrouter (opkg)
- scripts/provision.sh (GL) and provision-synology.sh (Synology)
- syno-balance/ — new WAN balancer replacing aiwanbal (SmartWAN adapter)
- kit-connect/ — unified connectivity SPK (Tailscale + reverse SSH)
- docs/deployment/synology-rt2600ac-checklist.md — 62-point checklist
- docs/provisioning/device-identity.md — fleet identity spec
- docs/pilot/checklist.md — field pilot validation
- x4078_20260721.dss — reference config backup

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Allen McGhan
2026-07-22 00:07:14 +00:00
commit f15ac69925
41 changed files with 5083 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/recon/2026-06-30-xe3000-environment.md
+384
View File
@@ -0,0 +1,384 @@
# GL-XE3000 Device Reconnaissance (Task 0.2)
**Date:** 2026-06-30
**Unit:** bench `bus01` — GL.iNet GL-XE3000 (Puli AX)
**Access path:** LAB → `kit-fleet-hub` (DO droplet 167.172.237.162, WG 10.88.0.1) → jump SSH
`ssh -i /root/.ssh/busfleet root@10.88.0.2` (router WG IP 10.88.0.2).
**Access confirmed:** yes — `ubus call system board` returns `"model": "GL.iNet GL-XE3000"`.
---
## 1. Resolved facts table (feeds later plan tasks)
| Fact | Value |
|------|-------|
| Cellular ifname (WAN1 member) | **`rmnet_mhi0`** (QMI proto `qcm`, control node `/dev/mhi_QMI0`, UCI iface `modem_0001`, APN `fast.t-mobile.com`) |
| Starlink WAN ifname (WAN2) | **`eth0`** (UCI iface `wan`, proto `dhcp`). NOTE: currently `DOWN`/`NO-CARRIER` — no dish attached to bench unit. `eth1` = LAN (`br-lan`). |
| mwan3 present? | **NO** — `/etc/config/mwan3` absent, `mwan3` package not installed → Task 3.0 / Phase 2 must `opkg install mwan3`. GL ships its own multiwan (`gl-sdk4-ui-multiwan`, `kmod-gl-sdk4-kmwan`, ubus `hotplug.kmwan`) which we are NOT using. |
| Signal read command | **Preferred:** `ubus call modem.signal get_signals '{"time":1}'` → JSON `{rsrp,rsrq,sinr,slot,network_type,strength,mode}`. **AT fallback:** `gl_modem -B 1-1.2 AT AT+QCSQ` → `+QCSQ: "NR5G",<rssi>,<sinr>,<rsrp>`. **`uqmi` is NOT usable** (see Implications). |
| SIM select command | Read: `gl_modem -B 1-1.2 AT AT+QUIMSLOT?` → `+QUIMSLOT: 1`. Switch: `gl_modem -B 1-1.2 AT AT+QUIMSLOT=2` (confirmed slot-query works; switch command not exercised to avoid dropping the live link). |
| GPS read command | `gl_modem -B 1-1.2 AT AT+QGPSLOC=2` — but GNSS must be enabled first: `AT+QGPS=1`. Currently `AT+QGPS?` = `0` (disabled), so `QGPSLOC` returns `+CME ERROR: 505` (no fix). |
| grpcurl available? | **NO** — `command -v grpcurl` = MISSING → Phase 3 (Starlink) must bundle a static arm64 `grpcurl` binary in the opkg package. |
| Firewall zone of WG client iface | **`wgclient1`** — its own dedicated zone named `wgclient1` (network `wgclient1`, input DROP). NOT `wan`. The access-fix rule is `firewall.@rule[22]` name `Allow-WG-mgmt-SSH`, `src='wgclient1'`, `src_ip='10.88.0.0/24'`, dest_port 22, ACCEPT. |
| Modem USB bus (for `gl_modem -B`) | **`1-1.2`** (Quectel RM520N-GL, 4 USB interfaces at `1-1.2:1.0-1.3`) |
---
## 2. Raw command outputs
### `ubus call system board`
```json
{
"kernel": "5.4.211",
"hostname": "GL-XE3000",
"system": "ARMv8 Processor rev 4",
"model": "GL.iNet GL-XE3000",
"board_name": "glinet,xe3000-emmc",
"release": {
"distribution": "OpenWrt",
"version": "21.02-SNAPSHOT",
"target": "mediatek/mt7981"
}
}
```
### `/etc/config/network` (relevant excerpts)
```
config device # LAN bridge
option name 'br-lan'
list ports 'eth1'
config interface 'lan'
option device 'br-lan'
option ipaddr '192.168.8.1'
config device
option name 'eth0'
config interface 'wan' # Starlink WAN port
option device 'eth0'
option proto 'dhcp'
option metric '10'
config interface 'modem_0001' # Cellular / 5G
option apn 'fast.t-mobile.com'
option ip_type 'IPV4V6'
option proto 'qcm'
option device '/dev/mhi_QMI0'
option metric '40'
option iccid '8901260417776090740F'
config interface 'wgclient1' # WireGuard out-of-band mgmt
option proto 'wgclient'
option config 'peer_2001'
option ip4table '1001'
```
(Also present but unused/disabled: `wwan`, `secondwan`, `guest`, various GL vpn policy rules.)
### `/etc/config/mwan3`
```
NO mwan3
```
### `opkg list-installed` (filtered: mwan3|grpc|speedtest|modem|gl-|uqmi|quectel)
Notable entries (full list captured in session):
```
gl-sdk4-modem git-2025.290.37125-7c3efcc-1
gl-sdk4-ui-multiwan git-2025.259.31350-8a2a5c6-1
gl-sdk4-ui-modemsignallog git-2024.303.14204-4241be7-1
kmod-gl-pcie-modem 5.4.211+20221009-1
kmod-gl-sdk4-kmwan 5.4.211+git-2025.304.07170-bb470c0-1
uqmi 2020-11-22-0a19b5b7-3
```
No `mwan3`, no `grpcurl`, no `speedtest` (Ookla) package installed.
### `ip -br addr`
```
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 DOWN # Starlink WAN — no carrier
eth1 UP # LAN member
ra0 UP # 2.4G wifi
rmnet_mhi0 UP 100.112.80.45/30 # Cellular (CGNAT)
br-lan UP 192.168.8.1/24
wgclient1 UNKNOWN 10.88.0.2/32 # WG mgmt tunnel
```
### `ip route`
```
default via 100.112.80.46 dev rmnet_mhi0 proto static src 100.112.80.45 metric 40
100.112.80.44/30 dev rmnet_mhi0 proto static scope link metric 40
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
```
### Modem device nodes
```
/dev/mhi_BHI /dev/mhi_DIAG /dev/mhi_DUN /dev/mhi_LOOPBACK /dev/mhi_QMI0
/dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2 /dev/ttyUSB3
```
No `/dev/cdc-wdm*` — modem is on the **MHI/PCIe** transport, not USB-CDC.
### Tool availability
```
gl_modem: /usr/bin/gl_modem
uqmi: /sbin/uqmi
grpcurl: MISSING
mmcli: MISSING
gpsd: MISSING
```
### `gl_modem` usage
```
gl_modem -B <bus> AT <AT_CMD> # send AT command; bus for this unit = 1-1.2
```
### Signal — ubus (preferred)
```json
// ubus call modem.signal get_signals '{"time":1}'
{
"signals": [{
"sinr": 13, "slot": 1, "network_type": "NR5G-SA",
"timestamp": 1782917019, "strength": 4,
"rsrp": -96, "rsrq": -10, "mode": 5
}]
}
```
`ubus -v list modem.signal` → methods `get_signals {time:Integer}`, `upload_cloud_signals {action:String}`.
### Signal — AT fallback
```
gl_modem -B 1-1.2 AT AT+QCSQ
+QCSQ: "NR5G",-97,14,-10 # <sysmode>,<rssi>,<sinr>,<rsrp>
```
### SIM slot
```
gl_modem -B 1-1.2 AT AT+QUIMSLOT?
+QUIMSLOT: 1 # active SIM = slot 1
```
### GPS / GNSS
```
gl_modem -B 1-1.2 AT AT+QGPS?
+QGPS: 0 # GNSS engine currently OFF
gl_modem -B 1-1.2 AT AT+QGPSLOC=2
+CME ERROR: 505 # no fix (engine not enabled)
```
### Starlink WAN port link
```
eth0 DOWN <NO-CARRIER,BROADCAST,MULTICAST,UP>
```
### Firewall (WG mgmt zone + access-fix rule)
```
firewall.wgclient1 = zone
firewall.wgclient1.name='wgclient1'
firewall.wgclient1.network='wgclient1'
firewall.wgclient1.input='DROP'
firewall.@rule[22].name='Allow-WG-mgmt-SSH'
firewall.@rule[22].src='wgclient1'
firewall.@rule[22].src_ip='10.88.0.0/24'
```
Zone[1] `wan` groups: `wan wan6 wwan secondwan modem_0001 modem_0001_4` (input DROP).
---
## 3. Implications for the plan
1. **mwan3 is NOT installed (plan flagged this as possible).** `/etc/config/mwan3` is absent and no `mwan3`
opkg is present. Phase 2 / Task 3.0 must `opkg install mwan3` (verify it exists in GL's feed for
`mediatek/mt7981` @ 21.02-SNAPSHOT; if not, add the upstream OpenWrt 21.02 feed or bundle it).
GL ships its own `kmwan`/multiwan stack (`kmod-gl-sdk4-kmwan`, `gl-sdk4-ui-multiwan`, ubus
`hotplug.kmwan`) — the spec locks on `mwan3`, so we intentionally do NOT use GL's kmwan. Watch for
conflict/overlap between the two multiwan systems on the same box (routing marks, hotplug).
2. **Firewall zone assumption in Task 0.1 was WRONG.** Task 0.1 Step 1 guessed `src='wan'` (with a
"verify / adjust if wgclient" note). The WG client interface is actually in its **own zone `wgclient1`**,
and that is what the working fix rule uses (`@rule[22] src='wgclient1' src_ip='10.88.0.0/24'`). Any
future mgmt/telemetry firewall rules must target zone **`wgclient1`**, not `wan`.
3. **`uqmi` is effectively unusable for signal reads.** `uqmi -d /dev/mhi_QMI0 --get-signal-info` **hangs
(timeout)** — GL's modem manager holds the QMI control channel (single-client QMUX). Use the GL ubus
path `modem.signal get_signals` (clean JSON, no device contention) as the primary signal source, with
`gl_modem -B 1-1.2 AT AT+QCSQ` as the AT fallback. **Phase 4 Task 4.1 should target ubus first**, not
uqmi as the plan's example suggested.
4. **`gl_modem` requires a `-B <bus>` argument** (the plan's examples `gl_modem AT AT+...` omit it and would
fail). The bus on this unit is **`1-1.2`**. This must be discovered per-unit (parse from
`/sys/bus/usb/devices` or GL config) rather than hard-coded, since it could differ across units —
record for the provisioning profile (Phase 8).
5. **GPS/GNSS is present but OFF by default.** `AT+QGPS?` = 0, so `AT+QGPSLOC=2` errors `+CME ERROR: 505`.
Phase 5 Task 5.1 (`gps_fix`) must **enable GNSS first** (`AT+QGPS=1`, one-time / on boot) and handle the
"no fix yet" (error 505/516) state gracefully before location is available. GPS command form is
confirmed as Quectel `AT+QGPSLOC=2` (decimal degrees).
6. **`grpcurl` absent (plan flagged this).** Phase 3 Task 3.3 must bundle a static **arm64 (ARMv8)**
`grpcurl` binary in the opkg package for Starlink gRPC.
7. **No Starlink dish on the bench unit** (`eth0` NO-CARRIER). WAN2 (Starlink) integration and any
two-WAN balancing test cannot be validated on this bench unit as-is — need a dish on `eth0`, or test
WAN2 with a substitute uplink. The cellular WAN (`rmnet_mhi0`) is the only live uplink right now, and
it carries the WG mgmt tunnel (CGNAT `100.112.80.45/30`, T-Mobile).
8. **Ookla speedtest CLI not installed** — Phase 3 Task 3.2 must bundle/install it (or rely on the curl
fallback), consistent with the plan's fallback design.
---
## 4. Task 2.0 — mwan3 / kmwan changeover (2026-07-01)
Performed on bench `bus01` via the WG mgmt path. This section feeds the Phase 8 provisioning profile.
### GL multiwan (kmwan) — exact init name + stand-down
- **GL multiwan init script name: `kmwan`** — `/etc/init.d/kmwan` (the ONLY `mwan|kmwan` init on the box;
confirms recon §1 `kmod-gl-sdk4-kmwan`). It is a `/etc/rc.common` script, `START=21`, sourcing
`/lib/functions/kmwan.sh`. It was **enabled** but had **no running daemon** (routing is managed via
hotplug scripts / a single static default route, not a persistent process).
- **Exact commands used to stand it down** (Phase 8 must run these):
```
/etc/init.d/kmwan disable # rc=0 → NOW_DISABLED (won't start on boot)
/etc/init.d/kmwan stop # rc=0
```
Verify with `/etc/init.d/kmwan enabled` (returns non-zero = disabled) and
`ps w | grep -i mwan | grep -v grep` (no kmwan process).
### mwan3 install
- Installed from **GL's own feed** (no third-party/upstream feed needed):
`opkg update && opkg install mwan3` → **`mwan3 2.8.15-2`** (`mwan3_2.8.15-2_all.ipk`,
feed `glinet_gli_pub` @ v21.02.3). Binary at `/usr/sbin/mwan3`.
- mwan3 ships a **stock default `/etc/config/mwan3`** (dated 2022) with interfaces
`wan/wwan/tethering` (+v6). **NONE of these match our live cellular uplink** (`rmnet_mhi0`, UCI iface
`modem_0001`), so mwan3 is currently **inert**: `mwan3 status` shows every interface "offline and
tracking is down", **no active policies, no active user rules**. mwan3's `mwan3rtmon` route-monitor
daemon runs but does not touch the default route (member config is a later task).
### Safety / recovery notes for Phase 8
- **`setsid` and `nohup` are BOTH ABSENT** on this GL firmware (BusyBox build). For detached/background
processes use **`start-stop-daemon -S -b -x <prog> -- <args>`** (present at `/sbin/start-stop-daemon`).
Note: `start-stop-daemon -b -x /bin/sh` will refuse to launch if it matches an already-running
`/bin/sh` ("… is already running") — give the watchdog a unique interpreter/exec name or wrap it so the
`-x` match is unique.
- **Post-change verification (mgmt access + default route SURVIVED):** after `kmwan disable/stop`, a fresh
probe returned `"model": "GL.iNet GL-XE3000"`, default route unchanged
(`default via 100.116.118.126 dev rmnet_mhi0 … metric 40` — CGNAT src rotated from the recon-day
`100.112.80.45` to `100.116.118.125`, normal T-Mobile lease change), and `wgclient1 10.88.0.2/32` up.
The WG mgmt routing rules (`from/to 10.88.0.2 lookup 1001`, table 1001) are intact. **Disabling kmwan
did not disturb routing** — expected, since kmwan had no active daemon and the live path is a single
static default on `rmnet_mhi0`.
---
## 5. Task 2.1 pre-check — second uplink (ATT Fiber on eth0) added (2026-07-01)
**Status: BLOCKED (device shell unreachable) — but the load-bearing WG-underlay question is answered
from hub-side WireGuard evidence.**
### What happened
The operator plugged ATT Fiber into `eth0` (WAN2 stand-in for Starlink). Immediately afterward the
router's SSH shell (via the WG mgmt tunnel `10.88.0.2:22`) became **unreachable** from the fleet hub,
even though the WireGuard tunnel itself is **healthy**. All on-device commands (eth0 lease, `ip route`
metrics, `ip route get`, `ip rule`, `mwan3 status`) therefore **could not be run**. No device state was
mutated. The facts below are what the hub can observe about the tunnel from `10.88.0.1` (wg0).
### THE KEY ANSWER — WG mgmt path now rides eth0 (ATT Fiber)
The WireGuard peer **endpoint changed** from the T-Mobile CGNAT `100.x.x.x` seen in §2/§4 to a **public,
non-CGNAT address**, which is the ATT Fiber public IP:
```
# hub: wg show wg0 endpoints
0Xn7aGlyxHQ/X9uo8lsOeI3EFA7KO+zlO+qIhrzsAF8= 99.163.124.171:44881
# latest-handshakes: 6 s ago (fresh); transfer: 335 KiB rx / 244 KiB sent (growing)
```
`99.163.124.171` is a routable public IP (not `100.64.0.0/10` CGNAT), i.e. the router is now sourcing its
WG handshakes out **eth0 / ATT Fiber**, not `rmnet_mhi0` / cellular. **The WG mgmt underlay rides eth0
(ATT Fiber).** Corollary: **downing `eth0` would sever the WG mgmt path / SSH access.**
### Safe failover-test direction
- **DO NOT bring down `eth0` (ATT Fiber)** for the failover drill — it carries the WG underlay; downing
it cuts mgmt/SSH.
- **`rmnet_mhi0` (cellular) is the SAFE interface to bring down** for the mwan3 failover test — it does
NOT currently carry the WG underlay handshake. (This is the *inverse* of the pre-fiber state in §2/§4,
where cellular carried mgmt and eth0 was NO-CARRIER.)
### Current mgmt-reachability fault (flag for operator — needs on-device follow-up)
The WG control plane is up but **no in-tunnel application traffic returns from the router**:
```
# hub, over the tunnel:
ip route get 10.88.0.2 -> 10.88.0.2 dev wg0 src 10.88.0.1 (route correct)
ping -c3 10.88.0.2 -> 100% loss (even 56-byte payloads)
nc -vz 10.88.0.2 22 -> timed out ("Operation now in progress")
# 4 s window: wg 'sent' climbed 244308 -> (SYN retransmits reach router), 'rx' FLAT
```
Interpretation: the router **receives** hub traffic (encrypted `sent` counter grows) but **replies with
nothing** — its in-tunnel return path for ICMP/TCP is broken. Most likely the policy-routing that steers
tunnel replies (table `1001`, `from/to 10.88.0.2 lookup 1001`, per §4) or the WG interface's reverse-path
was disturbed when the fiber default route (metric 10) came up and displaced cellular (metric 40) as the
preferred default. This must be diagnosed/repaired **on-device via a console/LAN path** (the WG path is
the very thing that's down) before Task 2.1 routing changes proceed. Not attempted here (read-only,
no-mutate scope).
### Facts NOT captured (require device shell — blocked)
- eth0/wan DHCP lease (ATT subnet / gateway / netmask), eth0 carrier flag from the device side.
- Full `ip route` default table with metrics (expected eth0 metric 10 preferred over cellular metric 40 —
strongly implied by the endpoint move to fiber, but not directly read).
- `rmnet_mhi0` current CGNAT IP, `ip route get 167.172.237.162` / `ip route get 99.163.124.171`.
- `mwan3 status` / `mwan3 interfaces` view of the two ifaces, `ip rule` table listing.
Re-run §5 in full once console/LAN access to `bus01` is restored and the WG mgmt path is repaired.
---
## 6. WG mgmt tunnel root cause + fix (2026-07-01)
**Symptom:** after ATT Fiber plugged into eth0 (metric 10 preferred default), WG handshakes were
fresh but all in-tunnel data (ICMP/SSH from 10.88.0.1 to 10.88.0.2) returned 100% loss.
**Root cause:** `mwan3` adds all directly-connected networks to the `mwan3_connected` ipset.
10.88.0.0/24 is a directly-connected route (wgclient1), so mwan3's iptables `mwan3_connected`
chain marks packets destined for that prefix with `0x3f00`. The ip rule
`9000: not from all fwmark 0/0xf000 lookup main` then routes them via the **main table**,
which has **no route for 10.88.0.0/24** (wgclient1 routes are only in table 1001 due to
`ip4table='1001'` in UCI). The packet is silently dropped — never reaches WG.
**Fix (applied 2026-07-01):**
Add an ip rule at priority 99 (before any mwan3 fwmark rule) routing packets to 10.88.0.0/24
via table 1001:
```
ip rule add to 10.88.0.0/24 priority 99 lookup 1001
```
Persisted as a UCI network rule so netifd recreates it on every boot:
```
uci add network rule
uci set network.@rule[-1].lookup='1001'
uci set network.@rule[-1].dest='10.88.0.0/24'
uci set network.@rule[-1].priority='99'
uci commit network
```
Also applied for robustness: `wgclient1 mtu=1280`, rp_filter=2 persisted via /etc/sysctl.conf
and /etc/hotplug.d/iface/99-busrouter-rpfilter (re-asserts on WAN flap).
**Failover note:** `scripts/diag/wg-mgmt-diagnose.sh` and `scripts/diag/wg-mgmt-fix.sh` in
this repo implement the fix. The underlying keepalive (`persistent_keepalive=25`) was already
configured from a prior session; the new ip rule was the missing piece.
**Phase 8 provisioning profile must include** this UCI rule — without it, any device with
mwan3 configured will have the WG mgmt tunnel silently drop in-tunnel data.
## 7. Task 2.1 bench-test findings (2026-07-01)
mwan3 configured with `modem_0001` (cellular) + `wan` (fiber/eth0), policy `balanced`.
Both interfaces tracked online (1.1.1.1 + 8.8.8.8, reliability 1, down=3, up=3 x 5s interval).
**Failover drill result:**
- `ip link set rmnet_mhi0 down` -> mwan3 declares `modem_0001 error` within ~15s (checked)
- Policy shifts to `wan (100%)` automatically (checked)
- WG mgmt tunnel (riding eth0) stays alive throughout failover (checked)
- **Recovery gotcha:** `ip link set rmnet_mhi0 up` does NOT restore the QMI data session.
Must use `ifdown modem_0001 && ifup modem_0001` to reconnect. After ~20s both interfaces
return to `balanced: wan(50%) modem_0001(50%)` (checked)
**Phase 9 pilot:** fiber failover (bring fiber down, WG must ride cellular) should be tested
once the provisioning robustness fix (section 6 ip rule) is confirmed to survive both uplink combos.