# GL-XE3000 Device Reconnaissance (Task 0.2)

**Date:** 2026-06-30
**Unit:** bench `bus01` — GL.iNet GL-XE3000 (Puli AX)
**Access path:** LAB → `kit-fleet-hub` (DO droplet 167.172.237.162, WG 10.88.0.1) → jump SSH
`ssh -i /root/.ssh/busfleet root@10.88.0.2` (router WG IP 10.88.0.2).
**Access confirmed:** yes — `ubus call system board` returns `"model": "GL.iNet GL-XE3000"`.

---

## 1. Resolved facts table (feeds later plan tasks)

| Fact | Value |
|------|-------|
| Cellular ifname (WAN1 member) | **`rmnet_mhi0`** (QMI proto `qcm`, control node `/dev/mhi_QMI0`, UCI iface `modem_0001`, APN `fast.t-mobile.com`) |
| Starlink WAN ifname (WAN2) | **`eth0`** (UCI iface `wan`, proto `dhcp`). NOTE: currently `DOWN`/`NO-CARRIER` — no dish attached to bench unit. `eth1` = LAN (`br-lan`). |
| mwan3 present? | **NO** — `/etc/config/mwan3` absent, `mwan3` package not installed → Task 3.0 / Phase 2 must `opkg install mwan3`. GL ships its own multiwan (`gl-sdk4-ui-multiwan`, `kmod-gl-sdk4-kmwan`, ubus `hotplug.kmwan`) which we are NOT using. |
| Signal read command | **Preferred:** `ubus call modem.signal get_signals '{"time":1}'` → JSON `{rsrp,rsrq,sinr,slot,network_type,strength,mode}`. **AT fallback:** `gl_modem -B 1-1.2 AT AT+QCSQ` → `+QCSQ: "NR5G",<rssi>,<sinr>,<rsrp>`. **`uqmi` is NOT usable** (see Implications). |
| SIM select command | Read: `gl_modem -B 1-1.2 AT AT+QUIMSLOT?` → `+QUIMSLOT: 1`. Switch: `gl_modem -B 1-1.2 AT AT+QUIMSLOT=2` (confirmed slot-query works; switch command not exercised to avoid dropping the live link). |
| GPS read command | `gl_modem -B 1-1.2 AT AT+QGPSLOC=2` — but GNSS must be enabled first: `AT+QGPS=1`. Currently `AT+QGPS?` = `0` (disabled), so `QGPSLOC` returns `+CME ERROR: 505` (no fix). |
| grpcurl available? | **NO** — `command -v grpcurl` = MISSING → Phase 3 (Starlink) must bundle a static arm64 `grpcurl` binary in the opkg package. |
| Firewall zone of WG client iface | **`wgclient1`** — its own dedicated zone named `wgclient1` (network `wgclient1`, input DROP). NOT `wan`. The access-fix rule is `firewall.@rule[22]` name `Allow-WG-mgmt-SSH`, `src='wgclient1'`, `src_ip='10.88.0.0/24'`, dest_port 22, ACCEPT. |
| Modem USB bus (for `gl_modem -B`) | **`1-1.2`** (Quectel RM520N-GL, 4 USB interfaces at `1-1.2:1.0-1.3`) |

---

## 2. Raw command outputs

### `ubus call system board`
```json
{
	"kernel": "5.4.211",
	"hostname": "GL-XE3000",
	"system": "ARMv8 Processor rev 4",
	"model": "GL.iNet GL-XE3000",
	"board_name": "glinet,xe3000-emmc",
	"release": {
		"distribution": "OpenWrt",
		"version": "21.02-SNAPSHOT",
		"target": "mediatek/mt7981"
	}
}
```

### `/etc/config/network` (relevant excerpts)
```
config device            # LAN bridge
	option name 'br-lan'
	list ports 'eth1'
config interface 'lan'
	option device 'br-lan'
	option ipaddr '192.168.8.1'
config device
	option name 'eth0'
config interface 'wan'   # Starlink WAN port
	option device 'eth0'
	option proto 'dhcp'
	option metric '10'
config interface 'modem_0001'   # Cellular / 5G
	option apn 'fast.t-mobile.com'
	option ip_type 'IPV4V6'
	option proto 'qcm'
	option device '/dev/mhi_QMI0'
	option metric '40'
	option iccid '8901260417776090740F'
config interface 'wgclient1'    # WireGuard out-of-band mgmt
	option proto 'wgclient'
	option config 'peer_2001'
	option ip4table '1001'
```
(Also present but unused/disabled: `wwan`, `secondwan`, `guest`, various GL vpn policy rules.)

### `/etc/config/mwan3`
```
NO mwan3
```

### `opkg list-installed` (filtered: mwan3|grpc|speedtest|modem|gl-|uqmi|quectel)
Notable entries (full list captured in session):
```
gl-sdk4-modem            git-2025.290.37125-7c3efcc-1
gl-sdk4-ui-multiwan      git-2025.259.31350-8a2a5c6-1
gl-sdk4-ui-modemsignallog git-2024.303.14204-4241be7-1
kmod-gl-pcie-modem       5.4.211+20221009-1
kmod-gl-sdk4-kmwan       5.4.211+git-2025.304.07170-bb470c0-1
uqmi                     2020-11-22-0a19b5b7-3
```
No `mwan3`, no `grpcurl`, no `speedtest` (Ookla) package installed.

### `ip -br addr`
```
lo               UNKNOWN   127.0.0.1/8 ::1/128
eth0             DOWN                              # Starlink WAN — no carrier
eth1             UP                                # LAN member
ra0              UP                                # 2.4G wifi
rmnet_mhi0       UP        100.112.80.45/30        # Cellular (CGNAT)
br-lan           UP        192.168.8.1/24
wgclient1        UNKNOWN   10.88.0.2/32            # WG mgmt tunnel
```

### `ip route`
```
default via 100.112.80.46 dev rmnet_mhi0 proto static src 100.112.80.45 metric 40
100.112.80.44/30 dev rmnet_mhi0 proto static scope link metric 40
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1
```

### Modem device nodes
```
/dev/mhi_BHI  /dev/mhi_DIAG  /dev/mhi_DUN  /dev/mhi_LOOPBACK  /dev/mhi_QMI0
/dev/ttyUSB0  /dev/ttyUSB1  /dev/ttyUSB2  /dev/ttyUSB3
```
No `/dev/cdc-wdm*` — modem is on the **MHI/PCIe** transport, not USB-CDC.

### Tool availability
```
gl_modem: /usr/bin/gl_modem
uqmi:     /sbin/uqmi
grpcurl:  MISSING
mmcli:    MISSING
gpsd:     MISSING
```

### `gl_modem` usage
```
gl_modem -B <bus> AT <AT_CMD>       # send AT command; bus for this unit = 1-1.2
```

### Signal — ubus (preferred)
```json
// ubus call modem.signal get_signals '{"time":1}'
{
	"signals": [{
		"sinr": 13, "slot": 1, "network_type": "NR5G-SA",
		"timestamp": 1782917019, "strength": 4,
		"rsrp": -96, "rsrq": -10, "mode": 5
	}]
}
```
`ubus -v list modem.signal` → methods `get_signals {time:Integer}`, `upload_cloud_signals {action:String}`.

### Signal — AT fallback
```
gl_modem -B 1-1.2 AT AT+QCSQ
+QCSQ: "NR5G",-97,14,-10        # <sysmode>,<rssi>,<sinr>,<rsrp>
```

### SIM slot
```
gl_modem -B 1-1.2 AT AT+QUIMSLOT?
+QUIMSLOT: 1                    # active SIM = slot 1
```

### GPS / GNSS
```
gl_modem -B 1-1.2 AT AT+QGPS?
+QGPS: 0                        # GNSS engine currently OFF
gl_modem -B 1-1.2 AT AT+QGPSLOC=2
+CME ERROR: 505                 # no fix (engine not enabled)
```

### Starlink WAN port link
```
eth0  DOWN  <NO-CARRIER,BROADCAST,MULTICAST,UP>
```

### Firewall (WG mgmt zone + access-fix rule)
```
firewall.wgclient1 = zone
firewall.wgclient1.name='wgclient1'
firewall.wgclient1.network='wgclient1'
firewall.wgclient1.input='DROP'
firewall.@rule[22].name='Allow-WG-mgmt-SSH'
firewall.@rule[22].src='wgclient1'
firewall.@rule[22].src_ip='10.88.0.0/24'
```
Zone[1] `wan` groups: `wan wan6 wwan secondwan modem_0001 modem_0001_4` (input DROP).

---

## 3. Implications for the plan

1. **mwan3 is NOT installed (plan flagged this as possible).** `/etc/config/mwan3` is absent and no `mwan3`
   opkg is present. Phase 2 / Task 3.0 must `opkg install mwan3` (verify it exists in GL's feed for
   `mediatek/mt7981` @ 21.02-SNAPSHOT; if not, add the upstream OpenWrt 21.02 feed or bundle it).
   GL ships its own `kmwan`/multiwan stack (`kmod-gl-sdk4-kmwan`, `gl-sdk4-ui-multiwan`, ubus
   `hotplug.kmwan`) — the spec locks on `mwan3`, so we intentionally do NOT use GL's kmwan. Watch for
   conflict/overlap between the two multiwan systems on the same box (routing marks, hotplug).

2. **Firewall zone assumption in Task 0.1 was WRONG.** Task 0.1 Step 1 guessed `src='wan'` (with a
   "verify / adjust if wgclient" note). The WG client interface is actually in its **own zone `wgclient1`**,
   and that is what the working fix rule uses (`@rule[22] src='wgclient1' src_ip='10.88.0.0/24'`). Any
   future mgmt/telemetry firewall rules must target zone **`wgclient1`**, not `wan`.

3. **`uqmi` is effectively unusable for signal reads.** `uqmi -d /dev/mhi_QMI0 --get-signal-info` **hangs
   (timeout)** — GL's modem manager holds the QMI control channel (single-client QMUX). Use the GL ubus
   path `modem.signal get_signals` (clean JSON, no device contention) as the primary signal source, with
   `gl_modem -B 1-1.2 AT AT+QCSQ` as the AT fallback. **Phase 4 Task 4.1 should target ubus first**, not
   uqmi as the plan's example suggested.

4. **`gl_modem` requires a `-B <bus>` argument** (the plan's examples `gl_modem AT AT+...` omit it and would
   fail). The bus on this unit is **`1-1.2`**. This must be discovered per-unit (parse from
   `/sys/bus/usb/devices` or GL config) rather than hard-coded, since it could differ across units —
   record for the provisioning profile (Phase 8).

5. **GPS/GNSS is present but OFF by default.** `AT+QGPS?` = 0, so `AT+QGPSLOC=2` errors `+CME ERROR: 505`.
   Phase 5 Task 5.1 (`gps_fix`) must **enable GNSS first** (`AT+QGPS=1`, one-time / on boot) and handle the
   "no fix yet" (error 505/516) state gracefully before location is available. GPS command form is
   confirmed as Quectel `AT+QGPSLOC=2` (decimal degrees).

6. **`grpcurl` absent (plan flagged this).** Phase 3 Task 3.3 must bundle a static **arm64 (ARMv8)**
   `grpcurl` binary in the opkg package for Starlink gRPC.

7. **No Starlink dish on the bench unit** (`eth0` NO-CARRIER). WAN2 (Starlink) integration and any
   two-WAN balancing test cannot be validated on this bench unit as-is — need a dish on `eth0`, or test
   WAN2 with a substitute uplink. The cellular WAN (`rmnet_mhi0`) is the only live uplink right now, and
   it carries the WG mgmt tunnel (CGNAT `100.112.80.45/30`, T-Mobile).

8. **Ookla speedtest CLI not installed** — Phase 3 Task 3.2 must bundle/install it (or rely on the curl
   fallback), consistent with the plan's fallback design.

---

## 4. Task 2.0 — mwan3 / kmwan changeover (2026-07-01)

Performed on bench `bus01` via the WG mgmt path. This section feeds the Phase 8 provisioning profile.

### GL multiwan (kmwan) — exact init name + stand-down
- **GL multiwan init script name: `kmwan`** — `/etc/init.d/kmwan` (the ONLY `mwan|kmwan` init on the box;
  confirms recon §1 `kmod-gl-sdk4-kmwan`). It is a `/etc/rc.common` script, `START=21`, sourcing
  `/lib/functions/kmwan.sh`. It was **enabled** but had **no running daemon** (routing is managed via
  hotplug scripts / a single static default route, not a persistent process).
- **Exact commands used to stand it down** (Phase 8 must run these):
  ```
  /etc/init.d/kmwan disable      # rc=0  → NOW_DISABLED (won't start on boot)
  /etc/init.d/kmwan stop         # rc=0
  ```
  Verify with `/etc/init.d/kmwan enabled` (returns non-zero = disabled) and
  `ps w | grep -i mwan | grep -v grep` (no kmwan process).

### mwan3 install
- Installed from **GL's own feed** (no third-party/upstream feed needed):
  `opkg update && opkg install mwan3` → **`mwan3 2.8.15-2`** (`mwan3_2.8.15-2_all.ipk`,
  feed `glinet_gli_pub` @ v21.02.3). Binary at `/usr/sbin/mwan3`.
- mwan3 ships a **stock default `/etc/config/mwan3`** (dated 2022) with interfaces
  `wan/wwan/tethering` (+v6). **NONE of these match our live cellular uplink** (`rmnet_mhi0`, UCI iface
  `modem_0001`), so mwan3 is currently **inert**: `mwan3 status` shows every interface "offline and
  tracking is down", **no active policies, no active user rules**. mwan3's `mwan3rtmon` route-monitor
  daemon runs but does not touch the default route (member config is a later task).

### Safety / recovery notes for Phase 8
- **`setsid` and `nohup` are BOTH ABSENT** on this GL firmware (BusyBox build). For detached/background
  processes use **`start-stop-daemon -S -b -x <prog> -- <args>`** (present at `/sbin/start-stop-daemon`).
  Note: `start-stop-daemon -b -x /bin/sh` will refuse to launch if it matches an already-running
  `/bin/sh` ("… is already running") — give the watchdog a unique interpreter/exec name or wrap it so the
  `-x` match is unique.
- **Post-change verification (mgmt access + default route SURVIVED):** after `kmwan disable/stop`, a fresh
  probe returned `"model": "GL.iNet GL-XE3000"`, default route unchanged
  (`default via 100.116.118.126 dev rmnet_mhi0 … metric 40` — CGNAT src rotated from the recon-day
  `100.112.80.45` to `100.116.118.125`, normal T-Mobile lease change), and `wgclient1 10.88.0.2/32` up.
  The WG mgmt routing rules (`from/to 10.88.0.2 lookup 1001`, table 1001) are intact. **Disabling kmwan
  did not disturb routing** — expected, since kmwan had no active daemon and the live path is a single
  static default on `rmnet_mhi0`.

---

## 5. Task 2.1 pre-check — second uplink (ATT Fiber on eth0) added (2026-07-01)

**Status: BLOCKED (device shell unreachable) — but the load-bearing WG-underlay question is answered
from hub-side WireGuard evidence.**

### What happened
The operator plugged ATT Fiber into `eth0` (WAN2 stand-in for Starlink). Immediately afterward the
router's SSH shell (via the WG mgmt tunnel `10.88.0.2:22`) became **unreachable** from the fleet hub,
even though the WireGuard tunnel itself is **healthy**. All on-device commands (eth0 lease, `ip route`
metrics, `ip route get`, `ip rule`, `mwan3 status`) therefore **could not be run**. No device state was
mutated. The facts below are what the hub can observe about the tunnel from `10.88.0.1` (wg0).

### THE KEY ANSWER — WG mgmt path now rides eth0 (ATT Fiber)
The WireGuard peer **endpoint changed** from the T-Mobile CGNAT `100.x.x.x` seen in §2/§4 to a **public,
non-CGNAT address**, which is the ATT Fiber public IP:

```
# hub: wg show wg0 endpoints
0Xn7aGlyxHQ/X9uo8lsOeI3EFA7KO+zlO+qIhrzsAF8=   99.163.124.171:44881
# latest-handshakes: 6 s ago (fresh);  transfer: 335 KiB rx / 244 KiB sent (growing)
```

`99.163.124.171` is a routable public IP (not `100.64.0.0/10` CGNAT), i.e. the router is now sourcing its
WG handshakes out **eth0 / ATT Fiber**, not `rmnet_mhi0` / cellular. **The WG mgmt underlay rides eth0
(ATT Fiber).** Corollary: **downing `eth0` would sever the WG mgmt path / SSH access.**

### Safe failover-test direction
- **DO NOT bring down `eth0` (ATT Fiber)** for the failover drill — it carries the WG underlay; downing
  it cuts mgmt/SSH.
- **`rmnet_mhi0` (cellular) is the SAFE interface to bring down** for the mwan3 failover test — it does
  NOT currently carry the WG underlay handshake. (This is the *inverse* of the pre-fiber state in §2/§4,
  where cellular carried mgmt and eth0 was NO-CARRIER.)

### Current mgmt-reachability fault (flag for operator — needs on-device follow-up)
The WG control plane is up but **no in-tunnel application traffic returns from the router**:

```
# hub, over the tunnel:
ip route get 10.88.0.2      -> 10.88.0.2 dev wg0 src 10.88.0.1   (route correct)
ping -c3 10.88.0.2          -> 100% loss (even 56-byte payloads)
nc -vz 10.88.0.2 22         -> timed out ("Operation now in progress")
# 4 s window: wg 'sent' climbed 244308 -> (SYN retransmits reach router), 'rx' FLAT
```

Interpretation: the router **receives** hub traffic (encrypted `sent` counter grows) but **replies with
nothing** — its in-tunnel return path for ICMP/TCP is broken. Most likely the policy-routing that steers
tunnel replies (table `1001`, `from/to 10.88.0.2 lookup 1001`, per §4) or the WG interface's reverse-path
was disturbed when the fiber default route (metric 10) came up and displaced cellular (metric 40) as the
preferred default. This must be diagnosed/repaired **on-device via a console/LAN path** (the WG path is
the very thing that's down) before Task 2.1 routing changes proceed. Not attempted here (read-only,
no-mutate scope).

### Facts NOT captured (require device shell — blocked)
- eth0/wan DHCP lease (ATT subnet / gateway / netmask), eth0 carrier flag from the device side.
- Full `ip route` default table with metrics (expected eth0 metric 10 preferred over cellular metric 40 —
  strongly implied by the endpoint move to fiber, but not directly read).
- `rmnet_mhi0` current CGNAT IP, `ip route get 167.172.237.162` / `ip route get 99.163.124.171`.
- `mwan3 status` / `mwan3 interfaces` view of the two ifaces, `ip rule` table listing.

Re-run §5 in full once console/LAN access to `bus01` is restored and the WG mgmt path is repaired.

---

## 6. WG mgmt tunnel root cause + fix (2026-07-01)

**Symptom:** after ATT Fiber plugged into eth0 (metric 10 preferred default), WG handshakes were
fresh but all in-tunnel data (ICMP/SSH from 10.88.0.1 to 10.88.0.2) returned 100% loss.

**Root cause:** `mwan3` adds all directly-connected networks to the `mwan3_connected` ipset.
10.88.0.0/24 is a directly-connected route (wgclient1), so mwan3's iptables `mwan3_connected`
chain marks packets destined for that prefix with `0x3f00`. The ip rule
`9000: not from all fwmark 0/0xf000 lookup main` then routes them via the **main table**,
which has **no route for 10.88.0.0/24** (wgclient1 routes are only in table 1001 due to
`ip4table='1001'` in UCI). The packet is silently dropped — never reaches WG.

**Fix (applied 2026-07-01):**
Add an ip rule at priority 99 (before any mwan3 fwmark rule) routing packets to 10.88.0.0/24
via table 1001:

```
ip rule add to 10.88.0.0/24 priority 99 lookup 1001
```

Persisted as a UCI network rule so netifd recreates it on every boot:

```
uci add network rule
uci set network.@rule[-1].lookup='1001'
uci set network.@rule[-1].dest='10.88.0.0/24'
uci set network.@rule[-1].priority='99'
uci commit network
```

Also applied for robustness: `wgclient1 mtu=1280`, rp_filter=2 persisted via /etc/sysctl.conf
and /etc/hotplug.d/iface/99-busrouter-rpfilter (re-asserts on WAN flap).

**Failover note:** `scripts/diag/wg-mgmt-diagnose.sh` and `scripts/diag/wg-mgmt-fix.sh` in
this repo implement the fix. The underlying keepalive (`persistent_keepalive=25`) was already
configured from a prior session; the new ip rule was the missing piece.

**Phase 8 provisioning profile must include** this UCI rule — without it, any device with
mwan3 configured will have the WG mgmt tunnel silently drop in-tunnel data.

## 7. Task 2.1 bench-test findings (2026-07-01)

mwan3 configured with `modem_0001` (cellular) + `wan` (fiber/eth0), policy `balanced`.
Both interfaces tracked online (1.1.1.1 + 8.8.8.8, reliability 1, down=3, up=3 x 5s interval).

**Failover drill result:**
- `ip link set rmnet_mhi0 down` -> mwan3 declares `modem_0001 error` within ~15s (checked)
- Policy shifts to `wan (100%)` automatically (checked)
- WG mgmt tunnel (riding eth0) stays alive throughout failover (checked)
- **Recovery gotcha:** `ip link set rmnet_mhi0 up` does NOT restore the QMI data session.
  Must use `ifdown modem_0001 && ifup modem_0001` to reconnect. After ~20s both interfaces
  return to `balanced: wan(50%) modem_0001(50%)` (checked)

**Phase 9 pilot:** fiber failover (bring fiber down, WG must ride cellular) should be tested
once the provisioning robustness fix (section 6 ip rule) is confirmed to survive both uplink combos.
