# GL-XE3000 Device Reconnaissance (Task 0.2) **Date:** 2026-06-30 **Unit:** bench `bus01` — GL.iNet GL-XE3000 (Puli AX) **Access path:** LAB → `kit-fleet-hub` (DO droplet 167.172.237.162, WG 10.88.0.1) → jump SSH `ssh -i /root/.ssh/busfleet root@10.88.0.2` (router WG IP 10.88.0.2). **Access confirmed:** yes — `ubus call system board` returns `"model": "GL.iNet GL-XE3000"`. --- ## 1. Resolved facts table (feeds later plan tasks) | Fact | Value | |------|-------| | Cellular ifname (WAN1 member) | **`rmnet_mhi0`** (QMI proto `qcm`, control node `/dev/mhi_QMI0`, UCI iface `modem_0001`, APN `fast.t-mobile.com`) | | Starlink WAN ifname (WAN2) | **`eth0`** (UCI iface `wan`, proto `dhcp`). NOTE: currently `DOWN`/`NO-CARRIER` — no dish attached to bench unit. `eth1` = LAN (`br-lan`). | | mwan3 present? | **NO** — `/etc/config/mwan3` absent, `mwan3` package not installed → Task 3.0 / Phase 2 must `opkg install mwan3`. GL ships its own multiwan (`gl-sdk4-ui-multiwan`, `kmod-gl-sdk4-kmwan`, ubus `hotplug.kmwan`) which we are NOT using. | | Signal read command | **Preferred:** `ubus call modem.signal get_signals '{"time":1}'` → JSON `{rsrp,rsrq,sinr,slot,network_type,strength,mode}`. **AT fallback:** `gl_modem -B 1-1.2 AT AT+QCSQ` → `+QCSQ: "NR5G",,,`. **`uqmi` is NOT usable** (see Implications). | | SIM select command | Read: `gl_modem -B 1-1.2 AT AT+QUIMSLOT?` → `+QUIMSLOT: 1`. Switch: `gl_modem -B 1-1.2 AT AT+QUIMSLOT=2` (confirmed slot-query works; switch command not exercised to avoid dropping the live link). | | GPS read command | `gl_modem -B 1-1.2 AT AT+QGPSLOC=2` — but GNSS must be enabled first: `AT+QGPS=1`. Currently `AT+QGPS?` = `0` (disabled), so `QGPSLOC` returns `+CME ERROR: 505` (no fix). | | grpcurl available? | **NO** — `command -v grpcurl` = MISSING → Phase 3 (Starlink) must bundle a static arm64 `grpcurl` binary in the opkg package. | | Firewall zone of WG client iface | **`wgclient1`** — its own dedicated zone named `wgclient1` (network `wgclient1`, input DROP). NOT `wan`. The access-fix rule is `firewall.@rule[22]` name `Allow-WG-mgmt-SSH`, `src='wgclient1'`, `src_ip='10.88.0.0/24'`, dest_port 22, ACCEPT. | | Modem USB bus (for `gl_modem -B`) | **`1-1.2`** (Quectel RM520N-GL, 4 USB interfaces at `1-1.2:1.0-1.3`) | --- ## 2. Raw command outputs ### `ubus call system board` ```json { "kernel": "5.4.211", "hostname": "GL-XE3000", "system": "ARMv8 Processor rev 4", "model": "GL.iNet GL-XE3000", "board_name": "glinet,xe3000-emmc", "release": { "distribution": "OpenWrt", "version": "21.02-SNAPSHOT", "target": "mediatek/mt7981" } } ``` ### `/etc/config/network` (relevant excerpts) ``` config device # LAN bridge option name 'br-lan' list ports 'eth1' config interface 'lan' option device 'br-lan' option ipaddr '192.168.8.1' config device option name 'eth0' config interface 'wan' # Starlink WAN port option device 'eth0' option proto 'dhcp' option metric '10' config interface 'modem_0001' # Cellular / 5G option apn 'fast.t-mobile.com' option ip_type 'IPV4V6' option proto 'qcm' option device '/dev/mhi_QMI0' option metric '40' option iccid '8901260417776090740F' config interface 'wgclient1' # WireGuard out-of-band mgmt option proto 'wgclient' option config 'peer_2001' option ip4table '1001' ``` (Also present but unused/disabled: `wwan`, `secondwan`, `guest`, various GL vpn policy rules.) ### `/etc/config/mwan3` ``` NO mwan3 ``` ### `opkg list-installed` (filtered: mwan3|grpc|speedtest|modem|gl-|uqmi|quectel) Notable entries (full list captured in session): ``` gl-sdk4-modem git-2025.290.37125-7c3efcc-1 gl-sdk4-ui-multiwan git-2025.259.31350-8a2a5c6-1 gl-sdk4-ui-modemsignallog git-2024.303.14204-4241be7-1 kmod-gl-pcie-modem 5.4.211+20221009-1 kmod-gl-sdk4-kmwan 5.4.211+git-2025.304.07170-bb470c0-1 uqmi 2020-11-22-0a19b5b7-3 ``` No `mwan3`, no `grpcurl`, no `speedtest` (Ookla) package installed. ### `ip -br addr` ``` lo UNKNOWN 127.0.0.1/8 ::1/128 eth0 DOWN # Starlink WAN — no carrier eth1 UP # LAN member ra0 UP # 2.4G wifi rmnet_mhi0 UP 100.112.80.45/30 # Cellular (CGNAT) br-lan UP 192.168.8.1/24 wgclient1 UNKNOWN 10.88.0.2/32 # WG mgmt tunnel ``` ### `ip route` ``` default via 100.112.80.46 dev rmnet_mhi0 proto static src 100.112.80.45 metric 40 100.112.80.44/30 dev rmnet_mhi0 proto static scope link metric 40 192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1 ``` ### Modem device nodes ``` /dev/mhi_BHI /dev/mhi_DIAG /dev/mhi_DUN /dev/mhi_LOOPBACK /dev/mhi_QMI0 /dev/ttyUSB0 /dev/ttyUSB1 /dev/ttyUSB2 /dev/ttyUSB3 ``` No `/dev/cdc-wdm*` — modem is on the **MHI/PCIe** transport, not USB-CDC. ### Tool availability ``` gl_modem: /usr/bin/gl_modem uqmi: /sbin/uqmi grpcurl: MISSING mmcli: MISSING gpsd: MISSING ``` ### `gl_modem` usage ``` gl_modem -B AT # send AT command; bus for this unit = 1-1.2 ``` ### Signal — ubus (preferred) ```json // ubus call modem.signal get_signals '{"time":1}' { "signals": [{ "sinr": 13, "slot": 1, "network_type": "NR5G-SA", "timestamp": 1782917019, "strength": 4, "rsrp": -96, "rsrq": -10, "mode": 5 }] } ``` `ubus -v list modem.signal` → methods `get_signals {time:Integer}`, `upload_cloud_signals {action:String}`. ### Signal — AT fallback ``` gl_modem -B 1-1.2 AT AT+QCSQ +QCSQ: "NR5G",-97,14,-10 # ,,, ``` ### SIM slot ``` gl_modem -B 1-1.2 AT AT+QUIMSLOT? +QUIMSLOT: 1 # active SIM = slot 1 ``` ### GPS / GNSS ``` gl_modem -B 1-1.2 AT AT+QGPS? +QGPS: 0 # GNSS engine currently OFF gl_modem -B 1-1.2 AT AT+QGPSLOC=2 +CME ERROR: 505 # no fix (engine not enabled) ``` ### Starlink WAN port link ``` eth0 DOWN ``` ### Firewall (WG mgmt zone + access-fix rule) ``` firewall.wgclient1 = zone firewall.wgclient1.name='wgclient1' firewall.wgclient1.network='wgclient1' firewall.wgclient1.input='DROP' firewall.@rule[22].name='Allow-WG-mgmt-SSH' firewall.@rule[22].src='wgclient1' firewall.@rule[22].src_ip='10.88.0.0/24' ``` Zone[1] `wan` groups: `wan wan6 wwan secondwan modem_0001 modem_0001_4` (input DROP). --- ## 3. Implications for the plan 1. **mwan3 is NOT installed (plan flagged this as possible).** `/etc/config/mwan3` is absent and no `mwan3` opkg is present. Phase 2 / Task 3.0 must `opkg install mwan3` (verify it exists in GL's feed for `mediatek/mt7981` @ 21.02-SNAPSHOT; if not, add the upstream OpenWrt 21.02 feed or bundle it). GL ships its own `kmwan`/multiwan stack (`kmod-gl-sdk4-kmwan`, `gl-sdk4-ui-multiwan`, ubus `hotplug.kmwan`) — the spec locks on `mwan3`, so we intentionally do NOT use GL's kmwan. Watch for conflict/overlap between the two multiwan systems on the same box (routing marks, hotplug). 2. **Firewall zone assumption in Task 0.1 was WRONG.** Task 0.1 Step 1 guessed `src='wan'` (with a "verify / adjust if wgclient" note). The WG client interface is actually in its **own zone `wgclient1`**, and that is what the working fix rule uses (`@rule[22] src='wgclient1' src_ip='10.88.0.0/24'`). Any future mgmt/telemetry firewall rules must target zone **`wgclient1`**, not `wan`. 3. **`uqmi` is effectively unusable for signal reads.** `uqmi -d /dev/mhi_QMI0 --get-signal-info` **hangs (timeout)** — GL's modem manager holds the QMI control channel (single-client QMUX). Use the GL ubus path `modem.signal get_signals` (clean JSON, no device contention) as the primary signal source, with `gl_modem -B 1-1.2 AT AT+QCSQ` as the AT fallback. **Phase 4 Task 4.1 should target ubus first**, not uqmi as the plan's example suggested. 4. **`gl_modem` requires a `-B ` argument** (the plan's examples `gl_modem AT AT+...` omit it and would fail). The bus on this unit is **`1-1.2`**. This must be discovered per-unit (parse from `/sys/bus/usb/devices` or GL config) rather than hard-coded, since it could differ across units — record for the provisioning profile (Phase 8). 5. **GPS/GNSS is present but OFF by default.** `AT+QGPS?` = 0, so `AT+QGPSLOC=2` errors `+CME ERROR: 505`. Phase 5 Task 5.1 (`gps_fix`) must **enable GNSS first** (`AT+QGPS=1`, one-time / on boot) and handle the "no fix yet" (error 505/516) state gracefully before location is available. GPS command form is confirmed as Quectel `AT+QGPSLOC=2` (decimal degrees). 6. **`grpcurl` absent (plan flagged this).** Phase 3 Task 3.3 must bundle a static **arm64 (ARMv8)** `grpcurl` binary in the opkg package for Starlink gRPC. 7. **No Starlink dish on the bench unit** (`eth0` NO-CARRIER). WAN2 (Starlink) integration and any two-WAN balancing test cannot be validated on this bench unit as-is — need a dish on `eth0`, or test WAN2 with a substitute uplink. The cellular WAN (`rmnet_mhi0`) is the only live uplink right now, and it carries the WG mgmt tunnel (CGNAT `100.112.80.45/30`, T-Mobile). 8. **Ookla speedtest CLI not installed** — Phase 3 Task 3.2 must bundle/install it (or rely on the curl fallback), consistent with the plan's fallback design. --- ## 4. Task 2.0 — mwan3 / kmwan changeover (2026-07-01) Performed on bench `bus01` via the WG mgmt path. This section feeds the Phase 8 provisioning profile. ### GL multiwan (kmwan) — exact init name + stand-down - **GL multiwan init script name: `kmwan`** — `/etc/init.d/kmwan` (the ONLY `mwan|kmwan` init on the box; confirms recon §1 `kmod-gl-sdk4-kmwan`). It is a `/etc/rc.common` script, `START=21`, sourcing `/lib/functions/kmwan.sh`. It was **enabled** but had **no running daemon** (routing is managed via hotplug scripts / a single static default route, not a persistent process). - **Exact commands used to stand it down** (Phase 8 must run these): ``` /etc/init.d/kmwan disable # rc=0 → NOW_DISABLED (won't start on boot) /etc/init.d/kmwan stop # rc=0 ``` Verify with `/etc/init.d/kmwan enabled` (returns non-zero = disabled) and `ps w | grep -i mwan | grep -v grep` (no kmwan process). ### mwan3 install - Installed from **GL's own feed** (no third-party/upstream feed needed): `opkg update && opkg install mwan3` → **`mwan3 2.8.15-2`** (`mwan3_2.8.15-2_all.ipk`, feed `glinet_gli_pub` @ v21.02.3). Binary at `/usr/sbin/mwan3`. - mwan3 ships a **stock default `/etc/config/mwan3`** (dated 2022) with interfaces `wan/wwan/tethering` (+v6). **NONE of these match our live cellular uplink** (`rmnet_mhi0`, UCI iface `modem_0001`), so mwan3 is currently **inert**: `mwan3 status` shows every interface "offline and tracking is down", **no active policies, no active user rules**. mwan3's `mwan3rtmon` route-monitor daemon runs but does not touch the default route (member config is a later task). ### Safety / recovery notes for Phase 8 - **`setsid` and `nohup` are BOTH ABSENT** on this GL firmware (BusyBox build). For detached/background processes use **`start-stop-daemon -S -b -x -- `** (present at `/sbin/start-stop-daemon`). Note: `start-stop-daemon -b -x /bin/sh` will refuse to launch if it matches an already-running `/bin/sh` ("… is already running") — give the watchdog a unique interpreter/exec name or wrap it so the `-x` match is unique. - **Post-change verification (mgmt access + default route SURVIVED):** after `kmwan disable/stop`, a fresh probe returned `"model": "GL.iNet GL-XE3000"`, default route unchanged (`default via 100.116.118.126 dev rmnet_mhi0 … metric 40` — CGNAT src rotated from the recon-day `100.112.80.45` to `100.116.118.125`, normal T-Mobile lease change), and `wgclient1 10.88.0.2/32` up. The WG mgmt routing rules (`from/to 10.88.0.2 lookup 1001`, table 1001) are intact. **Disabling kmwan did not disturb routing** — expected, since kmwan had no active daemon and the live path is a single static default on `rmnet_mhi0`. --- ## 5. Task 2.1 pre-check — second uplink (ATT Fiber on eth0) added (2026-07-01) **Status: BLOCKED (device shell unreachable) — but the load-bearing WG-underlay question is answered from hub-side WireGuard evidence.** ### What happened The operator plugged ATT Fiber into `eth0` (WAN2 stand-in for Starlink). Immediately afterward the router's SSH shell (via the WG mgmt tunnel `10.88.0.2:22`) became **unreachable** from the fleet hub, even though the WireGuard tunnel itself is **healthy**. All on-device commands (eth0 lease, `ip route` metrics, `ip route get`, `ip rule`, `mwan3 status`) therefore **could not be run**. No device state was mutated. The facts below are what the hub can observe about the tunnel from `10.88.0.1` (wg0). ### THE KEY ANSWER — WG mgmt path now rides eth0 (ATT Fiber) The WireGuard peer **endpoint changed** from the T-Mobile CGNAT `100.x.x.x` seen in §2/§4 to a **public, non-CGNAT address**, which is the ATT Fiber public IP: ``` # hub: wg show wg0 endpoints 0Xn7aGlyxHQ/X9uo8lsOeI3EFA7KO+zlO+qIhrzsAF8= 99.163.124.171:44881 # latest-handshakes: 6 s ago (fresh); transfer: 335 KiB rx / 244 KiB sent (growing) ``` `99.163.124.171` is a routable public IP (not `100.64.0.0/10` CGNAT), i.e. the router is now sourcing its WG handshakes out **eth0 / ATT Fiber**, not `rmnet_mhi0` / cellular. **The WG mgmt underlay rides eth0 (ATT Fiber).** Corollary: **downing `eth0` would sever the WG mgmt path / SSH access.** ### Safe failover-test direction - **DO NOT bring down `eth0` (ATT Fiber)** for the failover drill — it carries the WG underlay; downing it cuts mgmt/SSH. - **`rmnet_mhi0` (cellular) is the SAFE interface to bring down** for the mwan3 failover test — it does NOT currently carry the WG underlay handshake. (This is the *inverse* of the pre-fiber state in §2/§4, where cellular carried mgmt and eth0 was NO-CARRIER.) ### Current mgmt-reachability fault (flag for operator — needs on-device follow-up) The WG control plane is up but **no in-tunnel application traffic returns from the router**: ``` # hub, over the tunnel: ip route get 10.88.0.2 -> 10.88.0.2 dev wg0 src 10.88.0.1 (route correct) ping -c3 10.88.0.2 -> 100% loss (even 56-byte payloads) nc -vz 10.88.0.2 22 -> timed out ("Operation now in progress") # 4 s window: wg 'sent' climbed 244308 -> (SYN retransmits reach router), 'rx' FLAT ``` Interpretation: the router **receives** hub traffic (encrypted `sent` counter grows) but **replies with nothing** — its in-tunnel return path for ICMP/TCP is broken. Most likely the policy-routing that steers tunnel replies (table `1001`, `from/to 10.88.0.2 lookup 1001`, per §4) or the WG interface's reverse-path was disturbed when the fiber default route (metric 10) came up and displaced cellular (metric 40) as the preferred default. This must be diagnosed/repaired **on-device via a console/LAN path** (the WG path is the very thing that's down) before Task 2.1 routing changes proceed. Not attempted here (read-only, no-mutate scope). ### Facts NOT captured (require device shell — blocked) - eth0/wan DHCP lease (ATT subnet / gateway / netmask), eth0 carrier flag from the device side. - Full `ip route` default table with metrics (expected eth0 metric 10 preferred over cellular metric 40 — strongly implied by the endpoint move to fiber, but not directly read). - `rmnet_mhi0` current CGNAT IP, `ip route get 167.172.237.162` / `ip route get 99.163.124.171`. - `mwan3 status` / `mwan3 interfaces` view of the two ifaces, `ip rule` table listing. Re-run §5 in full once console/LAN access to `bus01` is restored and the WG mgmt path is repaired. --- ## 6. WG mgmt tunnel root cause + fix (2026-07-01) **Symptom:** after ATT Fiber plugged into eth0 (metric 10 preferred default), WG handshakes were fresh but all in-tunnel data (ICMP/SSH from 10.88.0.1 to 10.88.0.2) returned 100% loss. **Root cause:** `mwan3` adds all directly-connected networks to the `mwan3_connected` ipset. 10.88.0.0/24 is a directly-connected route (wgclient1), so mwan3's iptables `mwan3_connected` chain marks packets destined for that prefix with `0x3f00`. The ip rule `9000: not from all fwmark 0/0xf000 lookup main` then routes them via the **main table**, which has **no route for 10.88.0.0/24** (wgclient1 routes are only in table 1001 due to `ip4table='1001'` in UCI). The packet is silently dropped — never reaches WG. **Fix (applied 2026-07-01):** Add an ip rule at priority 99 (before any mwan3 fwmark rule) routing packets to 10.88.0.0/24 via table 1001: ``` ip rule add to 10.88.0.0/24 priority 99 lookup 1001 ``` Persisted as a UCI network rule so netifd recreates it on every boot: ``` uci add network rule uci set network.@rule[-1].lookup='1001' uci set network.@rule[-1].dest='10.88.0.0/24' uci set network.@rule[-1].priority='99' uci commit network ``` Also applied for robustness: `wgclient1 mtu=1280`, rp_filter=2 persisted via /etc/sysctl.conf and /etc/hotplug.d/iface/99-busrouter-rpfilter (re-asserts on WAN flap). **Failover note:** `scripts/diag/wg-mgmt-diagnose.sh` and `scripts/diag/wg-mgmt-fix.sh` in this repo implement the fix. The underlying keepalive (`persistent_keepalive=25`) was already configured from a prior session; the new ip rule was the missing piece. **Phase 8 provisioning profile must include** this UCI rule — without it, any device with mwan3 configured will have the WG mgmt tunnel silently drop in-tunnel data. ## 7. Task 2.1 bench-test findings (2026-07-01) mwan3 configured with `modem_0001` (cellular) + `wan` (fiber/eth0), policy `balanced`. Both interfaces tracked online (1.1.1.1 + 8.8.8.8, reliability 1, down=3, up=3 x 5s interval). **Failover drill result:** - `ip link set rmnet_mhi0 down` -> mwan3 declares `modem_0001 error` within ~15s (checked) - Policy shifts to `wan (100%)` automatically (checked) - WG mgmt tunnel (riding eth0) stays alive throughout failover (checked) - **Recovery gotcha:** `ip link set rmnet_mhi0 up` does NOT restore the QMI data session. Must use `ifdown modem_0001 && ifup modem_0001` to reconnect. After ~20s both interfaces return to `balanced: wan(50%) modem_0001(50%)` (checked) **Phase 9 pilot:** fiber failover (bring fiber down, WG must ride cellular) should be tested once the provisioning robustness fix (section 6 ip rule) is confirmed to survive both uplink combos.