kit-busrouter/docs/recon/2026-06-30-xe3000-environment.md

GL-XE3000 Device Reconnaissance (Task 0.2)

Date: 2026-06-30 Unit: bench bus01 — GL.iNet GL-XE3000 (Puli AX) Access path: LAB → kit-fleet-hub (DO droplet 167.172.237.162, WG 10.88.0.1) → jump SSH ssh -i /root/.ssh/busfleet root@10.88.0.2 (router WG IP 10.88.0.2). Access confirmed: yes — ubus call system board returns "model": "GL.iNet GL-XE3000".

---

1. Resolved facts table (feeds later plan tasks)

Fact	Value
Cellular ifname (WAN1 member)	rmnet_mhi0 (QMI proto qcm, control node /dev/mhi_QMI0, UCI iface modem_0001, APN fast.t-mobile.com)
Starlink WAN ifname (WAN2)	eth0 (UCI iface wan, proto dhcp). NOTE: currently DOWN/NO-CARRIER — no dish attached to bench unit. eth1 = LAN (br-lan).
mwan3 present?	NO — /etc/config/mwan3 absent, mwan3 package not installed → Task 3.0 / Phase 2 must opkg install mwan3. GL ships its own multiwan (gl-sdk4-ui-multiwan, kmod-gl-sdk4-kmwan, ubus hotplug.kmwan) which we are NOT using.
Signal read command	Preferred: ubus call modem.signal get_signals '{"time":1}' → JSON {rsrp,rsrq,sinr,slot,network_type,strength,mode}. AT fallback: gl_modem -B 1-1.2 AT AT+QCSQ → +QCSQ: "NR5G",<rssi>,<sinr>,<rsrp>. uqmi is NOT usable (see Implications).
SIM select command	Read: gl_modem -B 1-1.2 AT AT+QUIMSLOT? → +QUIMSLOT: 1. Switch: gl_modem -B 1-1.2 AT AT+QUIMSLOT=2 (confirmed slot-query works; switch command not exercised to avoid dropping the live link).
GPS read command	gl_modem -B 1-1.2 AT AT+QGPSLOC=2 — but GNSS must be enabled first: AT+QGPS=1. Currently AT+QGPS? = 0 (disabled), so QGPSLOC returns +CME ERROR: 505 (no fix).
grpcurl available?	NO — command -v grpcurl = MISSING → Phase 3 (Starlink) must bundle a static arm64 grpcurl binary in the opkg package.
Firewall zone of WG client iface	wgclient1 — its own dedicated zone named wgclient1 (network wgclient1, input DROP). NOT wan. The access-fix rule is firewall.@rule[22] name Allow-WG-mgmt-SSH, src='wgclient1', src_ip='10.88.0.0/24', dest_port 22, ACCEPT.
Modem USB bus (for gl_modem -B)	1-1.2 (Quectel RM520N-GL, 4 USB interfaces at 1-1.2:1.0-1.3)

---

2. Raw command outputs

ubus call system board

{
	"kernel": "5.4.211",
	"hostname": "GL-XE3000",
	"system": "ARMv8 Processor rev 4",
	"model": "GL.iNet GL-XE3000",
	"board_name": "glinet,xe3000-emmc",
	"release": {
		"distribution": "OpenWrt",
		"version": "21.02-SNAPSHOT",
		"target": "mediatek/mt7981"
	}
}

/etc/config/network (relevant excerpts)

config device            # LAN bridge
	option name 'br-lan'
	list ports 'eth1'
config interface 'lan'
	option device 'br-lan'
	option ipaddr '192.168.8.1'
config device
	option name 'eth0'
config interface 'wan'   # Starlink WAN port
	option device 'eth0'
	option proto 'dhcp'
	option metric '10'
config interface 'modem_0001'   # Cellular / 5G
	option apn 'fast.t-mobile.com'
	option ip_type 'IPV4V6'
	option proto 'qcm'
	option device '/dev/mhi_QMI0'
	option metric '40'
	option iccid '8901260417776090740F'
config interface 'wgclient1'    # WireGuard out-of-band mgmt
	option proto 'wgclient'
	option config 'peer_2001'
	option ip4table '1001'

(Also present but unused/disabled: wwan, secondwan, guest, various GL vpn policy rules.)

/etc/config/mwan3

NO mwan3

opkg list-installed (filtered: mwan3|grpc|speedtest|modem|gl-|uqmi|quectel)

Notable entries (full list captured in session):

gl-sdk4-modem            git-2025.290.37125-7c3efcc-1
gl-sdk4-ui-multiwan      git-2025.259.31350-8a2a5c6-1
gl-sdk4-ui-modemsignallog git-2024.303.14204-4241be7-1
kmod-gl-pcie-modem       5.4.211+20221009-1
kmod-gl-sdk4-kmwan       5.4.211+git-2025.304.07170-bb470c0-1
uqmi                     2020-11-22-0a19b5b7-3

No mwan3, no grpcurl, no speedtest (Ookla) package installed.

ip -br addr

lo               UNKNOWN   127.0.0.1/8 ::1/128
eth0             DOWN                              # Starlink WAN — no carrier
eth1             UP                                # LAN member
ra0              UP                                # 2.4G wifi
rmnet_mhi0       UP        100.112.80.45/30        # Cellular (CGNAT)
br-lan           UP        192.168.8.1/24
wgclient1        UNKNOWN   10.88.0.2/32            # WG mgmt tunnel

ip route

default via 100.112.80.46 dev rmnet_mhi0 proto static src 100.112.80.45 metric 40
100.112.80.44/30 dev rmnet_mhi0 proto static scope link metric 40
192.168.8.0/24 dev br-lan proto kernel scope link src 192.168.8.1

Modem device nodes

/dev/mhi_BHI  /dev/mhi_DIAG  /dev/mhi_DUN  /dev/mhi_LOOPBACK  /dev/mhi_QMI0
/dev/ttyUSB0  /dev/ttyUSB1  /dev/ttyUSB2  /dev/ttyUSB3

No /dev/cdc-wdm* — modem is on the MHI/PCIe transport, not USB-CDC.

Tool availability

gl_modem: /usr/bin/gl_modem
uqmi:     /sbin/uqmi
grpcurl:  MISSING
mmcli:    MISSING
gpsd:     MISSING

gl_modem usage

gl_modem -B <bus> AT <AT_CMD>       # send AT command; bus for this unit = 1-1.2

Signal — ubus (preferred)

// ubus call modem.signal get_signals '{"time":1}'
{
	"signals": [{
		"sinr": 13, "slot": 1, "network_type": "NR5G-SA",
		"timestamp": 1782917019, "strength": 4,
		"rsrp": -96, "rsrq": -10, "mode": 5
	}]
}

ubus -v list modem.signal → methods get_signals {time:Integer}, upload_cloud_signals {action:String}.

Signal — AT fallback

gl_modem -B 1-1.2 AT AT+QCSQ
+QCSQ: "NR5G",-97,14,-10        # <sysmode>,<rssi>,<sinr>,<rsrp>

SIM slot

gl_modem -B 1-1.2 AT AT+QUIMSLOT?
+QUIMSLOT: 1                    # active SIM = slot 1

GPS / GNSS

gl_modem -B 1-1.2 AT AT+QGPS?
+QGPS: 0                        # GNSS engine currently OFF
gl_modem -B 1-1.2 AT AT+QGPSLOC=2
+CME ERROR: 505                 # no fix (engine not enabled)

Starlink WAN port link

eth0  DOWN  <NO-CARRIER,BROADCAST,MULTICAST,UP>

Firewall (WG mgmt zone + access-fix rule)

firewall.wgclient1 = zone
firewall.wgclient1.name='wgclient1'
firewall.wgclient1.network='wgclient1'
firewall.wgclient1.input='DROP'
firewall.@rule[22].name='Allow-WG-mgmt-SSH'
firewall.@rule[22].src='wgclient1'
firewall.@rule[22].src_ip='10.88.0.0/24'

Zone[1] wan groups: wan wan6 wwan secondwan modem_0001 modem_0001_4 (input DROP).

---

3. Implications for the plan

1. mwan3 is NOT installed (plan flagged this as possible). /etc/config/mwan3 is absent and no mwan3 opkg is present. Phase 2 / Task 3.0 must opkg install mwan3 (verify it exists in GL's feed for mediatek/mt7981 @ 21.02-SNAPSHOT; if not, add the upstream OpenWrt 21.02 feed or bundle it). GL ships its own kmwan/multiwan stack (kmod-gl-sdk4-kmwan, gl-sdk4-ui-multiwan, ubus hotplug.kmwan) — the spec locks on mwan3, so we intentionally do NOT use GL's kmwan. Watch for conflict/overlap between the two multiwan systems on the same box (routing marks, hotplug).

2. Firewall zone assumption in Task 0.1 was WRONG. Task 0.1 Step 1 guessed src='wan' (with a "verify / adjust if wgclient" note). The WG client interface is actually in its own zone wgclient1, and that is what the working fix rule uses (@rule[22] src='wgclient1' src_ip='10.88.0.0/24'). Any future mgmt/telemetry firewall rules must target zone wgclient1, not wan.

3. uqmi is effectively unusable for signal reads. uqmi -d /dev/mhi_QMI0 --get-signal-info hangs (timeout) — GL's modem manager holds the QMI control channel (single-client QMUX). Use the GL ubus path modem.signal get_signals (clean JSON, no device contention) as the primary signal source, with gl_modem -B 1-1.2 AT AT+QCSQ as the AT fallback. Phase 4 Task 4.1 should target ubus first, not uqmi as the plan's example suggested.

4. gl_modem requires a -B <bus> argument (the plan's examples gl_modem AT AT+... omit it and would fail). The bus on this unit is 1-1.2. This must be discovered per-unit (parse from /sys/bus/usb/devices or GL config) rather than hard-coded, since it could differ across units — record for the provisioning profile (Phase 8).

5. GPS/GNSS is present but OFF by default. AT+QGPS? = 0, so AT+QGPSLOC=2 errors +CME ERROR: 505. Phase 5 Task 5.1 (gps_fix) must enable GNSS first (AT+QGPS=1, one-time / on boot) and handle the "no fix yet" (error 505/516) state gracefully before location is available. GPS command form is confirmed as Quectel AT+QGPSLOC=2 (decimal degrees).

6. grpcurl absent (plan flagged this). Phase 3 Task 3.3 must bundle a static arm64 (ARMv8) grpcurl binary in the opkg package for Starlink gRPC.

7. No Starlink dish on the bench unit (eth0 NO-CARRIER). WAN2 (Starlink) integration and any two-WAN balancing test cannot be validated on this bench unit as-is — need a dish on eth0, or test WAN2 with a substitute uplink. The cellular WAN (rmnet_mhi0) is the only live uplink right now, and it carries the WG mgmt tunnel (CGNAT 100.112.80.45/30, T-Mobile).

8. Ookla speedtest CLI not installed — Phase 3 Task 3.2 must bundle/install it (or rely on the curl fallback), consistent with the plan's fallback design.

---

4. Task 2.0 — mwan3 / kmwan changeover (2026-07-01)

Performed on bench bus01 via the WG mgmt path. This section feeds the Phase 8 provisioning profile.

GL multiwan (kmwan) — exact init name + stand-down

• GL multiwan init script name: kmwan — /etc/init.d/kmwan (the ONLY mwan|kmwan init on the box; confirms recon §1 kmod-gl-sdk4-kmwan). It is a /etc/rc.common script, START=21, sourcing /lib/functions/kmwan.sh. It was enabled but had no running daemon (routing is managed via hotplug scripts / a single static default route, not a persistent process).
• Exact commands used to stand it down (Phase 8 must run these):

  /etc/init.d/kmwan disable      # rc=0  → NOW_DISABLED (won't start on boot)
  /etc/init.d/kmwan stop         # rc=0
  

  Verify with /etc/init.d/kmwan enabled (returns non-zero = disabled) and ps w | grep -i mwan | grep -v grep (no kmwan process).

mwan3 install

• Installed from GL's own feed (no third-party/upstream feed needed): opkg update && opkg install mwan3 → mwan3 2.8.15-2 (mwan3_2.8.15-2_all.ipk, feed glinet_gli_pub @ v21.02.3). Binary at /usr/sbin/mwan3.
• mwan3 ships a stock default /etc/config/mwan3 (dated 2022) with interfaces wan/wwan/tethering (+v6). NONE of these match our live cellular uplink (rmnet_mhi0, UCI iface modem_0001), so mwan3 is currently inert: mwan3 status shows every interface "offline and tracking is down", no active policies, no active user rules. mwan3's mwan3rtmon route-monitor daemon runs but does not touch the default route (member config is a later task).

Safety / recovery notes for Phase 8

• setsid and nohup are BOTH ABSENT on this GL firmware (BusyBox build). For detached/background processes use start-stop-daemon -S -b -x <prog> -- <args> (present at /sbin/start-stop-daemon). Note: start-stop-daemon -b -x /bin/sh will refuse to launch if it matches an already-running /bin/sh ("… is already running") — give the watchdog a unique interpreter/exec name or wrap it so the -x match is unique.
• Post-change verification (mgmt access + default route SURVIVED): after kmwan disable/stop, a fresh probe returned "model": "GL.iNet GL-XE3000", default route unchanged (default via 100.116.118.126 dev rmnet_mhi0 … metric 40 — CGNAT src rotated from the recon-day 100.112.80.45 to 100.116.118.125, normal T-Mobile lease change), and wgclient1 10.88.0.2/32 up. The WG mgmt routing rules (from/to 10.88.0.2 lookup 1001, table 1001) are intact. Disabling kmwan did not disturb routing — expected, since kmwan had no active daemon and the live path is a single static default on rmnet_mhi0.

---

5. Task 2.1 pre-check — second uplink (ATT Fiber on eth0) added (2026-07-01)

Status: BLOCKED (device shell unreachable) — but the load-bearing WG-underlay question is answered from hub-side WireGuard evidence.

What happened

The operator plugged ATT Fiber into eth0 (WAN2 stand-in for Starlink). Immediately afterward the router's SSH shell (via the WG mgmt tunnel 10.88.0.2:22) became unreachable from the fleet hub, even though the WireGuard tunnel itself is healthy. All on-device commands (eth0 lease, ip route metrics, ip route get, ip rule, mwan3 status) therefore could not be run. No device state was mutated. The facts below are what the hub can observe about the tunnel from 10.88.0.1 (wg0).

THE KEY ANSWER — WG mgmt path now rides eth0 (ATT Fiber)

The WireGuard peer endpoint changed from the T-Mobile CGNAT 100.x.x.x seen in §2/§4 to a public, non-CGNAT address, which is the ATT Fiber public IP:

# hub: wg show wg0 endpoints
0Xn7aGlyxHQ/X9uo8lsOeI3EFA7KO+zlO+qIhrzsAF8=   99.163.124.171:44881
# latest-handshakes: 6 s ago (fresh);  transfer: 335 KiB rx / 244 KiB sent (growing)

99.163.124.171 is a routable public IP (not 100.64.0.0/10 CGNAT), i.e. the router is now sourcing its WG handshakes out eth0 / ATT Fiber, not rmnet_mhi0 / cellular. The WG mgmt underlay rides eth0 (ATT Fiber). Corollary: downing eth0 would sever the WG mgmt path / SSH access.

Safe failover-test direction

• DO NOT bring down eth0 (ATT Fiber) for the failover drill — it carries the WG underlay; downing it cuts mgmt/SSH.
• rmnet_mhi0 (cellular) is the SAFE interface to bring down for the mwan3 failover test — it does NOT currently carry the WG underlay handshake. (This is the inverse of the pre-fiber state in §2/§4, where cellular carried mgmt and eth0 was NO-CARRIER.)

Current mgmt-reachability fault (flag for operator — needs on-device follow-up)

The WG control plane is up but no in-tunnel application traffic returns from the router:

# hub, over the tunnel:
ip route get 10.88.0.2      -> 10.88.0.2 dev wg0 src 10.88.0.1   (route correct)
ping -c3 10.88.0.2          -> 100% loss (even 56-byte payloads)
nc -vz 10.88.0.2 22         -> timed out ("Operation now in progress")
# 4 s window: wg 'sent' climbed 244308 -> (SYN retransmits reach router), 'rx' FLAT

Interpretation: the router receives hub traffic (encrypted sent counter grows) but replies with nothing — its in-tunnel return path for ICMP/TCP is broken. Most likely the policy-routing that steers tunnel replies (table 1001, from/to 10.88.0.2 lookup 1001, per §4) or the WG interface's reverse-path was disturbed when the fiber default route (metric 10) came up and displaced cellular (metric 40) as the preferred default. This must be diagnosed/repaired on-device via a console/LAN path (the WG path is the very thing that's down) before Task 2.1 routing changes proceed. Not attempted here (read-only, no-mutate scope).

Facts NOT captured (require device shell — blocked)

• eth0/wan DHCP lease (ATT subnet / gateway / netmask), eth0 carrier flag from the device side.
• Full ip route default table with metrics (expected eth0 metric 10 preferred over cellular metric 40 — strongly implied by the endpoint move to fiber, but not directly read).
• rmnet_mhi0 current CGNAT IP, ip route get 167.172.237.162 / ip route get 99.163.124.171.
• mwan3 status / mwan3 interfaces view of the two ifaces, ip rule table listing.

Re-run §5 in full once console/LAN access to bus01 is restored and the WG mgmt path is repaired.

---

6. WG mgmt tunnel root cause + fix (2026-07-01)

Symptom: after ATT Fiber plugged into eth0 (metric 10 preferred default), WG handshakes were fresh but all in-tunnel data (ICMP/SSH from 10.88.0.1 to 10.88.0.2) returned 100% loss.

Root cause: mwan3 adds all directly-connected networks to the mwan3_connected ipset. 10.88.0.0/24 is a directly-connected route (wgclient1), so mwan3's iptables mwan3_connected chain marks packets destined for that prefix with 0x3f00. The ip rule 9000: not from all fwmark 0/0xf000 lookup main then routes them via the main table, which has no route for 10.88.0.0/24 (wgclient1 routes are only in table 1001 due to ip4table='1001' in UCI). The packet is silently dropped — never reaches WG.

Fix (applied 2026-07-01): Add an ip rule at priority 99 (before any mwan3 fwmark rule) routing packets to 10.88.0.0/24 via table 1001:

ip rule add to 10.88.0.0/24 priority 99 lookup 1001

Persisted as a UCI network rule so netifd recreates it on every boot:

uci add network rule
uci set network.@rule[-1].lookup='1001'
uci set network.@rule[-1].dest='10.88.0.0/24'
uci set network.@rule[-1].priority='99'
uci commit network

Also applied for robustness: wgclient1 mtu=1280, rp_filter=2 persisted via /etc/sysctl.conf and /etc/hotplug.d/iface/99-busrouter-rpfilter (re-asserts on WAN flap).

Failover note: scripts/diag/wg-mgmt-diagnose.sh and scripts/diag/wg-mgmt-fix.sh in this repo implement the fix. The underlying keepalive (persistent_keepalive=25) was already configured from a prior session; the new ip rule was the missing piece.

Phase 8 provisioning profile must include this UCI rule — without it, any device with mwan3 configured will have the WG mgmt tunnel silently drop in-tunnel data.

7. Task 2.1 bench-test findings (2026-07-01)

mwan3 configured with modem_0001 (cellular) + wan (fiber/eth0), policy balanced. Both interfaces tracked online (1.1.1.1 + 8.8.8.8, reliability 1, down=3, up=3 x 5s interval).

Failover drill result:

• ip link set rmnet_mhi0 down -> mwan3 declares modem_0001 error within ~15s (checked)
• Policy shifts to wan (100%) automatically (checked)
• WG mgmt tunnel (riding eth0) stays alive throughout failover (checked)
• Recovery gotcha: ip link set rmnet_mhi0 up does NOT restore the QMI data session. Must use ifdown modem_0001 && ifup modem_0001 to reconnect. After ~20s both interfaces return to balanced: wan(50%) modem_0001(50%) (checked)

Phase 9 pilot: fiber failover (bring fiber down, WG must ride cellular) should be tested once the provisioning robustness fix (section 6 ip rule) is confirmed to survive both uplink combos.