# Handoff — WG mgmt incident + resume Task 2.1 (2026-07-01) **Audience:** local Claude Code running on a machine with **LAN access** to the bench router. **Why this exists:** the cloud session hit the router only over the WireGuard mgmt tunnel, which is currently DOWN. The remaining work must be driven from the LAN side. This doc + the two scripts in `scripts/diag/` contain everything needed to (1) restore WG mgmt, (2) make the tunnel robust across either WAN, then (3) resume Task 2.1. --- ## 1. Current project state - **Phase 0** (0.1 SSH, 0.2 recon, 0.3 package skeleton) — DONE, committed, plan boxes checked. - **Phase 1** (scoring/decision pure-shell libs + bats) — DONE, committed. - **Task 2.0** (install `mwan3 2.8.15-2` from GL's own feed; disable GL `kmwan`) — DONE + independently verified. Commit `4f33ca3`. Recon doc §4. Default route + WG mgmt survived that change. - **BLOCKER (this doc):** WG mgmt tunnel to the bench router (`10.88.0.2`) went dead the moment ATT fiber was plugged into `eth0`. On-device Task 2.1 work is blocked until mgmt is restored. Plan of record: `docs/superpowers/plans/2026-06-30-pioneer-bus-router-balancer.md` Recon / ground truth: `docs/recon/2026-06-30-xe3000-environment.md` (see §5 for the fiber pre-check). --- ## 2. Environment & access | Thing | Value | |------|-------| | Router | GL.iNet **GL-XE3000**, OpenWrt 21.02-SNAPSHOT, busybox/ash, kernel 5.4 | | Router LAN | **`192.168.8.1`** — `ssh root@192.168.8.1`, password **`kitPLANE1!!`** (dropbear) | | Router WG mgmt IP | `10.88.0.2` (iface `wgclient1`, route table `1001`) | | Uplink 1 (cellular) | `rmnet_mhi0` — T-Mobile CGNAT, default-route metric **40** | | Uplink 2 (fiber, = "Starlink" stand-in) | `eth0` / UCI iface `wan` — ATT fiber, metric **10** (now the preferred default). Public IP seen by hub: `99.163.124.171` | | Fleet hub | DO droplet **`kit-fleet-hub`**, public **`167.172.237.162`**, WG **`10.88.0.1`**, wg iface `wg0`. Cloud session reaches it via `mcp__lab-do__do_ssh_command droplet="kit-fleet-hub"`. | | NOT the hub | `162.243.83.36` is a *different* lab server — ignore it for this work. | **Reachability rule:** while WG is down, the **LAN is the only way in**. The hub has NO route to `192.168.8.1` (router LAN is private); and no interactive shell survives over the broken tunnel. --- ## 3. The incident — evidence gathered so far **Symptom:** fiber plugged into `eth0` → fiber (metric 10) became the default route → the WG *underlay* roamed onto fiber (hub now sees the peer at `99.163.124.171:44881`, was T-Mobile CGNAT `100.x`). Handshakes succeed **both** directions and are always "fresh", but **no data flows** — ICMP/SSH over the tunnel get zero replies in either direction. **Verified CLEAN on the hub** (`167.172.237.162`, checked directly via DO API): - `wg show` peer allowed-ips = **`10.88.0.2/32`** ✓ - `ip route get 10.88.0.2` → **`dev wg0 src 10.88.0.1`** ✓; `net.ipv4.ip_forward = 1` ✓ - Firewall: `nft list ruleset` **empty**; iptables INPUT/FORWARD/OUTPUT policy **ACCEPT, 0 packets** ✓ - Endpoint roam **succeeded** (hub has the fiber endpoint, fresh handshake). - → The three "usual hub culprits" (allowed-ips, firewall, return route) are **all ruled out**. **Verified on the router** (LAN side): - `rp_filter` = **0** on all interfaces → **NOT** the cause (an earlier hypothesis; discard it). - Routing correct: `ip route get 10.88.0.1 from 10.88.0.2` → **`dev wgclient1 table 1001`** ✓ - `wgclient1` MTU = 1420 (WG default). - Source-bound ping (`ping -I 10.88.0.2 10.88.0.1`) **fails**, but `wg` transfer incremented only ~32 bytes — suspiciously small for 3 ICMP packets (~430B expected), so it's unclear the data packets actually hit the wire. **Measurement caveat:** hub-side and router-side counters were sampled at *different moments*, so cross-side counter comparisons are unreliable. The §B capture in the diagnose script fixes this by capturing on the router itself while it generates traffic. **Ruled out:** rp_filter (=0), hub firewall (none), hub return route (correct), hub allowed-ips (correct), MTU-for-tiny-pings (a 144B encapsulated ping ≈ the 148B handshake that succeeds — MTU only kills *large* packets like SSH data, not pings). **Leading hypotheses, in priority order:** 1. **`persistent-keepalive` missing** on the router's `wgclient1` peer → behind ATT fiber NAT the return mapping expires between handshakes (T-Mobile CGNAT likely masked this). *Not yet confirmed — we never captured the router's `wg show`.* **Get `wg show` first.** 2. **Router-side cryptokey routing / peer AllowedIPs** doesn't include `10.88.0.1` → inbound replies silently dropped by WG. Also revealed by `wg show`. 3. **MTU/PMTU black-hole** for large packets — unlikely to block pings but WILL break SSH data; worth pre-empting with `wgclient1 mtu 1280` + MSS clamp. --- ## 4. DO THIS FIRST — restore mgmt (LAN, `root@192.168.8.1`) ### Step 1 — diagnose (read-only, safe) ```sh sh scripts/diag/wg-mgmt-diagnose.sh # or paste it; it's plain ash ``` The single most important output is **`wg show`** (§A) — we have never seen the router's peer config. Look for: - `persistent keepalive: (none)` → hypothesis #1 is live. - `allowed ips:` on the hub peer — must include `10.88.0.1` (else hypothesis #2). - §B capture: does `[eth0]` show **inbound** `167.172.237.162.51820 →` packets during the ping? - inbound present → replies arrive on fiber → decrypt/inner issue (chase AllowedIPs/keys). - outbound only → router's packets aren't leaving, or hub isn't returning → coordinate a hub capture. ### Step 2 — apply the fix (mutating; `wg-mgmt-fix.sh` is gated, read its header) Default fix order (safe, reversible, and each independently supports "works over either WAN"): ```sh sh scripts/diag/wg-mgmt-fix.sh keepalive # set persistent_keepalive=25 (try FIRST) # re-test: ping -c3 -I 10.88.0.2 10.88.0.1 → expect replies sh scripts/diag/wg-mgmt-fix.sh mtu # set wgclient1 mtu=1280 (if keepalive alone didn't fix) sh scripts/diag/wg-mgmt-fix.sh rpfilter # persist loose rp_filter (already 0 at runtime; makes it durable) ``` Each is `uci set … ; uci commit ; ifup wgclient1` — the tunnel bounces (~5s) but nothing else is touched. ### Step 3 — verify restored ```sh ping -c3 -W2 -I 10.88.0.2 10.88.0.1 # replies from the hub = tunnel data path healthy ``` When that replies, mgmt over WG is back. (The cloud session can then re-confirm from the hub with `ssh -i /root/.ssh/busfleet root@10.88.0.2 'ubus call system board'` — but you can also just keep driving from the LAN.) --- ## 5. Robustness goal — WG mgmt must survive on EITHER WAN Bake these in (they're the durable fix, not just incident cleanup), then fold into the Phase 8 provisioning profile: - `persistent_keepalive=25` — holds the NAT mapping open on whichever WAN is active. - `wgclient1 mtu=1280` — fits under cellular **and** ATT-fiber/PPPoE path MTUs; kills PMTU black-holes. - rp_filter loose (`=2`) persisted via `/etc/sysctl.conf`; if GL re-tightens on WAN flap, add a `/etc/hotplug.d/iface/` hook that re-asserts it (the fix script writes one under `rpfilter`). - The mgmt return rule `from 10.88.0.2 lookup 1001` is already default-route-independent — leave it. - After the fix, **prove failover**: `ifdown wan` (fiber) → mgmt should ride cellular; `ifup wan` → rides fiber again. Do this only once mgmt is confirmed healthy, and see the guardrail below. --- ## 6. THEN resume Task 2.1 (per the plan) Follow `superpowers:subagent-driven-development` (implement → spec review → code-quality review → commit → check the plan box), same as Tasks 0.3 / 2.0. Task 2.1 files & specifics (from the plan): - `package/kit-busrouter/files/usr/lib/busrouter/wan-mwan.sh` - mwan3 config: members **cellular** (`modem_0001` / `rmnet_mhi0`) + **fiber-as-WAN2** (`wan` / `eth0`); policy `balanced`; `track_ip` 1.1.1.1 & 8.8.8.8; `reliability 1`; `down 3`; `up 3`. - `mwan_set_weight ` = `uci set mwan3..weight` + `mwan3 restart`, **guarded by `weight_changed`** (already implemented in `lib-decide.sh`) so we don't flush conntrack needlessly. - Rewrite mwan3's stock config — it ships referencing `wan/wwan/tethering`, none of which match our live ifaces (noted in Task 2.0). **Failover drill — SAFE direction (critical):** - Fiber currently carries the WG mgmt underlay, so it is **SAFE to bounce CELLULAR** (`rmnet_mhi0`) for the failover test, and **NEVER down `eth0`/fiber** while it carries mgmt. - **Always re-check first:** `ip route get 167.172.237.162` tells you which uplink the WG underlay is on *right now*; down the OTHER one. (After you set persistent_keepalive + prove either-WAN failover, the mgmt-bearing uplink can shift — re-check every time.) --- ## 7. Guardrails (learned this session) - **No dead-man watchdog on this firmware:** `setsid` and `nohup` are both absent, and `start-stop-daemon -x /bin/sh` self-matches the session shell. So you cannot arm a reliable auto-rollback over SSH. Mitigation: do risky routing changes **from the LAN side** (you're physically present) so a mistake doesn't lock you out; verify mgmt after every mutation. - **No `base64` on the router** — transfer files by decoding on the hub/LAN box and piping cleartext into `cat >` on the router, or just paste scripts. - **busybox `ping` has no `-M do`** — can't do a clean DF/PMTU probe with it; infer MTU issues from large-vs-small behavior or `tracepath` if present. - Verify mgmt access (a WG-sourced ping or SSH) after **every** routing/firewall/mwan3 change. ```