kit-busrouter/docs/handoff/2026-07-01-wg-mgmt-incident-and-resume.md

# Handoff — WG mgmt incident + resume Task 2.1 (2026-07-01)

**Audience:** local Claude Code running on a machine with **LAN access** to the bench router.
**Why this exists:** the cloud session hit the router only over the WireGuard mgmt tunnel, which is
currently DOWN. The remaining work must be driven from the LAN side. This doc + the two scripts in
`scripts/diag/` contain everything needed to (1) restore WG mgmt, (2) make the tunnel robust across
either WAN, then (3) resume Task 2.1.

---

## 1. Current project state

- **Phase 0** (0.1 SSH, 0.2 recon, 0.3 package skeleton) — DONE, committed, plan boxes checked.
- **Phase 1** (scoring/decision pure-shell libs + bats) — DONE, committed.
- **Task 2.0** (install `mwan3 2.8.15-2` from GL's own feed; disable GL `kmwan`) — DONE + independently
  verified. Commit `4f33ca3`. Recon doc §4. Default route + WG mgmt survived that change.
- **BLOCKER (this doc):** WG mgmt tunnel to the bench router (`10.88.0.2`) went dead the moment ATT
  fiber was plugged into `eth0`. On-device Task 2.1 work is blocked until mgmt is restored.

Plan of record: `docs/superpowers/plans/2026-06-30-pioneer-bus-router-balancer.md`
Recon / ground truth: `docs/recon/2026-06-30-xe3000-environment.md` (see §5 for the fiber pre-check).

---

## 2. Environment & access

| Thing | Value |
|------|-------|
| Router | GL.iNet **GL-XE3000**, OpenWrt 21.02-SNAPSHOT, busybox/ash, kernel 5.4 |
| Router LAN | **`192.168.8.1`** — `ssh root@192.168.8.1`, password **`kitPLANE1!!`** (dropbear) |
| Router WG mgmt IP | `10.88.0.2` (iface `wgclient1`, route table `1001`) |
| Uplink 1 (cellular) | `rmnet_mhi0` — T-Mobile CGNAT, default-route metric **40** |
| Uplink 2 (fiber, = "Starlink" stand-in) | `eth0` / UCI iface `wan` — ATT fiber, metric **10** (now the preferred default). Public IP seen by hub: `99.163.124.171` |
| Fleet hub | DO droplet **`kit-fleet-hub`**, public **`167.172.237.162`**, WG **`10.88.0.1`**, wg iface `wg0`. Cloud session reaches it via `mcp__lab-do__do_ssh_command droplet="kit-fleet-hub"`. |
| NOT the hub | `162.243.83.36` is a *different* lab server — ignore it for this work. |

**Reachability rule:** while WG is down, the **LAN is the only way in**. The hub has NO route to
`192.168.8.1` (router LAN is private); and no interactive shell survives over the broken tunnel.

---

## 3. The incident — evidence gathered so far

**Symptom:** fiber plugged into `eth0` → fiber (metric 10) became the default route → the WG *underlay*
roamed onto fiber (hub now sees the peer at `99.163.124.171:44881`, was T-Mobile CGNAT `100.x`).
Handshakes succeed **both** directions and are always "fresh", but **no data flows** — ICMP/SSH over
the tunnel get zero replies in either direction.

**Verified CLEAN on the hub** (`167.172.237.162`, checked directly via DO API):
- `wg show` peer allowed-ips = **`10.88.0.2/32`** ✓
- `ip route get 10.88.0.2` → **`dev wg0 src 10.88.0.1`** ✓; `net.ipv4.ip_forward = 1` ✓
- Firewall: `nft list ruleset` **empty**; iptables INPUT/FORWARD/OUTPUT policy **ACCEPT, 0 packets** ✓
- Endpoint roam **succeeded** (hub has the fiber endpoint, fresh handshake).
- → The three "usual hub culprits" (allowed-ips, firewall, return route) are **all ruled out**.

**Verified on the router** (LAN side):
- `rp_filter` = **0** on all interfaces → **NOT** the cause (an earlier hypothesis; discard it).
- Routing correct: `ip route get 10.88.0.1 from 10.88.0.2` → **`dev wgclient1 table 1001`** ✓
- `wgclient1` MTU = 1420 (WG default).
- Source-bound ping (`ping -I 10.88.0.2 10.88.0.1`) **fails**, but `wg` transfer incremented only ~32
  bytes — suspiciously small for 3 ICMP packets (~430B expected), so it's unclear the data packets
  actually hit the wire.

**Measurement caveat:** hub-side and router-side counters were sampled at *different moments*, so
cross-side counter comparisons are unreliable. The §B capture in the diagnose script fixes this by
capturing on the router itself while it generates traffic.

**Ruled out:** rp_filter (=0), hub firewall (none), hub return route (correct), hub allowed-ips
(correct), MTU-for-tiny-pings (a 144B encapsulated ping ≈ the 148B handshake that succeeds — MTU only
kills *large* packets like SSH data, not pings).

**Leading hypotheses, in priority order:**
1. **`persistent-keepalive` missing** on the router's `wgclient1` peer → behind ATT fiber NAT the
   return mapping expires between handshakes (T-Mobile CGNAT likely masked this). *Not yet confirmed —
   we never captured the router's `wg show`.* **Get `wg show` first.**
2. **Router-side cryptokey routing / peer AllowedIPs** doesn't include `10.88.0.1` → inbound replies
   silently dropped by WG. Also revealed by `wg show`.
3. **MTU/PMTU black-hole** for large packets — unlikely to block pings but WILL break SSH data; worth
   pre-empting with `wgclient1 mtu 1280` + MSS clamp.

---

## 4. DO THIS FIRST — restore mgmt (LAN, `root@192.168.8.1`)

### Step 1 — diagnose (read-only, safe)
```sh
sh scripts/diag/wg-mgmt-diagnose.sh        # or paste it; it's plain ash
```
The single most important output is **`wg show`** (§A) — we have never seen the router's peer config.
Look for:
- `persistent keepalive: (none)` → hypothesis #1 is live.
- `allowed ips:` on the hub peer — must include `10.88.0.1` (else hypothesis #2).
- §B capture: does `[eth0]` show **inbound** `167.172.237.162.51820 →` packets during the ping?
  - inbound present → replies arrive on fiber → decrypt/inner issue (chase AllowedIPs/keys).
  - outbound only → router's packets aren't leaving, or hub isn't returning → coordinate a hub capture.

### Step 2 — apply the fix (mutating; `wg-mgmt-fix.sh` is gated, read its header)
Default fix order (safe, reversible, and each independently supports "works over either WAN"):
```sh
sh scripts/diag/wg-mgmt-fix.sh keepalive   # set persistent_keepalive=25  (try FIRST)
#   re-test:  ping -c3 -I 10.88.0.2 10.88.0.1   → expect replies
sh scripts/diag/wg-mgmt-fix.sh mtu         # set wgclient1 mtu=1280       (if keepalive alone didn't fix)
sh scripts/diag/wg-mgmt-fix.sh rpfilter    # persist loose rp_filter (already 0 at runtime; makes it durable)
```
Each is `uci set … ; uci commit ; ifup wgclient1` — the tunnel bounces (~5s) but nothing else is touched.

### Step 3 — verify restored
```sh
ping -c3 -W2 -I 10.88.0.2 10.88.0.1        # replies from the hub = tunnel data path healthy
```
When that replies, mgmt over WG is back. (The cloud session can then re-confirm from the hub with
`ssh -i /root/.ssh/busfleet root@10.88.0.2 'ubus call system board'` — but you can also just keep
driving from the LAN.)

---

## 5. Robustness goal — WG mgmt must survive on EITHER WAN

Bake these in (they're the durable fix, not just incident cleanup), then fold into the Phase 8
provisioning profile:
- `persistent_keepalive=25` — holds the NAT mapping open on whichever WAN is active.
- `wgclient1 mtu=1280` — fits under cellular **and** ATT-fiber/PPPoE path MTUs; kills PMTU black-holes.
- rp_filter loose (`=2`) persisted via `/etc/sysctl.conf`; if GL re-tightens on WAN flap, add a
  `/etc/hotplug.d/iface/` hook that re-asserts it (the fix script writes one under `rpfilter`).
- The mgmt return rule `from 10.88.0.2 lookup 1001` is already default-route-independent — leave it.
- After the fix, **prove failover**: `ifdown wan` (fiber) → mgmt should ride cellular; `ifup wan` →
  rides fiber again. Do this only once mgmt is confirmed healthy, and see the guardrail below.

---

## 6. THEN resume Task 2.1 (per the plan)

Follow `superpowers:subagent-driven-development` (implement → spec review → code-quality review →
commit → check the plan box), same as Tasks 0.3 / 2.0.

Task 2.1 files & specifics (from the plan):
- `package/kit-busrouter/files/usr/lib/busrouter/wan-mwan.sh`
- mwan3 config: members **cellular** (`modem_0001` / `rmnet_mhi0`) + **fiber-as-WAN2** (`wan` / `eth0`);
  policy `balanced`; `track_ip` 1.1.1.1 & 8.8.8.8; `reliability 1`; `down 3`; `up 3`.
- `mwan_set_weight <member> <weight>` = `uci set mwan3.<member>.weight` + `mwan3 restart`, **guarded by
  `weight_changed`** (already implemented in `lib-decide.sh`) so we don't flush conntrack needlessly.
- Rewrite mwan3's stock config — it ships referencing `wan/wwan/tethering`, none of which match our
  live ifaces (noted in Task 2.0).

**Failover drill — SAFE direction (critical):**
- Fiber currently carries the WG mgmt underlay, so it is **SAFE to bounce CELLULAR** (`rmnet_mhi0`) for
  the failover test, and **NEVER down `eth0`/fiber** while it carries mgmt.
- **Always re-check first:** `ip route get 167.172.237.162` tells you which uplink the WG underlay is
  on *right now*; down the OTHER one. (After you set persistent_keepalive + prove either-WAN failover,
  the mgmt-bearing uplink can shift — re-check every time.)

---

## 7. Guardrails (learned this session)

- **No dead-man watchdog on this firmware:** `setsid` and `nohup` are both absent, and
  `start-stop-daemon -x /bin/sh` self-matches the session shell. So you cannot arm a reliable
  auto-rollback over SSH. Mitigation: do risky routing changes **from the LAN side** (you're physically
  present) so a mistake doesn't lock you out; verify mgmt after every mutation.
- **No `base64` on the router** — transfer files by decoding on the hub/LAN box and piping cleartext
  into `cat >` on the router, or just paste scripts.
- **busybox `ping` has no `-M do`** — can't do a clean DF/PMTU probe with it; infer MTU issues from
  large-vs-small behavior or `tracepath` if present.
- Verify mgmt access (a WG-sourced ping or SSH) after **every** routing/firewall/mwan3 change.
```