allen/kit-busrouter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
kit-busrouter/docs/deployment/synology-rt2600ac-checklist.md
T
kitadmin 879f97c843 fix: x4078 remote recovery + daemon wait-wedge bug + SPK rebuild 
- docs/incident-log: mark x4078 recovered remotely via Tailscale (no
  physical access needed); document the recovery steps and root cause
- connect-daemon.sh: remove bare `wait` that wedged the retry loop
  forever on the setsid'd tailscaled child — once the tunnel failed the
  daemon could not self-heal until the package was restarted; now the
  main loop falls through start_tunnel (already blocking+retrying) and
  restarts the whole cycle cleanly
- SPK rebuilt (32839680 bytes) with the daemon fix included
- Also documents the two latent lessons: (1) a field router is only
  inaccessible when ALL three paths fail; (2) always check Tailscale +
  QuickConnect before assuming a truck roll is required

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 22:23:53 +00:00

340 lines
19 KiB
Markdown
Raw Permalink Blame History

# Synology RT2600ac — Fleet Deployment Checklist
**Version:** 1.2 · **Last updated:** 2026-07-22 · **Audited against:** x4078 DSS config backup
Every Synology RT2600ac bus router MUST pass ALL sections before departing the yard. Items flagged 🔴 were missed on x4078 and caused remote-access emergencies.
---
## 0. Pre-flight (before touching SRM)
- [ ] **0.1** Insert SD card into router (required for Package Center and third-party packages)
- [ ] **0.2** Power on router; connect LAN port to a PC for setup
- [ ] **0.3** Verify default LAN IP `192.168.1.1` is reachable
---
## 1. SRM First-Time Setup Wizard
- [ ] **1.1** 🔴 **Admin password:** `kitPLANE1!!` (user: `kitadmin`)
- [ ] **1.2** 🔴 **Timezone:** `(GMT-06:00) Central Time (US & Canada)` — Chicago, CST/CDT
- NOT UTC, NOT Central America (Mexico City)
- [ ] **1.3** 🔴 **WiFi Main SSID:** `<DEVICE_ID> Pioneer Admin` (e.g. `x4078 Pioneer Admin`)
- Security: WPA2-PSK · Password: `Pioneer321!`
- Smart Connect: **ON** (single SSID for both bands)
- Do NOT add `_2.4G` or `_5G` suffixes to the SSID
- [ ] **1.4** 🔴 **WiFi Guest SSID:** `<DEVICE_ID> Pioneer` (e.g. `x4078 Pioneer`)
- Security: WPA2-PSK · Password: `Pioneer123`
- Client isolation: ON
- Smart Connect: ON
- Delete any default `SynologyGuest_*` networks
- [ ] **1.5** **LAN subnet:** Keep default `192.168.1.0/24` (lbr0)
- [ ] **1.6** **Guest subnet:** `192.168.2.0/24` (gbr0, SRM default)
- [ ] **1.7** **Hostname:** Set to `<DEVICE_ID>` (e.g. `x4078`)
- [ ] **1.8** **Router login port:** Keep default HTTP 8000 / HTTPS 8001
---
## 2. SRM Control Panel — System Settings
- [ ] **2.1** 🔴 **QuickConnect:** Enable · ID = `<DEVICE_ID>` (e.g. `x4078`)
- Sign in with Synology Account (create `pioneercoach@keylinkit.net` if needed)
- Verify: `https://<DEVICE_ID>.us2.quickconnect.to/webman/index.cgi` loads
- [ ] **2.2** 🔴 **DSM Auto-Update:** Control Panel → System → Update & Restore → Update Settings
- **Automatically check for updates:** ON
- **Automatically install important updates:** ON
- **Update schedule:** Daily, off-peak (e.g. 03:00)
- [ ] **2.3** 🔴 **Database auto-update:** Security Advisor → Settings → Auto-update definitions
- [ ] **2.4** 🔴 **Notifications:** Control Panel → Notification → Enable push/email notifications
- Enable: System health, WAN status changes, package events
- Email: configure SMTP or use SRM notification relay
- [ ] **2.5** **SSH:** Control Panel → Terminal & SNMP → Enable SSH on port 2223
- [ ] **2.6** **Regional Options:** Verify language, time format, and date format
- [ ] **2.7** 🔴 **Disable WPS:** Wi-Fi Connect → WPS → **Disabled** on both bands
- [ ] **2.8** 🔴 **Login page title:** Control Panel → System → Login Portal → Title = `<DEVICE_ID>` (e.g. `x5925`)
- [ ] **2.9** 🔴 **Sign in to Synology Account:** Control Panel → QuickConnect → Sign in with Synology Account
- Required for QuickConnect relay to work. Use `pioneercoach@keylinkit.net` or create if needed.
- [ ] **2.10** 🔴 **Do NOT use DoH:** Network Center → Internet → DNS → Do NOT use DNS over HTTPS
- Standard DNS only: `1.1.1.2`, `9.9.9.9`, `8.8.8.8`
---
## 2b. Security — Safe Access
- [ ] **2b.1** 🔴 **Install Safe Access:** Package Center → Install "Safe Access" (Synology SRM package)
- [ ] **2b.2** 🔴 **Enable malware blocking on Primary network:**
- Safe Access → Primary Network → Web Filter → Block malicious websites
- Enable **Google Safe Browsing**
- [ ] **2b.3** 🔴 **Enable malware blocking on Guest network:**
- Safe Access → Guest Network → Web Filter → Block malicious websites
- Enable Google Safe Browsing
- [ ] **2b.4** 🔴 **Google Safe Browsing API key:** Use `AIzaSyBqjLxf1BpD-gPgMqOV1whr6KtomP_D3MM`
- [ ] **2b.5** 🔴 **Enable Threat Intelligence database:** Safe Access → Settings → Threat Intelligence → ON
- Auto-update definitions
- [ ] **2b.6** 🔴 **DoS Protection:** Network Center → Security → DoS Protection → **Enabled**
- [ ] **2b.7** **Safe Access profiles:** Create a profile for fleet devices with appropriate filtering
---
## 2c. Traffic Control — Bandwidth Management
- [ ] **2c.1** 🔴 **Enable Traffic Control:** Network Center → Traffic Control → Enable
- [ ] **2c.2** 🔴 **Per-device throttle:** Set maximum bandwidth per client to **80% of total WAN capacity**
- Dynamic: adjust based on measured WAN speed
- Guest network: stricter cap (50% of total)
- [ ] **2c.3** 🔴 **Run speed test on new WAN detection:**
- When a WAN interface comes up for the first time, run a speed test to measure capacity
- Set traffic control caps based on measured throughput
- Re-test if WAN IP changes (new ISP/tower)
- [ ] **2c.4** **QoS priority:** Primary network gets higher priority than Guest
- Primary: 70% guaranteed minimum
- Guest: 30% maximum (burst to 50% when idle)
---
## 3. SmartWAN Configuration
- [ ] **3.1** Network Center → Smart WAN → **Load Balancing with Failover** mode
- [ ] **3.2** 🔴 **WAN1 (eth0):** Primary — this will be Eyeride Eyenet
- [ ] **3.3** 🔴 **WAN2 (eth2):** Secondary — this will be Starlink
- [ ] **3.4** **Health check targets:** `1.1.1.2`, `9.9.9.9`, `8.8.8.8`
- Cloudflare malware-filtering (1.1.1.2) + Quad9 (9.9.9.9) + Google (8.8.8.8 fallback)
- [ ] **3.5** **Policy routing:** All traffic balanced; no manual pinning yet
---
## 4. Package Center Setup
- [ ] **4.1** Open Package Center → Accept terms of service when prompted
- [ ] **4.2** Package Center → Settings → Trust Level → **"Any publisher"**
- [ ] **4.3** Verify SD card is recognized (Package Center → Storage)
---
## 5. aiwanbal Package Install
- [ ] **5.1** Download `aiwanbal-1.3-0013.spk` from Gitea:
```
https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer/releases/download/v1.3-0013/aiwanbal-1.3-0013.spk
```
- [ ] **5.2** SCP to router: `scp -O -P 2223 aiwanbal-*.spk kitadmin@192.168.1.1:/tmp/`
- [ ] **5.3** Install via SSH: `echo "kitPLANE1!!" | sudo -S synopkg install /tmp/aiwanbal-*.spk`
- [ ] **5.4** Start: `echo "kitPLANE1!!" | sudo -S synopkg start aiwanbal`
- [ ] **5.5** 🔴 **Configure for mobile/bus environment:**
```sh
# In /etc/aiwanbal/aiwanbal.conf, set:
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf ENVIRONMENT mobile
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf SPEEDTEST_AUTO_ENABLED 0
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf DEVICE_NAME x4078
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf CHECK_INTERVAL 30
```
- [ ] **5.6** 🔴 **Set Eyeride GPS config** (if Eyeride password is available):
```sh
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_ENABLE 1
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_IP 192.168.10.1
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_PASSWORD <eyeride-password>
```
- [ ] **5.7** 🔴 **Verify ISP detection:** Check the aiwanbal UI at `/webman/3rdparty/aiwanbal/index.html`
- WAN1 should show carrier/ISP (Eyeride → T-Mobile/Verizon depending on SIM)
- WAN2 should show Starlink
- If not detecting, run auto-detection: `sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth0; sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth2`
---
## 6. kittunnel Package Install (Reverse SSH Tunnel)
- [ ] **6.1** Assign a unique port for this bus (see fleet port registry)
- [ ] **6.2** Download hardened kittunnel SPK (v1.1+) from Gitea
- [ ] **6.3** SCP to router and install via synopkg
- [ ] **6.4** Verify tunnel.conf has correct `REMOTE_PORT`
- [ ] **6.5** Start: `sudo synopkg start kittunnel`
- [ ] **6.6** Verify tunnel is connected: check port is LISTENING on VPS (162.243.83.36)
- [ ] **6.7** Verify end-to-end: `ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"`
---
## 6a. Kit-Connect (Replaces kittunnel + Tailscale — single SPK)
Kit-connect is the unified replacement for Sections 6 and 7. One SPK installs both the reverse SSH tunnel and Tailscale.
- [ ] **6a.1** Download SPK from Gitea: `https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk`
- [ ] **6a.2** SCP to router: `scp -O -P 2223 kit-connect-0.1-0001.spk kitadmin@192.168.1.1:/tmp/`
- [ ] **6a.3** Install: `echo "kitPLANE1!!" | sudo -S /usr/syno/bin/synopkg install /tmp/kit-connect-0.1-0001.spk`
- [ ] **6a.4** Verify config: `grep TUNNEL_LOCAL_SSH_PORT /etc/kit-connect/connect.conf` must show `2223` (NOT 22)
- [ ] **6a.5** ONLY after confirming two alternative access paths (QuickConnect + Tailscale), stop old tunnel: `sudo synopkg stop kittunnel`
- [ ] **6a.6** Start kit-connect: `sudo synopkg start kit-connect`
- [ ] **6a.7** Verify port is LISTENING on VPS (162.243.83.36) for this device
**⚠️ DO NOT attempt switchover remotely.** The old tunnel is the only management path. Switch only when physically present or when QuickConnect + Tailscale are both confirmed working.
## 7. Tailscale Install (Management Tailnet)
- [ ] **7.1** Download Tailscale static `linux_arm` binary
- [ ] **7.2** Install to `/usr/local/bin/tailscale` and `/usr/local/bin/tailscaled`
- [ ] **7.3** Start tailscaled in userspace mode:
```sh
/usr/bin/setsid /usr/local/bin/tailscaled \
--statedir=/var/packages/Tailscale/var/state \
--tun=userspace-networking \
--socket=/var/packages/Tailscale/var/run/tailscaled.sock &
```
- [ ] **7.4** 🔴 Join bus fleet tailnet:
```sh
tailscale up --auth-key tskey-auth-... --hostname <DEVICE_ID> --accept-routes=false --accept-dns=false
```
- [ ] **7.5** 🔴 Install rc.local with **correct paths** (`/usr/bin/setsid`, `/bin/mkdir`):
```sh
echo '#!/bin/sh' | sudo tee /usr/syno/etc/rc.d/rc.local
echo '/bin/mkdir -p /var/packages/Tailscale/var/state /var/packages/Tailscale/var/run' | sudo tee -a /usr/syno/etc/rc.d/rc.local
echo '/usr/bin/setsid /usr/local/bin/tailscaled --statedir=/var/packages/Tailscale/var/state --tun=userspace-networking --socket=/var/packages/Tailscale/var/run/tailscaled.sock &' | sudo tee -a /usr/syno/etc/rc.d/rc.local
sudo chmod +x /usr/syno/etc/rc.d/rc.local
```
- [ ] **7.6** 🔴 **Reboot test:** `sudo reboot` → wait 2 min → verify tailscaled is running
- [ ] **7.7** Verify Tailscale status shows router online at `pioneercoach@`
---
## 8. Device Identity & Hostname (Persistent)
- [ ] **8.1** 🔴 Create `/etc/busrouter/device-id` with `<DEVICE_ID>`:
```sh
echo "<DEVICE_ID>" | sudo tee /etc/busrouter/device-id
```
- [ ] **8.2** 🔴 Create `/etc/busrouter/version` with `0.0`:
```sh
echo "0.0" | sudo tee /etc/busrouter/version
```
- [ ] **8.3** 🔴 **Persistent hostname** — set in ALL locations:
```sh
sudo hostname <DEVICE_ID>
echo "<DEVICE_ID>" | sudo tee /etc/hostname
```
Also check SRM UI: Control Panel → Network → Router Name
- [ ] **8.4** 🔴 **DHCP hostname:** SRM → Network Center → Internet → WAN → edit → Hostname = `<DEVICE_ID>`
(Without this, DHCP sends `SynologyRouter` to upstream)
---
## 9. Telemetry Agent
- [ ] **9.1** SCP `telemetry-synology.sh` to `/usr/lib/busrouter/telemetry-synology.sh`
- [ ] **9.2** Configure `TELEMETRY_HUB` to fleet hub endpoint:
```sh
# Default: http://167.172.237.162:8080/api/telemetry
```
- [ ] **9.3** Add cron job (every 60s):
```sh
echo '* * * * * root TELEMETRY_HUB=http://167.172.237.162:8080/api/telemetry /usr/lib/busrouter/telemetry-synology.sh 2>/dev/null' | sudo tee -a /etc/crontab
```
- [ ] **9.4** Dry-run test: `sudo /usr/lib/busrouter/telemetry-synology.sh`
## 9b. Weekly Config Backup
- [ ] **9b.1** SCP `scripts/backup-config.sh` to `/usr/lib/busrouter/backup-config.sh`
- [ ] **9b.2** Add weekly cron (Sunday 03:00):
```sh
echo '0 3 * * 0 root BACKUP_HUB=http://167.172.237.162:8080 /usr/lib/busrouter/backup-config.sh 2>/dev/null' | sudo tee -a /etc/crontab
```
- [ ] **9b.3** Dry-run test: `sudo /usr/lib/busrouter/backup-config.sh`
- [ ] **9b.4** Verify backup appears on hub at `/opt/busfleet-hub/backups/<DEVICE_ID>/`
- [ ] **9b.5** 🔴 If a router dies, the most recent backup is the replacement config — never more than 7 days stale
---
## 10. Reboot & Final Verification
- [ ] **10.1** Reboot: `sudo reboot`
- [ ] **10.2** After reboot, verify ALL of these survive:
| Check | Command | Expected |
|-------|---------|----------|
| Hostname | `hostname` | `<DEVICE_ID>` |
| Device ID | `cat /etc/busrouter/device-id` | `<DEVICE_ID>` |
| aiwanbal | `sudo synopkg status aiwanbal` | started |
| kittunnel | `sudo synopkg status kittunnel` | started |
| Tailscale | `ps \| grep tailscaled` | running |
| Tailscale status | `tailscale status` | online, pioneercoach@ |
| WiFi SSID | Check phone WiFi scan | `<DEVICE_ID> Pioneer Admin` |
| Guest WiFi | Check phone WiFi scan | `<DEVICE_ID> Pioneer` |
| QuickConnect | Browser → `https://<DEVICE_ID>.us2.quickconnect.to` | SRM login loads |
| SSH tunnel | From lab: `ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"` | `<DEVICE_ID>` |
| Tailscale path | From lab: `ssh -J root@172.18.0.1 -p 2223 kitadmin@<tailscale-ip> "hostname"` | `<DEVICE_ID>` |
| WAN1 carrier | aiwanbal UI | Shows ISP |
| WAN2 carrier | aiwanbal UI | Shows Starlink |
| Telemetry cron | `grep telemetry /etc/crontab` | present |
| Backup cron | `grep backup /etc/crontab` | present |
| Auto-update | SRM UI → Update & Restore | Auto-check ON |
| Notifications | SRM UI → Notification | Enabled |
| WPS | Wi-Fi Connect → WPS | Disabled both bands |
| Safe Access | Package Center → Safe Access | Installed, malware ON |
| DoS Protection | Network Center → Security | Enabled |
| Traffic Control | Network Center → Traffic Control | Enabled, 80% cap |
| DoH | Network Center → Internet → DNS | OFF |
- [ ] **10.3** 🔴 **Run the reboot test before the bus leaves the yard.** There is no second chance.
- [ ] **10.4** Store credentials in vault under `busrouter/<DEVICE_ID>`
- [ ] **10.5** Record device in fleet registry with IPs, port, keys
---
## Fleet Port Registry
| Device | VPS Port | Tailscale IP | QuickConnect URL | Location | Notes |
|--------|----------|-------------|-------------------|----------|-------|
| x4078 | 2226 | `100.116.71.43` | `https://x4078.us2.quickconnect.to` | Bus (field) | Eyeride + Starlink |
| x4324 | 2225 | `100.91.204.32` | `https://x4324.us2.quickconnect.to` | Bus (field) | kit-connect |
| x4662 | 2229 | TBD | `https://x4662.us2.quickconnect.to` | Bus (field) | MR2200ac mesh satellite |
| x5925 | 2230 | TBD | `https://x5925.us2.quickconnect.to` | Office bench | |
| x9002 | 2231 | `100.82.60.8` | N/A (GL-XE3000) | Office bench | |
---
## Post-Deployment Rules 🔴
1. **Never tear down a working tunnel without two confirmed alternative access paths** (QuickConnect + Tailscale, or QuickConnect + LAN). The reverse SSH tunnel is the primary management channel — once it's gone, you are dead in the water.
2. **TUNNEL_LOCAL_SSH_PORT must be 2223 for Synology.** Synology SRM SSH listens on port 2223, not 22. The kit-connect wizard default was corrected to 2223 on 2026-07-22 after this mistake took down x4078.
3. **Don't restart sshd on the VPS** while tunnels are active. It kills established forwarding sessions and authorized_keys changes don't require a restart.
4. **If quickconnect.to is the only backup**, confirm it's actually working before touching anything. The relay can be slow or locked out by DoS protection.
5. **The old kittunnel is stable.** If it's working, leave it alone until a planned maintenance window with physical access.
## Incident Log — 2026-07-22
**Root cause**: The kit-connect wizard.sh used `command -v` (not available on SRM busybox) and defaulted `TUNNEL_LOCAL_SSH_PORT=22` instead of 2223. Combined with attempting remote switchover from old tunnel to kit-connect without confirmed backup paths, this took x4078 and x6340 offline.
**Resolution for x4662**: Kittunnel restarted via Package Center through QuickConnect. Tunnel stable on port 2229.
**Resolution for x4324**: Kit-connect daemon reconnected after authorized_keys fix on VPS. Tunnel stable on port 2225.
**Resolution for x4078 (2026-07-22, remote — NO physical access needed)**: Recovered entirely over the
still-live **Tailscale** path (`100.116.71.43`, SRM SSH `:2223`) — QuickConnect was also up the whole
time. The reverse tunnel was the *only* dead path. Root cause on-device: `TUNNEL_LOCAL_SSH_PORT=22` in
`/etc/kit-connect/connect.conf` (SRM sshd listens on 2223). Fix: SSH in over Tailscale from a tailnet
node (e.g. the hub), `sed -i 's/^TUNNEL_LOCAL_SSH_PORT=.*/TUNNEL_LOCAL_SSH_PORT=2223/'` the conf, then
`synopkg stop kit-connect; synopkg start kit-connect`. Tunnel came back on port 2226; end-to-end verified.
Lesson: **a field router is only "inaccessible" if ALL of {Tailscale, QuickConnect, reverse tunnel} are
down — check each before assuming physical access is required.**
**Latent daemon bug found**: `connect-daemon.sh` main loop ends each iteration with a bare `wait`, which
blocks forever on the `setsid`-launched `tailscaled` child. Once the tunnel started failing, the loop
wedged for ~58 min and never self-healed on the corrected config until the package was restarted. Fix
the `wait` (e.g. `wait "$SSH_PID"` or `wait -n`) so the retry loop can recover on its own.
**SPK fixed**: Rebuilt with `which` instead of `command -v` and `TUNNEL_LOCAL_SSH_PORT=2223`. Available at `https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk`.
---
## x4078 Audit (2026-07-21) — Findings from DSS backup
🔴 **Critical — must fix remotely or on next physical access:**
| Gap | Severity | Fix |
|-----|----------|-----|
| SSID: `x4078 Admin` → should be `x4078 Pioneer Admin` | Medium | SRM Wi-Fi settings |
| Guest SSID: `x4078 Pioneer Guest` → should be `x4078 Pioneer` | Medium | SRM Wi-Fi settings |
| Band SSIDs have `_2.4G`/`_5G` suffixes | Low | Disable per-band SSIDs, use SmartConnect only |
| Old `SynologyGuest_*` networks still present | Low | Delete in SRM Wi-Fi settings |
| Timezone was Central America, not CST/CDT (Chicago) | High | Fixed via QuickConnect SRM UI |
| Database auto-update OFF | High | Fixed via QuickConnect SRM UI |
| Notifications not enabled | High | Fixed via QuickConnect SRM UI |
| DHCP hostname sends `SynologyRouter` | Medium | SRM Network Center → WAN → Hostname |
| Hostname persistence failing on reboot | High | Verify `/etc/hostname` + SRM Network Center |
| ISP/carrier not detecting per WAN | Medium | Run `aiwanbal-detect.sh` on each WAN |
| rc.local has wrong `/bin/setsid` path → `/usr/bin/setsid` | High | Fixed already |
| Tailscale not boot-tested | Critical | Must test on next physical access |
Reference in New Issue View Git Blame Copy Permalink