kit-busrouter/docs/deployment/synology-rt2600ac-checklist.md

# Synology RT2600ac — Fleet Deployment Checklist

**Version:** 1.2 · **Last updated:** 2026-07-22 · **Audited against:** x4078 DSS config backup

Every Synology RT2600ac bus router MUST pass ALL sections before departing the yard. Items flagged 🔴 were missed on x4078 and caused remote-access emergencies.

---

## 0. Pre-flight (before touching SRM)

- [ ] **0.1** Insert SD card into router (required for Package Center and third-party packages)
- [ ] **0.2** Power on router; connect LAN port to a PC for setup
- [ ] **0.3** Verify default LAN IP `192.168.1.1` is reachable

---

## 1. SRM First-Time Setup Wizard

- [ ] **1.1** 🔴 **Admin password:** `kitPLANE1!!` (user: `kitadmin`)
- [ ] **1.2** 🔴 **Timezone:** `(GMT-06:00) Central Time (US & Canada)` — Chicago, CST/CDT
  - NOT UTC, NOT Central America (Mexico City)
- [ ] **1.3** 🔴 **WiFi Main SSID:** `<DEVICE_ID> Pioneer Admin` (e.g. `x4078 Pioneer Admin`)
  - Security: WPA2-PSK · Password: `Pioneer321!`
  - Smart Connect: **ON** (single SSID for both bands)
  - Do NOT add `_2.4G` or `_5G` suffixes to the SSID
- [ ] **1.4** 🔴 **WiFi Guest SSID:** `<DEVICE_ID> Pioneer` (e.g. `x4078 Pioneer`)
  - Security: WPA2-PSK · Password: `Pioneer123`
  - Client isolation: ON
  - Smart Connect: ON
  - Delete any default `SynologyGuest_*` networks
- [ ] **1.5** **LAN subnet:** Keep default `192.168.1.0/24` (lbr0)
- [ ] **1.6** **Guest subnet:** `192.168.2.0/24` (gbr0, SRM default)
- [ ] **1.7** **Hostname:** Set to `<DEVICE_ID>` (e.g. `x4078`)
- [ ] **1.8** **Router login port:** Keep default HTTP 8000 / HTTPS 8001

---

## 2. SRM Control Panel — System Settings

- [ ] **2.1** 🔴 **QuickConnect:** Enable · ID = `<DEVICE_ID>` (e.g. `x4078`)
  - Sign in with Synology Account (create `pioneercoach@keylinkit.net` if needed)
  - Verify: `https://<DEVICE_ID>.us2.quickconnect.to/webman/index.cgi` loads
- [ ] **2.2** 🔴 **DSM Auto-Update:** Control Panel → System → Update & Restore → Update Settings
  - **Automatically check for updates:** ON
  - **Automatically install important updates:** ON
  - **Update schedule:** Daily, off-peak (e.g. 03:00)
- [ ] **2.3** 🔴 **Database auto-update:** Security Advisor → Settings → Auto-update definitions
- [ ] **2.4** 🔴 **Notifications:** Control Panel → Notification → Enable push/email notifications
  - Enable: System health, WAN status changes, package events
  - Email: configure SMTP or use SRM notification relay
- [ ] **2.5** **SSH:** Control Panel → Terminal & SNMP → Enable SSH on port 2223
- [ ] **2.6** **Regional Options:** Verify language, time format, and date format
- [ ] **2.7** 🔴 **Disable WPS:** Wi-Fi Connect → WPS → **Disabled** on both bands
- [ ] **2.8** 🔴 **Login page title:** Control Panel → System → Login Portal → Title = `<DEVICE_ID>` (e.g. `x5925`)
- [ ] **2.9** 🔴 **Sign in to Synology Account:** Control Panel → QuickConnect → Sign in with Synology Account
  - Required for QuickConnect relay to work. Use `pioneercoach@keylinkit.net` or create if needed.
- [ ] **2.10** 🔴 **Do NOT use DoH:** Network Center → Internet → DNS → Do NOT use DNS over HTTPS
  - Standard DNS only: `1.1.1.2`, `9.9.9.9`, `8.8.8.8`

---

## 2b. Security — Safe Access

- [ ] **2b.1** 🔴 **Install Safe Access:** Package Center → Install "Safe Access" (Synology SRM package)
- [ ] **2b.2** 🔴 **Enable malware blocking on Primary network:**
  - Safe Access → Primary Network → Web Filter → Block malicious websites
  - Enable **Google Safe Browsing**
- [ ] **2b.3** 🔴 **Enable malware blocking on Guest network:**
  - Safe Access → Guest Network → Web Filter → Block malicious websites
  - Enable Google Safe Browsing
- [ ] **2b.4** 🔴 **Google Safe Browsing API key:** Use `AIzaSyBqjLxf1BpD-gPgMqOV1whr6KtomP_D3MM`
- [ ] **2b.5** 🔴 **Enable Threat Intelligence database:** Safe Access → Settings → Threat Intelligence → ON
  - Auto-update definitions
- [ ] **2b.6** 🔴 **DoS Protection:** Network Center → Security → DoS Protection → **Enabled**
- [ ] **2b.7** **Safe Access profiles:** Create a profile for fleet devices with appropriate filtering

---

## 2c. Traffic Control — Bandwidth Management

- [ ] **2c.1** 🔴 **Enable Traffic Control:** Network Center → Traffic Control → Enable
- [ ] **2c.2** 🔴 **Per-device throttle:** Set maximum bandwidth per client to **80% of total WAN capacity**
  - Dynamic: adjust based on measured WAN speed
  - Guest network: stricter cap (50% of total)
- [ ] **2c.3** 🔴 **Run speed test on new WAN detection:**
  - When a WAN interface comes up for the first time, run a speed test to measure capacity
  - Set traffic control caps based on measured throughput
  - Re-test if WAN IP changes (new ISP/tower)
- [ ] **2c.4** **QoS priority:** Primary network gets higher priority than Guest
  - Primary: 70% guaranteed minimum
  - Guest: 30% maximum (burst to 50% when idle)

---

## 3. SmartWAN Configuration

- [ ] **3.1** Network Center → Smart WAN → **Load Balancing with Failover** mode
- [ ] **3.2** 🔴 **WAN1 (eth0):** Primary — this will be Eyeride Eyenet
- [ ] **3.3** 🔴 **WAN2 (eth2):** Secondary — this will be Starlink
- [ ] **3.4** **Health check targets:** `1.1.1.2`, `9.9.9.9`, `8.8.8.8`
  - Cloudflare malware-filtering (1.1.1.2) + Quad9 (9.9.9.9) + Google (8.8.8.8 fallback)
- [ ] **3.5** **Policy routing:** All traffic balanced; no manual pinning yet

---

## 4. Package Center Setup

- [ ] **4.1** Open Package Center → Accept terms of service when prompted
- [ ] **4.2** Package Center → Settings → Trust Level → **"Any publisher"**
- [ ] **4.3** Verify SD card is recognized (Package Center → Storage)

---

## 5. aiwanbal Package Install

- [ ] **5.1** Download `aiwanbal-1.3-0013.spk` from Gitea:
  ```
  https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer/releases/download/v1.3-0013/aiwanbal-1.3-0013.spk
  ```
- [ ] **5.2** SCP to router: `scp -O -P 2223 aiwanbal-*.spk kitadmin@192.168.1.1:/tmp/`
- [ ] **5.3** Install via SSH: `echo "kitPLANE1!!" | sudo -S synopkg install /tmp/aiwanbal-*.spk`
- [ ] **5.4** Start: `echo "kitPLANE1!!" | sudo -S synopkg start aiwanbal`
- [ ] **5.5** 🔴 **Configure for mobile/bus environment:**
  ```sh
  # In /etc/aiwanbal/aiwanbal.conf, set:
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf ENVIRONMENT mobile
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf SPEEDTEST_AUTO_ENABLED 0
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf DEVICE_NAME x4078
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf CHECK_INTERVAL 30
  ```
- [ ] **5.6** 🔴 **Set Eyeride GPS config** (if Eyeride password is available):
  ```sh
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_ENABLE 1
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_IP 192.168.10.1
  echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_PASSWORD <eyeride-password>
  ```
- [ ] **5.7** 🔴 **Verify ISP detection:** Check the aiwanbal UI at `/webman/3rdparty/aiwanbal/index.html`
  - WAN1 should show carrier/ISP (Eyeride → T-Mobile/Verizon depending on SIM)
  - WAN2 should show Starlink
  - If not detecting, run auto-detection: `sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth0; sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth2`

---

## 6. kittunnel Package Install (Reverse SSH Tunnel)

- [ ] **6.1** Assign a unique port for this bus (see fleet port registry)
- [ ] **6.2** Download hardened kittunnel SPK (v1.1+) from Gitea
- [ ] **6.3** SCP to router and install via synopkg
- [ ] **6.4** Verify tunnel.conf has correct `REMOTE_PORT`
- [ ] **6.5** Start: `sudo synopkg start kittunnel`
- [ ] **6.6** Verify tunnel is connected: check port is LISTENING on VPS (162.243.83.36)
- [ ] **6.7** Verify end-to-end: `ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"`

---

## 6a. Kit-Connect (Replaces kittunnel + Tailscale — single SPK)

Kit-connect is the unified replacement for Sections 6 and 7. One SPK installs both the reverse SSH tunnel and Tailscale.

- [ ] **6a.1** Download SPK from Gitea: `https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk`
- [ ] **6a.2** SCP to router: `scp -O -P 2223 kit-connect-0.1-0001.spk kitadmin@192.168.1.1:/tmp/`
- [ ] **6a.3** Install: `echo "kitPLANE1!!" | sudo -S /usr/syno/bin/synopkg install /tmp/kit-connect-0.1-0001.spk`
- [ ] **6a.4** Verify config: `grep TUNNEL_LOCAL_SSH_PORT /etc/kit-connect/connect.conf` must show `2223` (NOT 22)
- [ ] **6a.5** ONLY after confirming two alternative access paths (QuickConnect + Tailscale), stop old tunnel: `sudo synopkg stop kittunnel`
- [ ] **6a.6** Start kit-connect: `sudo synopkg start kit-connect`
- [ ] **6a.7** Verify port is LISTENING on VPS (162.243.83.36) for this device

**⚠️ DO NOT attempt switchover remotely.** The old tunnel is the only management path. Switch only when physically present or when QuickConnect + Tailscale are both confirmed working.

## 7. Tailscale Install (Management Tailnet)

- [ ] **7.1** Download Tailscale static `linux_arm` binary
- [ ] **7.2** Install to `/usr/local/bin/tailscale` and `/usr/local/bin/tailscaled`
- [ ] **7.3** Start tailscaled in userspace mode:
  ```sh
  /usr/bin/setsid /usr/local/bin/tailscaled \
    --statedir=/var/packages/Tailscale/var/state \
    --tun=userspace-networking \
    --socket=/var/packages/Tailscale/var/run/tailscaled.sock &
  ```
- [ ] **7.4** 🔴 Join bus fleet tailnet:
  ```sh
  tailscale up --auth-key tskey-auth-... --hostname <DEVICE_ID> --accept-routes=false --accept-dns=false
  ```
- [ ] **7.5** 🔴 Install rc.local with **correct paths** (`/usr/bin/setsid`, `/bin/mkdir`):
  ```sh
  echo '#!/bin/sh' | sudo tee /usr/syno/etc/rc.d/rc.local
  echo '/bin/mkdir -p /var/packages/Tailscale/var/state /var/packages/Tailscale/var/run' | sudo tee -a /usr/syno/etc/rc.d/rc.local
  echo '/usr/bin/setsid /usr/local/bin/tailscaled --statedir=/var/packages/Tailscale/var/state --tun=userspace-networking --socket=/var/packages/Tailscale/var/run/tailscaled.sock &' | sudo tee -a /usr/syno/etc/rc.d/rc.local
  sudo chmod +x /usr/syno/etc/rc.d/rc.local
  ```
- [ ] **7.6** 🔴 **Reboot test:** `sudo reboot` → wait 2 min → verify tailscaled is running
- [ ] **7.7** Verify Tailscale status shows router online at `pioneercoach@`

---

## 8. Device Identity & Hostname (Persistent)

- [ ] **8.1** 🔴 Create `/etc/busrouter/device-id` with `<DEVICE_ID>`:
  ```sh
  echo "<DEVICE_ID>" | sudo tee /etc/busrouter/device-id
  ```
- [ ] **8.2** 🔴 Create `/etc/busrouter/version` with `0.0`:
  ```sh
  echo "0.0" | sudo tee /etc/busrouter/version
  ```
- [ ] **8.3** 🔴 **Persistent hostname** — set in ALL locations:
  ```sh
  sudo hostname <DEVICE_ID>
  echo "<DEVICE_ID>" | sudo tee /etc/hostname
  ```
  Also check SRM UI: Control Panel → Network → Router Name
- [ ] **8.4** 🔴 **DHCP hostname:** SRM → Network Center → Internet → WAN → edit → Hostname = `<DEVICE_ID>`
  (Without this, DHCP sends `SynologyRouter` to upstream)

---

## 9. Telemetry Agent

- [ ] **9.1** SCP `telemetry-synology.sh` to `/usr/lib/busrouter/telemetry-synology.sh`
- [ ] **9.2** Configure `TELEMETRY_HUB` to fleet hub endpoint:
  ```sh
  # Default: http://167.172.237.162:8080/api/telemetry
  ```
- [ ] **9.3** Add cron job (every 60s):
  ```sh
  echo '* * * * * root TELEMETRY_HUB=http://167.172.237.162:8080/api/telemetry /usr/lib/busrouter/telemetry-synology.sh 2>/dev/null' | sudo tee -a /etc/crontab
  ```
- [ ] **9.4** Dry-run test: `sudo /usr/lib/busrouter/telemetry-synology.sh`

## 9b. Weekly Config Backup

- [ ] **9b.1** SCP `scripts/backup-config.sh` to `/usr/lib/busrouter/backup-config.sh`
- [ ] **9b.2** Add weekly cron (Sunday 03:00):
  ```sh
  echo '0 3 * * 0 root BACKUP_HUB=http://167.172.237.162:8080 /usr/lib/busrouter/backup-config.sh 2>/dev/null' | sudo tee -a /etc/crontab
  ```
- [ ] **9b.3** Dry-run test: `sudo /usr/lib/busrouter/backup-config.sh`
- [ ] **9b.4** Verify backup appears on hub at `/opt/busfleet-hub/backups/<DEVICE_ID>/`
- [ ] **9b.5** 🔴 If a router dies, the most recent backup is the replacement config — never more than 7 days stale

---

## 10. Reboot & Final Verification

- [ ] **10.1** Reboot: `sudo reboot`
- [ ] **10.2** After reboot, verify ALL of these survive:

| Check | Command | Expected |
|-------|---------|----------|
| Hostname | `hostname` | `<DEVICE_ID>` |
| Device ID | `cat /etc/busrouter/device-id` | `<DEVICE_ID>` |
| aiwanbal | `sudo synopkg status aiwanbal` | started |
| kittunnel | `sudo synopkg status kittunnel` | started |
| Tailscale | `ps \| grep tailscaled` | running |
| Tailscale status | `tailscale status` | online, pioneercoach@ |
| WiFi SSID | Check phone WiFi scan | `<DEVICE_ID> Pioneer Admin` |
| Guest WiFi | Check phone WiFi scan | `<DEVICE_ID> Pioneer` |
| QuickConnect | Browser → `https://<DEVICE_ID>.us2.quickconnect.to` | SRM login loads |
| SSH tunnel | From lab: `ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"` | `<DEVICE_ID>` |
| Tailscale path | From lab: `ssh -J root@172.18.0.1 -p 2223 kitadmin@<tailscale-ip> "hostname"` | `<DEVICE_ID>` |
| WAN1 carrier | aiwanbal UI | Shows ISP |
| WAN2 carrier | aiwanbal UI | Shows Starlink |
| Telemetry cron | `grep telemetry /etc/crontab` | present |
| Backup cron | `grep backup /etc/crontab` | present |
| Auto-update | SRM UI → Update & Restore | Auto-check ON |
| Notifications | SRM UI → Notification | Enabled |
| WPS | Wi-Fi Connect → WPS | Disabled both bands |
| Safe Access | Package Center → Safe Access | Installed, malware ON |
| DoS Protection | Network Center → Security | Enabled |
| Traffic Control | Network Center → Traffic Control | Enabled, 80% cap |
| DoH | Network Center → Internet → DNS | OFF |

- [ ] **10.3** 🔴 **Run the reboot test before the bus leaves the yard.** There is no second chance.
- [ ] **10.4** Store credentials in vault under `busrouter/<DEVICE_ID>`
- [ ] **10.5** Record device in fleet registry with IPs, port, keys

---

## Fleet Port Registry

| Device | VPS Port | Tailscale IP | QuickConnect URL | Location | Notes |
|--------|----------|-------------|-------------------|----------|-------|
| x4078 | 2226 | `100.116.71.43` | `https://x4078.us2.quickconnect.to` | Bus (field) | Eyeride + Starlink |
| x4324 | 2225 | `100.91.204.32` | `https://x4324.us2.quickconnect.to` | Bus (field) | kit-connect |
| x4662 | 2229 | TBD | `https://x4662.us2.quickconnect.to` | Bus (field) | MR2200ac mesh satellite |
| x5925 | 2230 | TBD | `https://x5925.us2.quickconnect.to` | Office bench | |
| x9002 | 2231 | `100.82.60.8` | N/A (GL-XE3000) | Office bench | |

---
## Post-Deployment Rules 🔴

1. **Never tear down a working tunnel without two confirmed alternative access paths** (QuickConnect + Tailscale, or QuickConnect + LAN). The reverse SSH tunnel is the primary management channel — once it's gone, you are dead in the water.
2. **TUNNEL_LOCAL_SSH_PORT must be 2223 for Synology.** Synology SRM SSH listens on port 2223, not 22. The kit-connect wizard default was corrected to 2223 on 2026-07-22 after this mistake took down x4078.
3. **Don't restart sshd on the VPS** while tunnels are active. It kills established forwarding sessions and authorized_keys changes don't require a restart.
4. **If quickconnect.to is the only backup**, confirm it's actually working before touching anything. The relay can be slow or locked out by DoS protection.
5. **The old kittunnel is stable.** If it's working, leave it alone until a planned maintenance window with physical access.

## Incident Log — 2026-07-22

**Root cause**: The kit-connect wizard.sh used `command -v` (not available on SRM busybox) and defaulted `TUNNEL_LOCAL_SSH_PORT=22` instead of 2223. Combined with attempting remote switchover from old tunnel to kit-connect without confirmed backup paths, this took x4078 and x6340 offline.

**Resolution for x4662**: Kittunnel restarted via Package Center through QuickConnect. Tunnel stable on port 2229.
**Resolution for x4324**: Kit-connect daemon reconnected after authorized_keys fix on VPS. Tunnel stable on port 2225.
**Pending**: x4078 needs physical access — kit-connect daemon is installed and connecting but fails because it forwards to port 22 instead of 2223.

**SPK fixed**: Rebuilt with `which` instead of `command -v` and `TUNNEL_LOCAL_SSH_PORT=2223`. Available at `https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk`.

---
## x4078 Audit (2026-07-21) — Findings from DSS backup

🔴 **Critical — must fix remotely or on next physical access:**

| Gap | Severity | Fix |
|-----|----------|-----|
| SSID: `x4078 Admin` → should be `x4078 Pioneer Admin` | Medium | SRM Wi-Fi settings |
| Guest SSID: `x4078 Pioneer Guest` → should be `x4078 Pioneer` | Medium | SRM Wi-Fi settings |
| Band SSIDs have `_2.4G`/`_5G` suffixes | Low | Disable per-band SSIDs, use SmartConnect only |
| Old `SynologyGuest_*` networks still present | Low | Delete in SRM Wi-Fi settings |
| Timezone was Central America, not CST/CDT (Chicago) | High | Fixed via QuickConnect SRM UI |
| Database auto-update OFF | High | Fixed via QuickConnect SRM UI |
| Notifications not enabled | High | Fixed via QuickConnect SRM UI |
| DHCP hostname sends `SynologyRouter` | Medium | SRM Network Center → WAN → Hostname |
| Hostname persistence failing on reboot | High | Verify `/etc/hostname` + SRM Network Center |
| ISP/carrier not detecting per WAN | Medium | Run `aiwanbal-detect.sh` on each WAN |
| rc.local has wrong `/bin/setsid` path → `/usr/bin/setsid` | High | Fixed already |
| Tailscale not boot-tested | Critical | Must test on next physical access |