allen/kit-busrouter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
kit-busrouter/docs/deployment/synology-rt2600ac-checklist.md
T
kitadmin 879f97c843 fix: x4078 remote recovery + daemon wait-wedge bug + SPK rebuild 
- docs/incident-log: mark x4078 recovered remotely via Tailscale (no
  physical access needed); document the recovery steps and root cause
- connect-daemon.sh: remove bare `wait` that wedged the retry loop
  forever on the setsid'd tailscaled child — once the tunnel failed the
  daemon could not self-heal until the package was restarted; now the
  main loop falls through start_tunnel (already blocking+retrying) and
  restarts the whole cycle cleanly
- SPK rebuilt (32839680 bytes) with the daemon fix included
- Also documents the two latent lessons: (1) a field router is only
  inaccessible when ALL three paths fail; (2) always check Tailscale +
  QuickConnect before assuming a truck roll is required

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 22:23:53 +00:00

19 KiB
Raw Permalink Blame History

Synology RT2600ac — Fleet Deployment Checklist

Version: 1.2 · Last updated: 2026-07-22 · Audited against: x4078 DSS config backup

Every Synology RT2600ac bus router MUST pass ALL sections before departing the yard. Items flagged 🔴 were missed on x4078 and caused remote-access emergencies.

0. Pre-flight (before touching SRM)

  • 0.1 Insert SD card into router (required for Package Center and third-party packages)
  • 0.2 Power on router; connect LAN port to a PC for setup
  • 0.3 Verify default LAN IP 192.168.1.1 is reachable

1. SRM First-Time Setup Wizard

  • 1.1 🔴 Admin password: kitPLANE1!! (user: kitadmin)
  • 1.2 🔴 Timezone: (GMT-06:00) Central Time (US & Canada) — Chicago, CST/CDT
    • NOT UTC, NOT Central America (Mexico City)
  • 1.3 🔴 WiFi Main SSID: <DEVICE_ID> Pioneer Admin (e.g. x4078 Pioneer Admin)
    • Security: WPA2-PSK · Password: Pioneer321!
    • Smart Connect: ON (single SSID for both bands)
    • Do NOT add _2.4G or _5G suffixes to the SSID
  • 1.4 🔴 WiFi Guest SSID: <DEVICE_ID> Pioneer (e.g. x4078 Pioneer)
    • Security: WPA2-PSK · Password: Pioneer123
    • Client isolation: ON
    • Smart Connect: ON
    • Delete any default SynologyGuest_* networks
  • 1.5 LAN subnet: Keep default 192.168.1.0/24 (lbr0)
  • 1.6 Guest subnet: 192.168.2.0/24 (gbr0, SRM default)
  • 1.7 Hostname: Set to <DEVICE_ID> (e.g. x4078)
  • 1.8 Router login port: Keep default HTTP 8000 / HTTPS 8001

2. SRM Control Panel — System Settings

  • 2.1 🔴 QuickConnect: Enable · ID = <DEVICE_ID> (e.g. x4078)
    • Sign in with Synology Account (create pioneercoach@keylinkit.net if needed)
    • Verify: https://<DEVICE_ID>.us2.quickconnect.to/webman/index.cgi loads
  • 2.2 🔴 DSM Auto-Update: Control Panel → System → Update & Restore → Update Settings
    • Automatically check for updates: ON
    • Automatically install important updates: ON
    • Update schedule: Daily, off-peak (e.g. 03:00)
  • 2.3 🔴 Database auto-update: Security Advisor → Settings → Auto-update definitions
  • 2.4 🔴 Notifications: Control Panel → Notification → Enable push/email notifications
    • Enable: System health, WAN status changes, package events
    • Email: configure SMTP or use SRM notification relay
  • 2.5 SSH: Control Panel → Terminal & SNMP → Enable SSH on port 2223
  • 2.6 Regional Options: Verify language, time format, and date format
  • 2.7 🔴 Disable WPS: Wi-Fi Connect → WPS → Disabled on both bands
  • 2.8 🔴 Login page title: Control Panel → System → Login Portal → Title = <DEVICE_ID> (e.g. x5925)
  • 2.9 🔴 Sign in to Synology Account: Control Panel → QuickConnect → Sign in with Synology Account
    • Required for QuickConnect relay to work. Use pioneercoach@keylinkit.net or create if needed.
  • 2.10 🔴 Do NOT use DoH: Network Center → Internet → DNS → Do NOT use DNS over HTTPS
    • Standard DNS only: 1.1.1.2, 9.9.9.9, 8.8.8.8

2b. Security — Safe Access

  • 2b.1 🔴 Install Safe Access: Package Center → Install "Safe Access" (Synology SRM package)
  • 2b.2 🔴 Enable malware blocking on Primary network:
    • Safe Access → Primary Network → Web Filter → Block malicious websites
    • Enable Google Safe Browsing
  • 2b.3 🔴 Enable malware blocking on Guest network:
    • Safe Access → Guest Network → Web Filter → Block malicious websites
    • Enable Google Safe Browsing
  • 2b.4 🔴 Google Safe Browsing API key: Use AIzaSyBqjLxf1BpD-gPgMqOV1whr6KtomP_D3MM
  • 2b.5 🔴 Enable Threat Intelligence database: Safe Access → Settings → Threat Intelligence → ON
    • Auto-update definitions
  • 2b.6 🔴 DoS Protection: Network Center → Security → DoS Protection → Enabled
  • 2b.7 Safe Access profiles: Create a profile for fleet devices with appropriate filtering

2c. Traffic Control — Bandwidth Management

  • 2c.1 🔴 Enable Traffic Control: Network Center → Traffic Control → Enable
  • 2c.2 🔴 Per-device throttle: Set maximum bandwidth per client to 80% of total WAN capacity
    • Dynamic: adjust based on measured WAN speed
    • Guest network: stricter cap (50% of total)
  • 2c.3 🔴 Run speed test on new WAN detection:
    • When a WAN interface comes up for the first time, run a speed test to measure capacity
    • Set traffic control caps based on measured throughput
    • Re-test if WAN IP changes (new ISP/tower)
  • 2c.4 QoS priority: Primary network gets higher priority than Guest
    • Primary: 70% guaranteed minimum
    • Guest: 30% maximum (burst to 50% when idle)

3. SmartWAN Configuration

  • 3.1 Network Center → Smart WAN → Load Balancing with Failover mode
  • 3.2 🔴 WAN1 (eth0): Primary — this will be Eyeride Eyenet
  • 3.3 🔴 WAN2 (eth2): Secondary — this will be Starlink
  • 3.4 Health check targets: 1.1.1.2, 9.9.9.9, 8.8.8.8
    • Cloudflare malware-filtering (1.1.1.2) + Quad9 (9.9.9.9) + Google (8.8.8.8 fallback)
  • 3.5 Policy routing: All traffic balanced; no manual pinning yet

4. Package Center Setup

  • 4.1 Open Package Center → Accept terms of service when prompted
  • 4.2 Package Center → Settings → Trust Level → "Any publisher"
  • 4.3 Verify SD card is recognized (Package Center → Storage)

5. aiwanbal Package Install

  • 5.1 Download aiwanbal-1.3-0013.spk from Gitea: 
    https://git.keylinkit.net/kitadmin/KIT-Smart-Load-balancer/releases/download/v1.3-0013/aiwanbal-1.3-0013.spk
  • 5.2 SCP to router: scp -O -P 2223 aiwanbal-*.spk kitadmin@192.168.1.1:/tmp/
  • 5.3 Install via SSH: echo "kitPLANE1!!" | sudo -S synopkg install /tmp/aiwanbal-*.spk
  • 5.4 Start: echo "kitPLANE1!!" | sudo -S synopkg start aiwanbal
  • 5.5 🔴 Configure for mobile/bus environment: 
    # In /etc/aiwanbal/aiwanbal.conf, set:
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf ENVIRONMENT mobile
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf SPEEDTEST_AUTO_ENABLED 0
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf DEVICE_NAME x4078
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf CHECK_INTERVAL 30
  • 5.6 🔴 Set Eyeride GPS config (if Eyeride password is available): 
    echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_ENABLE 1
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_IP 192.168.10.1
echo "kitPLANE1!!" | sudo -S synosetkeyvalue /etc/aiwanbal/aiwanbal.conf EYERIDE_PASSWORD <eyeride-password>
  • 5.7 🔴 Verify ISP detection: Check the aiwanbal UI at /webman/3rdparty/aiwanbal/index.html
    • WAN1 should show carrier/ISP (Eyeride → T-Mobile/Verizon depending on SIM)
    • WAN2 should show Starlink
    • If not detecting, run auto-detection: sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth0; sudo /var/packages/aiwanbal/target/bin/aiwanbal-detect.sh eth2

6. kittunnel Package Install (Reverse SSH Tunnel)

  • 6.1 Assign a unique port for this bus (see fleet port registry)
  • 6.2 Download hardened kittunnel SPK (v1.1+) from Gitea
  • 6.3 SCP to router and install via synopkg
  • 6.4 Verify tunnel.conf has correct REMOTE_PORT
  • 6.5 Start: sudo synopkg start kittunnel
  • 6.6 Verify tunnel is connected: check port is LISTENING on VPS (162.243.83.36)
  • 6.7 Verify end-to-end: ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname"

6a. Kit-Connect (Replaces kittunnel + Tailscale — single SPK)

Kit-connect is the unified replacement for Sections 6 and 7. One SPK installs both the reverse SSH tunnel and Tailscale.

  • 6a.1 Download SPK from Gitea: https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk
  • 6a.2 SCP to router: scp -O -P 2223 kit-connect-0.1-0001.spk kitadmin@192.168.1.1:/tmp/
  • 6a.3 Install: echo "kitPLANE1!!" | sudo -S /usr/syno/bin/synopkg install /tmp/kit-connect-0.1-0001.spk
  • 6a.4 Verify config: grep TUNNEL_LOCAL_SSH_PORT /etc/kit-connect/connect.conf must show 2223 (NOT 22)
  • 6a.5 ONLY after confirming two alternative access paths (QuickConnect + Tailscale), stop old tunnel: sudo synopkg stop kittunnel
  • 6a.6 Start kit-connect: sudo synopkg start kit-connect
  • 6a.7 Verify port is LISTENING on VPS (162.243.83.36) for this device

⚠️ DO NOT attempt switchover remotely. The old tunnel is the only management path. Switch only when physically present or when QuickConnect + Tailscale are both confirmed working.

7. Tailscale Install (Management Tailnet)

  • 7.1 Download Tailscale static linux_arm binary
  • 7.2 Install to /usr/local/bin/tailscale and /usr/local/bin/tailscaled
  • 7.3 Start tailscaled in userspace mode: 
    /usr/bin/setsid /usr/local/bin/tailscaled \
  --statedir=/var/packages/Tailscale/var/state \
  --tun=userspace-networking \
  --socket=/var/packages/Tailscale/var/run/tailscaled.sock &
  • 7.4 🔴 Join bus fleet tailnet: 
    tailscale up --auth-key tskey-auth-... --hostname <DEVICE_ID> --accept-routes=false --accept-dns=false
  • 7.5 🔴 Install rc.local with correct paths (/usr/bin/setsid, /bin/mkdir): 
    echo '#!/bin/sh' | sudo tee /usr/syno/etc/rc.d/rc.local
echo '/bin/mkdir -p /var/packages/Tailscale/var/state /var/packages/Tailscale/var/run' | sudo tee -a /usr/syno/etc/rc.d/rc.local
echo '/usr/bin/setsid /usr/local/bin/tailscaled --statedir=/var/packages/Tailscale/var/state --tun=userspace-networking --socket=/var/packages/Tailscale/var/run/tailscaled.sock &' | sudo tee -a /usr/syno/etc/rc.d/rc.local
sudo chmod +x /usr/syno/etc/rc.d/rc.local
  • 7.6 🔴 Reboot test: sudo reboot → wait 2 min → verify tailscaled is running
  • 7.7 Verify Tailscale status shows router online at pioneercoach@

8. Device Identity & Hostname (Persistent)

  • 8.1 🔴 Create /etc/busrouter/device-id with <DEVICE_ID>: 
    echo "<DEVICE_ID>" | sudo tee /etc/busrouter/device-id
  • 8.2 🔴 Create /etc/busrouter/version with 0.0: 
    echo "0.0" | sudo tee /etc/busrouter/version
  • 8.3 🔴 Persistent hostname — set in ALL locations: 
    sudo hostname <DEVICE_ID>
echo "<DEVICE_ID>" | sudo tee /etc/hostname
    Also check SRM UI: Control Panel → Network → Router Name
  • 8.4 🔴 DHCP hostname: SRM → Network Center → Internet → WAN → edit → Hostname = <DEVICE_ID> (Without this, DHCP sends SynologyRouter to upstream)

9. Telemetry Agent

  • 9.1 SCP telemetry-synology.sh to /usr/lib/busrouter/telemetry-synology.sh
  • 9.2 Configure TELEMETRY_HUB to fleet hub endpoint: 
    # Default: http://167.172.237.162:8080/api/telemetry
  • 9.3 Add cron job (every 60s): 
    echo '* * * * * root TELEMETRY_HUB=http://167.172.237.162:8080/api/telemetry /usr/lib/busrouter/telemetry-synology.sh 2>/dev/null' | sudo tee -a /etc/crontab
  • 9.4 Dry-run test: sudo /usr/lib/busrouter/telemetry-synology.sh

9b. Weekly Config Backup

  • 9b.1 SCP scripts/backup-config.sh to /usr/lib/busrouter/backup-config.sh
  • 9b.2 Add weekly cron (Sunday 03:00): 
    echo '0 3 * * 0 root BACKUP_HUB=http://167.172.237.162:8080 /usr/lib/busrouter/backup-config.sh 2>/dev/null' | sudo tee -a /etc/crontab
  • 9b.3 Dry-run test: sudo /usr/lib/busrouter/backup-config.sh
  • 9b.4 Verify backup appears on hub at /opt/busfleet-hub/backups/<DEVICE_ID>/
  • 9b.5 🔴 If a router dies, the most recent backup is the replacement config — never more than 7 days stale

10. Reboot & Final Verification

  • 10.1 Reboot: sudo reboot
  • 10.2 After reboot, verify ALL of these survive:
Check Command Expected
Hostname hostname <DEVICE_ID>
Device ID cat /etc/busrouter/device-id <DEVICE_ID>
aiwanbal sudo synopkg status aiwanbal started
kittunnel sudo synopkg status kittunnel started
Tailscale ps | grep tailscaled running
Tailscale status tailscale status online, pioneercoach@
WiFi SSID Check phone WiFi scan <DEVICE_ID> Pioneer Admin
Guest WiFi Check phone WiFi scan <DEVICE_ID> Pioneer
QuickConnect Browser → https://<DEVICE_ID>.us2.quickconnect.to SRM login loads
SSH tunnel From lab: ssh -J root@172.18.0.1 -p <port> kitadmin@127.0.0.1 "hostname" <DEVICE_ID>
Tailscale path From lab: ssh -J root@172.18.0.1 -p 2223 kitadmin@<tailscale-ip> "hostname" <DEVICE_ID>
WAN1 carrier aiwanbal UI Shows ISP
WAN2 carrier aiwanbal UI Shows Starlink
Telemetry cron grep telemetry /etc/crontab present
Backup cron grep backup /etc/crontab present
Auto-update SRM UI → Update & Restore Auto-check ON
Notifications SRM UI → Notification Enabled
WPS Wi-Fi Connect → WPS Disabled both bands
Safe Access Package Center → Safe Access Installed, malware ON
DoS Protection Network Center → Security Enabled
Traffic Control Network Center → Traffic Control Enabled, 80% cap
DoH Network Center → Internet → DNS OFF
  • 10.3 🔴 Run the reboot test before the bus leaves the yard. There is no second chance.
  • 10.4 Store credentials in vault under busrouter/<DEVICE_ID>
  • 10.5 Record device in fleet registry with IPs, port, keys

Fleet Port Registry

Device VPS Port Tailscale IP QuickConnect URL Location Notes
x4078 2226 100.116.71.43 https://x4078.us2.quickconnect.to Bus (field) Eyeride + Starlink
x4324 2225 100.91.204.32 https://x4324.us2.quickconnect.to Bus (field) kit-connect
x4662 2229 TBD https://x4662.us2.quickconnect.to Bus (field) MR2200ac mesh satellite
x5925 2230 TBD https://x5925.us2.quickconnect.to Office bench
x9002 2231 100.82.60.8 N/A (GL-XE3000) Office bench

Post-Deployment Rules 🔴

  1. Never tear down a working tunnel without two confirmed alternative access paths (QuickConnect + Tailscale, or QuickConnect + LAN). The reverse SSH tunnel is the primary management channel — once it's gone, you are dead in the water.
  2. TUNNEL_LOCAL_SSH_PORT must be 2223 for Synology. Synology SRM SSH listens on port 2223, not 22. The kit-connect wizard default was corrected to 2223 on 2026-07-22 after this mistake took down x4078.
  3. Don't restart sshd on the VPS while tunnels are active. It kills established forwarding sessions and authorized_keys changes don't require a restart.
  4. If quickconnect.to is the only backup, confirm it's actually working before touching anything. The relay can be slow or locked out by DoS protection.
  5. The old kittunnel is stable. If it's working, leave it alone until a planned maintenance window with physical access.

Incident Log — 2026-07-22

Root cause: The kit-connect wizard.sh used command -v (not available on SRM busybox) and defaulted TUNNEL_LOCAL_SSH_PORT=22 instead of 2223. Combined with attempting remote switchover from old tunnel to kit-connect without confirmed backup paths, this took x4078 and x6340 offline.

Resolution for x4662: Kittunnel restarted via Package Center through QuickConnect. Tunnel stable on port 2229. Resolution for x4324: Kit-connect daemon reconnected after authorized_keys fix on VPS. Tunnel stable on port 2225. Resolution for x4078 (2026-07-22, remote — NO physical access needed): Recovered entirely over the still-live Tailscale path (100.116.71.43, SRM SSH :2223) — QuickConnect was also up the whole time. The reverse tunnel was the only dead path. Root cause on-device: TUNNEL_LOCAL_SSH_PORT=22 in /etc/kit-connect/connect.conf (SRM sshd listens on 2223). Fix: SSH in over Tailscale from a tailnet node (e.g. the hub), sed -i 's/^TUNNEL_LOCAL_SSH_PORT=.*/TUNNEL_LOCAL_SSH_PORT=2223/' the conf, then synopkg stop kit-connect; synopkg start kit-connect. Tunnel came back on port 2226; end-to-end verified. Lesson: a field router is only "inaccessible" if ALL of {Tailscale, QuickConnect, reverse tunnel} are down — check each before assuming physical access is required.

Latent daemon bug found: connect-daemon.sh main loop ends each iteration with a bare wait, which blocks forever on the setsid-launched tailscaled child. Once the tunnel started failing, the loop wedged for ~58 min and never self-healed on the corrected config until the package was restarted. Fix the wait (e.g. wait "$SSH_PID" or wait -n) so the retry loop can recover on its own.

SPK fixed: Rebuilt with which instead of command -v and TUNNEL_LOCAL_SSH_PORT=2223. Available at https://git.keylinkit.net/allen/kit-busrouter/raw/branch/main/kit-connect/kit-connect-0.1-0001.spk.

x4078 Audit (2026-07-21) — Findings from DSS backup

🔴 Critical — must fix remotely or on next physical access:

Gap Severity Fix
SSID: x4078 Admin → should be x4078 Pioneer Admin Medium SRM Wi-Fi settings
Guest SSID: x4078 Pioneer Guest → should be x4078 Pioneer Medium SRM Wi-Fi settings
Band SSIDs have _2.4G/_5G suffixes Low Disable per-band SSIDs, use SmartConnect only
Old SynologyGuest_* networks still present Low Delete in SRM Wi-Fi settings
Timezone was Central America, not CST/CDT (Chicago) High Fixed via QuickConnect SRM UI
Database auto-update OFF High Fixed via QuickConnect SRM UI
Notifications not enabled High Fixed via QuickConnect SRM UI
DHCP hostname sends SynologyRouter Medium SRM Network Center → WAN → Hostname
Hostname persistence failing on reboot High Verify /etc/hostname + SRM Network Center
ISP/carrier not detecting per WAN Medium Run aiwanbal-detect.sh on each WAN
rc.local has wrong /bin/setsid path → /usr/bin/setsid High Fixed already
Tailscale not boot-tested Critical Must test on next physical access
Reference in New Issue View Git Blame Copy Permalink