RustDesk Remote Connection Failure — Disabling UDP Investigation and Fix

A Mac mini runs RustDesk as a remote desktop server with a self-hosted ID/relay server. When connecting remotely from outside, the connection failed or timed out, while the RustDesk process was running locally. Since RustDesk was the only remote access method (no SSH, no other remote desktop), this was equivalent to a complete loss of remote access — a high-priority incident.

Investigation

1. Confirm RustDesk Process Status

pgrep -x RustDesk
# Outputs PID — RustDesk is running

Process is healthy, ruling out crashes or startup failures.

2. Locate Configuration Files

RustDesk config directory on macOS:

~/Library/Preferences/com.carriez.RustDesk/

Key files:

File	Purpose
RustDesk.toml	Identity (ID, keypair)
RustDesk2.toml	Network config (servers, options)

3. Check Network Configuration

cat ~/Library/Preferences/com.carriez.RustDesk/RustDesk2.toml

Found in the [options] section:

disable-udp = 'Y'

This was the root cause. disable-udp = 'Y' forces RustDesk to use TCP-only for P2P and relay communication. In certain NAT environments, TCP-only connections fail to penetrate or relay, causing remote connection timeouts.

Fix Steps

Step 1: Backup Configuration

cp ~/Library/Preferences/com.carriez.RustDesk/RustDesk2.toml \
   ~/Library/Preferences/com.carriez.RustDesk/RustDesk2.toml.bak.$(date +%Y%m%d_%H%M%S)

Step 2: Modify disable-udp

sed -i '' "s/disable-udp = 'Y'/disable-udp = 'N'/" \
  ~/Library/Preferences/com.carriez.RustDesk/RustDesk2.toml

Verify:

grep "disable-udp" ~/Library/Preferences/com.carriez.RustDesk/RustDesk2.toml
# Output: disable-udp = 'N'

Step 3: Restart RustDesk

RustDesk does not support hot-reloading configuration — a restart is required.

osascript -e 'quit app "RustDesk"'
sleep 2
open -a "RustDesk"

Verify:

pgrep -x RustDesk && echo "RustDesk restarted successfully"

Step 4: Remote Verification

Re-initiate connection from the client to confirm the session establishes normally.

Key Configuration Reference

RustDesk2.toml complete [options] section:

[options]
rendezvous_server = 'YOUR_SERVER_IP:21116'
relay-server = 'YOUR_SERVER_IP'
custom-rendezvous-server = 'YOUR_SERVER_IP'
direct-server = 'Y'
verification-method = 'use-permanent-password'
disable-udp = 'N'
allow-remote-config-modification = 'Y'
local-ip-addr = '192.168.x.x'

Field	Meaning
rendezvous_server	Self-hosted ID server address
relay-server	Relay server address
direct-server	Allow LAN direct connection
disable-udp	Disable UDP (N = enabled)
verification-method	Auth method: permanent password

Lessons Learned

1. RustDesk config changes require a restart — no hot-reload support
2. disable-udp = 'Y' is a hidden killer — the GUI toggle is easy to overlook, but in relay/NAT scenarios it causes complete connection failure
3. Remote config modification is high-risk — restart causes brief disconnection; ensure no critical sessions are active and have a backup access method
4. Always backup config files — TOML format is sensitive to quotes and whitespace; accidental edits can prevent RustDesk from starting
5. Periodically audit config for critical remote access tools to catch unintended changes