SSH is often the first administrative boundary around a GPU server, an inference VM or a self-hosted model. If that boundary is weak, an attacker may gain model files, application data, provider credentials and the ability to run expensive workloads.

This guide explains the protocol through the decisions AI operators actually make. For host configuration and model storage, start with Linux for AI developers.

What happens during a connection

When you run ssh user@host, the client and server:

1. establish a TCP connection;
2. negotiate protocol and cryptographic algorithms;
3. perform a key exchange to derive session keys;
4. verify the server’s host key;
5. authenticate the user or automation identity;
6. open an encrypted channel for a shell, command or tunnel.

Encryption protects the session, but server identity is equally important. Ignoring a changed host-key warning can turn encryption into a secure connection to the wrong machine.

Host verification

On first connection, verify the fingerprint through a trusted cloud console or provisioning record before accepting it. In automation, provision known host keys rather than disabling verification.

ssh-keygen -F gpu.example.com
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

A host key can legitimately change after rebuild or rotation. Investigate why before deleting the old entry. The dedicated host-key troubleshooting guide covers recovery.

Use separate identities

Create modern keys with a passphrase for humans:

ssh-keygen -t ed25519 -a 64 -C "operator@example.com"

Use separate keys or short-lived certificates for CI and deployment automation. A shared team key prevents attribution and makes revocation disruptive.

On the server:

• disable password authentication after key access is proven;
• disable direct root login;
• use personal accounts and sudo for administration;
• limit automation accounts to required commands and paths;
• remove keys immediately when access changes.

SSH controls entry to the host. Application and agent permissions still need their own identities; see AI security.

Bastion hosts and private GPU nodes

GPU workers and model servers generally should not expose SSH to the public internet. Put them on a private network and connect through a controlled bastion:

Host ai-bastion
  HostName bastion.example.com
  User operator
  IdentityFile ~/.ssh/operator_ed25519

Host gpu-worker-1
  HostName 10.0.2.14
  User operator
  ProxyJump ai-bastion

The bastion is a high-value control point. Restrict inbound networks, require strong identity, log access and keep it separate from model-serving workloads.

Port forwarding without accidental exposure

Local forwarding can provide temporary access to an internal dashboard or model endpoint:

ssh -N -L 127.0.0.1:11434:127.0.0.1:11434 operator@gpu.example.com

Binding locally to 127.0.0.1 avoids exposing the forwarded service to the surrounding network. A tunnel is useful for administration, not a substitute for production authentication, TLS and rate limiting.

Automation and deployments

CI should use a narrowly scoped deployment identity. Better still, use an agent or deployment service on the private network rather than opening SSH broadly.

If SSH is required:

• pin host keys;
• use a dedicated key or certificate;
• restrict source network and server account;
• avoid sending secrets in command-line arguments;
• deploy immutable releases and switch a symlink or service reference;
• log the commit and deployment identity.

Connect this process to Git workflows for coding agents and the AI operations hub.

Operational checklist

• Public root and password login disabled.
• Human and automation identities separated.
• Host fingerprints verified and managed.
• Private nodes reachable only through approved paths.
• Keys and certificates rotate without rebuilding applications.
• Failed logins and privileged commands monitored.
• Model directories and environment files inaccessible to ordinary users.
• Recovery access tested before an incident.

SSH is safe AI infrastructure when it provides authenticated, attributable and revocable administration—not merely an encrypted terminal.