I asked Claude to take a quick look at the security of a virtual machine that has nothing on it. No app, no data, nothing anyone could want. The summary came back: 10,304 failed SSH login attempts in the last 24 hours, from 54 distinct source IPs, the top offender alone good for 1,375 of them. Usernames tried: admin (402), ubuntu (323), user (294). And one line I have been quoting since: you are under continuous automated brute-force right now.
The original version was a thread in Hebrew.
I checked a second machine, on Hetzner instead of Azure. Same picture. Then the SSH log on one of my production servers: about 3,000 attempts a day, 19,000 a week. All of them failed. But failed so far is not the same as secure, so I went through the server methodically. If you run a VM, this is the baseline I now treat as mandatory.
Keys only, and verify it in the log, not the config
No SSH with passwords, only keys. The drop-in in /etc/ssh/sshd_config.d/ should contain:
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-passwordThen check the log, not just the config:
journalctl -u ssh --since "7 days ago" | grep -c "Failed password"What you want to see is zero. Zero Failed password lines in a week means attackers cannot even reach a password prompt; the door is really closed. Thousands of them with password auth supposedly disabled means your config is not doing what you think it is.
No account should have a password at all
sudo awk -F: '$2 !~ /^[!*]/ {print $1}' /etc/shadowThis prints every account that still has a usable password hash. It should return nothing. If there is nothing to guess, brute force has nothing to hit, even if a future bug or a careless config change reopens the door.
The firewall belongs outside the VM
Configure a firewall at the provider level: Hetzner Cloud Firewall, Azure NSG, an AWS security group. Open 22, 80 and 443 and block everything else.
The reason it has to live outside the box: if you use Docker, it writes its own iptables rules and bypasses ufw without asking. The provider's firewall it cannot bypass.
Verify from outside: a port blocked by the cloud firewall should time out, because packets are dropped before they ever reach the VM. If a closed port answers with an immediate refusal, the packet reached the machine, and your provider firewall is not actually in front of it.
One deploy-pipeline footnote while you are in there: GitHub Actions runners still do not speak IPv6, so deploying to an IPv6-only server will not work without NAT64 in the middle.
fail2ban, with escalating bans
Install fail2ban on sshd. The default ban is 10 minutes; raise it to an hour and turn on increments, so a returning attacker gets a day, then a week:
[sshd]
bantime = 1h
bantime.increment = true
bantime.factor = 2
bantime.maxtime = 1wThis cannot lock you out. Only failures count, and your key works on the first attempt.
Read the successful logins, not just the failures
journalctl -u ssh | grep AcceptedGroup the results by key fingerprint and source IP. Every line must have an explanation: your key from your addresses, the CI key from your CI provider's ranges. Anything that does not fit is not a finding for a report, it is a security incident.
Updates do not fully handle themselves
Having unattended-upgrades on is good, but it only installs security updates for the distribution itself. Third-party repos (Docker's, for example) and the reboot after a kernel update are on you. Check:
cat /var/run/reboot-requiredOn my box a new kernel had been installed for weeks and never run.
Secrets on disk: what matters and what does not
A plain .env file at mode 600, owned by root, in a 700 directory, on a key-only server, is fine for a single VM (in my view, at least). Anyone who can read it is already root, and already holds the database and the Docker socket. Encrypting it buys nothing there: the key would have to sit next to the file. Tools like sops and age solve secrets in git, not secrets on a running server.
What does matter with secrets:
- One copy off the server, in a password manager (I use Bitwarden)
- No old
.bakcopies rolling around - No duplicate keys in the file (compose takes the last one, which is fun to debug)
- A rotation you have actually exercised at least once
- Separate keys for staging and production
- Nothing ever printed to logs
Your CI deploy key is a root login stored on GitHub
The deploy key your CI uses is a root login to your server, sitting in GitHub. So:
- Turn on 2FA for the GitHub account
- Pin the server's host key in the workflow instead of trusting it on first use
- Put
restrictin front of the key inauthorized_keys, so a stolen copy gets no shell and no port forwarding
Your laptop is part of the server
FileVault on, screen lock on sleep, an SSH key with a passphrase. If the laptop is taken while it is open, the key is usable no matter what else you did. Only a hardware-bound key (Secure Enclave, or a YubiKey with touch) closes that one.
Never cat the cloud metadata endpoint
Never dump the cloud metadata endpoint into a log or a chat. With some providers it contains the root password from the moment the machine was created.
How to do this without breaking production
- Read-only scan first
- Summarize findings before touching anything
- One change at a time
- Verify after every change
- A revert copy for every file
- And everything documented
The most valuable output is not the hardening itself. It is the document you can hand a client who asks how their data is protected. All of it together, including the write-up: a few hours of work.
This is the unglamorous half of forward deployed engineering: the system is not done when the app runs, it is done when the box it runs on can sit on the public internet taking ten thousand break-in attempts a day, and none of them matter.
FAQ
Is thousands of failed SSH login attempts a day normal?
Yes. Any server with port 22 open to the internet gets scanned constantly. An empty VM with nothing on it took over 10,000 attempts in 24 hours. The attempts themselves are background noise; what matters is that password authentication is really off, no account has a password hash, and every successful login in the log is one you can explain.
Can fail2ban lock me out of my own server?
Not if you connect with an SSH key. Only failed logins count toward a ban, and a valid key succeeds on the first attempt. With ban time increments enabled, repeat attackers escalate from an hour to a day to a week while you never touch the counter.
Should I encrypt the .env file on a single VM?
It buys almost nothing there. Anyone who can read a root-owned file with mode 600 is already root on the box, and the decryption key would have to sit next to the file. Tools like sops and age solve secrets in git, not secrets on a running server. Spend the effort on an off-box copy, a rotation you have actually tested, separate staging and production keys, and keeping secrets out of logs.