A few weeks ago, AbuseIPDB started sending me rate-limit warnings. My first thought was: "someone is scanning my site again." I had seen WordPress scanners and random probes before, so I ignored it.

I should not have done that. On a Friday I was talking to a colleague and I mentioned that I have a personal website and that my design talent is highly lacking. They wanted to see my website so I sent them the link (https://carlsson.tech) and also looked at it myself. I noticed that the page was down, this have happened before when SQL server silently filled up all my disk space, so I was just thinking "ok, time to clear up space again, I really should automate this." I log in and start checking my disks, low and behold - out of disk space, but this was just the beginning of it all.

Setting the Stage

To start off we need to go through some of the specification of this website. I am hosting it in a Virtual Private Server (VPS) running Ubuntu 22.04 as operating system (OS). The back-end is a C# .NET Core application with an Angular SSR to SPA and using an SQL Server database. I am also loggin requests that are "odd" and look-up IP addresses that doesn't match a normal user behaviour and temporarily block IP addresses that show automated scanning behaviour (as in, they match up in AbuseIPDB and do requests that no normal human being would ever do and request URLs that doesn't exist such as wp-admin or similar, or trying to access system files directly like .htaccess, etc.). I have also recently started to cache more than before to improve performance.

This result in that I need to periodically need to manage SQL Server transaction log growth, as my current setup doesn't have a full backup/truncation strategy., and there are surprisingly many probes for vulnerable attack surfaces so those logs can baloon. And also with the new caching I suspected something might have gone wrong there as well so I probably should review that as well. So my mind was of course honing in on "out of disk space".

The symptom

It started simply: the webpage wouldn't load any content found in the database. My instinct was that I ran out of disk space, and it was the case:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       116G  116G     0 100% /

Disk. Completely full. Zero bytes available.

Easy fix, just clear out that nasty SQL Server transaction logs, glance over IP logs and see if that needed trimming as well. However, I did find it odd that the SQL Server process had failed on a new kind of error, and no matter what I did I just couldn't start it up again.

So I decided to keep on digging. I ran the command du -h --max-depth=1 /var to get more information what actually had happened. However, I had to switch TMPDIR to /dev/shm so sort would actually have somewhere to write its temp files. The glorious irony of a full disk breaking the tool you're using to diagnose the full disk was not lost on me) turned up three big offenders:

• /var/log/syslog — 40G
• /var/log/syslog.1 — 33G
• A custom logging directory I'd written myself, /var/log/carlssontech/ip-metadata — 28G

This stuck out to me, why on earth was ip-metadata that big? I did expect the syslog to be big. So I kept digging.

The ip-metadata turned out to be a real long-term issue: the IP lookup deamon I'd built for logging IP metadata daily, with - I'll admit it - no retention or rotation policy whatsoever. Looking at the file timestamps, it had been quietly writing about 1.1GB per day for roughly a month(!). I started having the scent of something out of the ordinary in the water, and because nothing was watching disk usage and nothing was cleaning up after it I did not make a proper inspection - as I should have done at the first AbuseIPDB request limit warning.

I also found that the syslog bloat had happened a bit too quick. And traced back to something much more dangerous:

A brute-force attack against my SQL Server's sa account.

They were hammering login attempts multiple times per second, all logged as expected by the system. This all aligned with a single day near the end of the timeline — the straw that broke an already-nearly-full disk.

How did they even find my database?

This was the part that actually surprised me. I wasn't running SQL Server on the default port 1433 — I'd changed it specifically to cut down on opportunistic scanning. Yeah - I know, that is Cyber Security 101 but I am not a Security specialist. And it cost me, there was a sustained, targeted attack against sa on my custom port. Security by obscurity had once again proven to be about as secure as expected. Changing the port is "security through obscurity," and it doesn't hold up against anyone doing an more serious port sweep. A custom port stops lazy scanners that only check 1433 by default. It does nothing against a botnet or scanner that just probes every open port and fingerprints whatever responds — SQL Server's connection handshake is a signature all its own, port number notwithstanding.

The real problem was that my database port was open to the entire internet in the first place, custom port or not.

Was I compromised?

Before fixing anything, I needed to know whether this attack had actually succeeded. This is the single most important step if you ever find yourself here - recovery isn't just "get the service running again," it's "confirm nothing got in" before you move on.

I did two checks that gave me confidence:

1. Every single failed login attempt in the SQL Server error log showed the same error — 18456, Severity 14, State 8 — which specifically means password mismatch. Not a different kind of failure, not a successful auth with a weird follow-up. Just, over and over, wrong password.
2. A quick check of sys.server_principals and sys.databases showed no unexpected logins, no unfamiliar databases, nothing planted.

That combination of consistent password-mismatch errors and no new logins/databases was enough to conclude the password held and nothing got through.

Worth noting: this was luck in the sense that the password was strong enough, not because anything was actively blocking the attempts. That's an important distinction to make It's why the actual fix isn't "hope your password is good". It's restricting from where someone can even attempt to log in.

The SQL Server that wouldn't restart

Freeing the disk space (clearing old logs, deleting the historical ip-metadata files) should have been the end of it. It wasn't. SQL Server still refused to start, crash-looping in under a second every time, with systemd eventually giving up entirely:

Start request repeated too quickly.

Digging into journalctl (since the SQL Server error log itself hadn't been touched in over a week, a sign the crash was happening before SQL Server even got far enough to log anything) uncovered the real error:

PalInstanceId: Unable to read instance id from file.
sqlservr: PAL initialization failed. Error: 101

SQL Server on Linux keeps an internal instance identifier at /var/opt/mssql/.system/instance_id. Mine was sitting at 0 bytes that is almost certainly corrupted mid-write when the disk hit 100% at the worst possible moment.

Permissions looked fine. No AppArmor denial. No immutable file attribute. The fix, in the end, was almost disappointing: stop the service, delete the empty instance_id file outright, and let SQL Server regenerate it fresh on the next start. It came up clean and has been stable since.

The fixes

Once SQL Server was back up, I made a few changes so this couldn't happen the same way twice:

1. Locked the database port down to known IPs only

Since I have a static IP, this was a simple ufw rule:

sudo ufw allow from <my-static-ip> to any port <sql-port> proto tcp
sudo ufw delete <rule-number-for-the-old-anywhere-rule>

Now the SQL port only accepts connections from my own IP - everyone else gets silently dropped before they ever reach SQL Server's login prompt.

2. Actually enforced key-only SSH

I thought I had password authentication disabled for SSH. I did not. sshd -T — which shows the effective config after all overrides are applied, rather than trusting a single config file - revealed a conflicting drop-in file from cloud-init was silently re-enabling password auth. Once I found and fixed the actual authoritative file, and confirmed it with a fresh connection attempt (forcing key auth off to make sure it was actually refused, not just untested), SSH became key-only for real. Make sure to find all the files that configurate things and change it in the correct place(s).

3. Set up disk space alerting

An hourly cron job now checks disk usage and emails me if it crosses 75%. Cheap insurance against ever finding out about a full disk via "why won't my website load" again.

Future work

After battling this for some time, I grew tired of doing fixes and similar, so I decided against implementing log rotation. This is not crucial and not an attack surface, so it could be done later. The combination of all the solutions make it possible to manage this easier via email alerts when I am running low of space.

If you don't have a static IP

Locking the firewall to a single IP is the simplest fix, but it doesn't work if your IP changes - IP rotation from their ISP, from switching networks, from working remotely. Here are the alternatives, roughly in order of how much effort they take to set up:

VPN into a private network

Instead of exposing the database port to the internet at all, put it on a private network/VPC and connect to it yourself via a VPN (WireGuard is a great lightweight option, and most cloud providers also offer a managed VPN gateway). Your database never has a public-facing port. You tunnel in first, then connect locally. This is the more "correct" solution long-term - public-facing database ports are a liability even with IP restrictions, since your IP is (by definition) something the internet can see too.

Bastion host / jump box

Run a small, hardened VM whose only job is accepting SSH connections, then tunnel your database connection through it (ssh -L local port forwarding). Your actual database server never accepts direct external connections at all — only the bastion does, and the bastion can itself be locked down harder than a general-purpose server (fewer open ports, more aggressive monitoring, disposable/rebuildable).

Cloud provider's private networking

If your web app and database live in the same cloud provider's network, you likely don't need the database port exposed to the public internet at all, only to other resources inside your VPC/VNet. This is the cleanest option if your app and database are both cloud-hosted: no tunnel, no VPN client, just network-level isolation. I actually didn't need any of the public-IP-restriction dance for this reason — my web app and database live on the same box — but if yours are split across servers, this is worth setting up before anything else.

Dynamic DNS + firewall automation

If you're stuck without a static IP but still want IP-based restriction rather than a VPN, some routers/services support dynamic DNS (a hostname that updates automatically when your IP changes), combined with a small script that periodically resolves your hostname and updates the firewall rule to match. More moving parts, more to maintain, but workable if VPN setup feels like overkill for your situation.

The actual lesson

None of this happened because of one big mistake — it was a chain of small ones, each fine in isolation:

• A logging directory with no rotation and retention policy (fine, until it isn't)
• A database port open to the world, "protected" by a non-default port number (fine, until someone scans properly)
• An SSH config I assumed was locked down, undone by a drop-in file I didn't know existed (fine, until you actually check)
• No disk alerting, so all of the above had weeks to compound silently before anything visibly broke

Any one of these alone probably wouldn't have taken the site down. Together, they did. If there's one habit I'd recommend out of all of this, it's the one that actually caught the SSH issue: don't trust that a setting is what you think it is - check the effective configuration directly, and verify changes with a fresh connection/test rather than assuming the file you edited is the one that matters.

So in essence:

In the end, I wasn't saved by good security, but by luck. My password was strong enough that the brute-force attack never succeeded, and before it ever could, the relentless stream of failed login attempts filled my disk with logs and crashed the very system the attacker was trying to get access to.