linux log triage, the logs told you first

the logs told you first, you just weren't reading

every linux box on the internet gets hammered with login attempts constantly. that's not paranoia, that's just tuesday. the difference between a normal tuesday and a bad one is usually sitting right there in your auth logs, waiting for you to look. most people don't look until something's already on fire. this is the walkthrough for looking before the fire, or at least figuring out how it started after.

step one, count the failures

your auth log (usually /var/log/auth.log on debian/ubuntu or /var/log/secure on rhel/centos) is a running diary of every login attempt on the box. the pattern you're hunting for is simple: a wall of failed password attempts from the same ip, followed by one clean success. that's not a coincidence, that's a brute-force that finally landed.

grep "Failed password" /var/log/auth.log | wc -l

that gives you a raw count. if it's in the thousands, someone's been knocking on your door for a while. now find out if they got in:

grep "Accepted password" /var/log/auth.log

if you see an accepted login right after a huge run of failures, especially at 3am, especially for an account that doesn't normally log in, treat that as a confirmed compromise until proven otherwise.

step two, pull the source ips

every login line has an ip attached to it. pull those out and you know exactly where the traffic came from:

grep "Accepted password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c

this gives you a count of successful logins per ip. one ip you don't recognize with a bunch of successful logins is your starting thread. pull it and start unraveling.

step three, check for new users and sudo abuse

attackers who get a foothold usually want persistence, and the laziest, most common way to get that is creating a new user or escalating an existing one. check who's actually supposed to be on your box:

cat /etc/passwd | grep -E "/bin/bash|/bin/sh"

if there's an account in there you don't remember creating, that's your answer. next, check sudo usage, because a normal low-priv account suddenly running sudo commands is a huge red flag:

grep "sudo" /var/log/auth.log

look for sudo commands you didn't run, at times you weren't working, from users who shouldn't have sudo rights in the first place. also worth checking /etc/sudoers and /etc/sudoers.d/ for any new lines granting privileges nobody approved.

step four, trace the ip across everything

once you've got a suspicious ip, don't stop at auth.log. that ip left footprints everywhere it touched. grep it across your whole log directory to build a timeline:

grep -r "203.0.113.45" /var/log/

this pulls every mention across auth logs, web server logs, mail logs, whatever's sitting in /var/log. now you can actually reconstruct the story: when they got in, what they touched, whether they pivoted to another service, whether they tried to cover their tracks. the logs are basically a crime scene, you just have to walk it in order instead of only looking at the last room.

the takeaway

the point of all this isn't to turn you into a full time incident responder, it's to get you into the habit of actually reading what your system is already telling you. most breaches aren't quiet, they're loud, they're just loud in a log file nobody checked. set a recurring reminder to spot check auth.log on anything internet-facing, even just once a week. better yet, ship your logs somewhere central (a simple syslog server or a lightweight tool like logwatch or fail2ban alerting) so patterns jump out instead of getting buried. lock down ssh with key-based auth and disable password logins entirely if you can, that alone kills most of the brute-force wall-of-failures problem before it starts. the story of what happened to your server is already written, the only question is whether you read it before or after it matters.