container compromised? your first 5 commands

container compromised? your first 5 commands

somebody pages you at 2am because a container is doing something weird. cpu spiking, outbound connections it has no business making, a process name that looks like it belongs in a horror movie. your gut says restart it and move on. your gut is wrong. restarting a compromised container is like hosing down a crime scene because the carpet looked messy. you just destroyed the evidence that would've told you how they got in and whether they're still in your other containers too.

here's the order of operations that actually preserves your ability to investigate, and keeps the damage from spreading while you figure it out.

1. docker top, see what's actually running

before you touch anything else, find out what processes are alive inside the container right now.

docker top <container_id>

this shows you pids, users, and commands currently running inside that container's namespace. you're looking for anything that shouldn't be there: a shell process spawned from your web server, a crypto miner binary, a reverse shell connection, something running as root that your app never needed root for. this is your "what is happening right this second" snapshot, and it disappears the moment the container stops, so grab it first.

2. docker diff, see what changed

docker diff <container_id>

every container starts from an image, which is basically a frozen, known-good filesystem. docker diff compares the current state of the container against that original image and lists every file that's been added, changed, or deleted since it started. an "a" means added, "c" means changed, "d" means deleted. attackers almost always drop something: a new binary in /tmp, a modified cron file, a backdoored config, an ssh key that wasn't yours. this command turns "something feels off" into an actual list of exactly what moved.

3. docker logs, figure out how they got in

docker logs <container_id>

this pulls stdout and stderr from the container, which usually means your application logs. this is where you hunt for the entry point. look for weird request patterns, repeated auth failures right before something succeeded, suspicious user agents, or requests hitting an endpoint that got exploited. if your app logs are thin, this is also your reminder for next time: ship logs somewhere outside the container, because if the container gets wiped, your evidence shouldn't go with it.

4. docker commit, snapshot the evidence

docker commit <container_id> evidence/compromised-container:$(date +%s)

this is the step people skip, and it's the one that matters most for anyone doing real forensics later. docker commit takes the container's current filesystem, exactly as the attacker left it, and saves it as a new image. now you have a frozen copy you can pull apart later: run it in an isolated sandbox, extract files, hand it to whoever does deeper incident response. without this step, every minute the container stays running is a minute closer to that evidence getting overwritten or cleaned up, intentionally or not.

5. docker network disconnect, cut it off

docker network disconnect <network_name> <container_id>

once you've got your snapshot, kill its ability to talk to anything. this severs the container from its network without killing the process, so it's still alive for you to poke at if needed, but it can't exfiltrate more data, call home, or pivot to other containers on the same network. this is your containment move. only after this should you actually stop it, and only after you've pulled whatever logs and artifacts you still need.

the takeaway

the instinct to restart and move on is understandable, nobody wants downtime, but it throws away the only chance you get to learn how the attacker got in. inspect first, diff what changed, read the logs for the entry point, commit the evidence, then isolate before you rebuild from a known-good image. do this in order every time and you turn a panic moment into an actual investigation. save this list somewhere your on-call team will actually find it at 2am, because that's exactly when you won't remember the order yourself.