Web Server Compromise

Step-by-step process for investigating a compromised Linux web server. Apache and WordPress assumed. The cross-host section covers the watering-hole pattern: lure host plus kit host. Pre-requisite: image mounted read-only per Mount Disk Images.

Placeholders: <host> / <host-a> / <host-b> are the mount-point names you chose at mount time (e.g. web1 if mounted at /mnt/web1). <vg> is the LVM volume group. <attacker-ip> is the IP you find in the access-log sweep. <theme> is the active WordPress theme directory. Fill in before running.

---

One-shot baseline (two-host case)

copy-paste block for the two-host watering-hole scenario (lure host + kit host). set HOST_A and HOST_B at the top to the mount paths you chose (full or relative to your working directory), then run. see the sections below for what each output means. for a single host, set only HOST_A and skip the cross-host comm line.

HOST_A=/path/to/host-a       # e.g. /mnt/host-a or ~/cases/day6/host-a
HOST_B=/path/to/host-b
 
for h in "$HOST_A" "$HOST_B"; do
  echo
  echo "================================================================"
  echo " $h"
  echo "================================================================"
 
  echo
  echo "--- identity ---"
  cat $h/etc/os-release | head -3
  cat $h/etc/hostname; readlink $h/etc/localtime
 
  echo
  echo "--- accounts ---"
  sudo diff $h/etc/passwd $h/etc/passwd-
  sudo diff $h/etc/shadow $h/etc/shadow-
 
  echo
  echo "--- access-log top IPs and UAs ---"
  awk '{print $1}' $h/var/log/apache2/access.log | sort | uniq -c | sort -rn | head
  awk -F'"' '{print $6}' $h/var/log/apache2/access.log | sort | uniq -c | sort -rn | head
 
  echo
  echo "--- persistence quick-sweep ---"
  sudo find $h/home $h/root -name authorized_keys -ls 2>/dev/null
  sudo ls -la $h/etc/cron.d/ $h/var/spool/cron/ 2>/dev/null
  sudo cat $h/root/.bash_history
done
 
echo
echo "================================================================"
echo " cross-host shared IPs"
echo "================================================================"
comm -12 <(awk '{print $1}' "$HOST_A"/var/log/apache2/access.log | sort -u) \
         <(awk '{print $1}' "$HOST_B"/var/log/apache2/access.log | sort -u)

---

Mount and baseline

mount each image read-only per Mount Disk Images. then collect host identity and account state on each host:

for h in <host-a> <host-b>; do
  echo "== $h =="
  cat /mnt/$h/etc/os-release; cat /mnt/$h/etc/hostname; readlink /mnt/$h/etc/localtime
  sudo diff /mnt/$h/etc/passwd /mnt/$h/etc/passwd-
  sudo diff /mnt/$h/etc/shadow /mnt/$h/etc/shadow-
done

empty diffs = no account changes since the last useradd / usermod. distro family decides every later path lookup (Web Log Triage for Apache vs Nginx, Log Triage for Debian vs RHEL).

---

Access-log broadest sweep

for h in <host-a> <host-b>; do
  echo "== $h IPs =="
  awk '{print $1}' /mnt/$h/var/log/apache2/access.log | sort | uniq -c | sort -rn | head
  echo "== $h UAs =="
  awk -F'"' '{print $6}' /mnt/$h/var/log/apache2/access.log | sort | uniq -c | sort -rn | head
done

shared external IP across two hosts = strongest lead. scanner UAs (WPScan, Nikto, DirBuster, sqlmap, Hydra, gobuster) name the tool, not the operator. drop Apache internal-dummy lines with grep -v 'internal dummy' before per-IP counts. field reference and per-status-code one-liners in Web Log Triage.

---

Initial access signals

web-app credential brute force

sudo awk '$7 ~ /login\.php$/ {print $1, $9}' /mnt/<host>/var/log/apache2/access.log | sort | uniq -c

burst of 200s (login form re-rendered on wrong password) then a 302 (success redirect) from the same IP. ATT&CK T1110.001 + T1078.

plugin / app exploit dropping a web shell

sudo grep -E "POST /.*plugin.*upload" /mnt/<host>/var/log/apache2/access.log
sudo grep -iE "c99|r57|b374k|wso|gel4y|simple-backdoor|cmd\.php|shell\.php" /mnt/<host>/var/log/apache2/access.log

POST to a vulnerable URI from curl / python-requests / Wget, then a GET to the dropped shell. ATT&CK T1190 + T1505.003.

confirm web-only: grep -E "Accepted|Failed" /mnt/<host>/var/log/auth.log. no Accepted from the attacker IP = web path only.

---

Web shell on disk

sudo find /mnt/<host>/var/www -type f \( -name '*.php' -o -name '*.jsp' -o -name '*.aspx' \) \
  | xargs sudo stat -c '%y %n' | sort -r | head -20

mtime at or before the first access-log hit on the URI = shell placed before it was used. hash with MD5 + SHA1.

---

Post-exploitation

sudo find /mnt/<host>/var/www -type d -newer /tmp/.before-incident
sudo find /mnt/<host>/var/www -name '*.zip' -o -name '*.tar.gz'

unfamiliar directories under wp-content/uploads/ or wp-content/themes/<theme>/, archives in the kit’s files/ directory, redirects or iframes injected into wp-config.php / theme / plugin / wp_posts / wp_options.

if one host carries a kit and the other does not, the other is the lure. ATT&CK T1608.004 on the lure, T1608.001 on the kit.

---

Persistence and privesc

run Linux Persistence Hunt on each host for the full sweep. quick multi-host pass over the major families:

for h in <host-a> <host-b>; do
  echo "== $h =="
  sudo diff /mnt/$h/etc/passwd /mnt/$h/etc/passwd-                          # account creation
  sudo find /mnt/$h/home /mnt/$h/root -name authorized_keys -ls 2>/dev/null # SSH keys
  sudo ls -la /mnt/$h/etc/cron.d/ /mnt/$h/var/spool/cron/ 2>/dev/null       # cron
  sudo cat /mnt/$h/root/.bash_history                                       # history clearing tell
done

attacker often stays at the web-server account (www-data / apache) or the app’s admin layer. record the web shell as a persistence positive (T1505.003) and list the system-level negatives (cron, systemd, account creation, SSH keys, sudoers, setuid, log clearing) separately.

---

Cross-host correlation

comm -12 <(awk '{print $1}' /mnt/<host-a>/var/log/apache2/access.log | sort -u) \
         <(awk '{print $1}' /mnt/<host-b>/var/log/apache2/access.log | sort -u)

build a per-host comparison: attacker IP, scanner UA, IA technique, role (lure / kit / both / neither).

not lateral movement

shared external IP across two public-facing hosts is not lateral movement. confirm with the negative:

sudo grep -E "Accepted|Failed" /mnt/<host-a>/var/log/auth.log /mnt/<host-b>/var/log/auth.log
sudo find /mnt/<host-a>/home /mnt/<host-a>/root /mnt/<host-b>/home /mnt/<host-b>/root -name authorized_keys -ls
sudo cat /mnt/<host-a>/etc/fstab /mnt/<host-b>/etc/fstab
sudo diff <(awk -F: '$1 ~ /^root$/ {print $2}' /mnt/<host-a>/etc/shadow) \
          <(awk -F: '$1 ~ /^root$/ {print $2}' /mnt/<host-b>/etc/shadow)

no host-to-host SSH + no shared keys + no shared root hash + no cross-mount = no lateral movement. ATT&CK T1021, T1570, T1550 all negative.

watering-hole pattern is one campaign run from outside: lure host stages drive-by content, kit host stages payload, victim hits T1189.

---

Timeline

for h in <host-a> <host-b>; do
  echo "== $h =="
  ls -la /mnt/$h/var/log/apache2/                              # rotation files (.gz)
  cat /mnt/$h/etc/timezone; readlink /mnt/$h/etc/localtime     # confirm server zone
done
sudo zgrep <attacker-ip> /mnt/<host>/var/log/apache2/access.log.*.gz    # historic rotations

per Timestamp Forensics for UTC conversion. UTC = local minus offset. Apache offsets are the server’s zone, not UTC and not the analyst’s zone. log-rotation gaps limit how far back the timeline reaches. sweep .gz archives to fill them.

---

IOC table

one row per artefact, no hash deduplication. defang IPs, URLs, domains.

Type	Value	Location
Source IP	attacker IP defanged	both access.log
User-Agent	scanner UA	both access.log
URI	dropped web-shell path	access.log + filesystem
File MD5 + SHA1	shell file, dropped payload	filesystem path
Path	exploit-kit directory	filesystem
CVE	known plugin / app vulnerability	plugin source
Domain	lure-host hostname defanged	access.log Host + Referer
Credential	cracked password, DB password from wp-config.php	source artefact

---

Pitfalls

• 200 to a ?cmd= URI is not confirmed RCE. compare response size against the target app’s default page
• shared attacker IP across two public-facing hosts is not lateral movement until you check for and find host-to-host pivot evidence
• a web shell at www-data is still persistence. list system-level negatives separately. don’t say “no persistence” because the attacker stayed at www-data
• the active access log misses rotated .gz archives. use zgrep
• access-log offsets are the server’s zone, not UTC

---

links:

Field Manual | Mount Disk Images | Web Log Triage | Linux Persistence Hunt | Filesystem Metadata | Log Triage | Timestamp Forensics | Reverse Shell Triage