DFIR-Field-Manual

Linux Persistence Hunt

6 min read

Linux persistence and privilege-escalation triage against a mounted forensic image. Six persistence families, four privesc paths, ATT&CK techniques inline. Image must be mounted read-only first - use Mount Disk Images workflow. Examples assume mount point /mnt/img.

Account-based persistence

[T1078]    Valid Accounts
[T1136]    Create Account

diff /mnt/img/etc/passwd /mnt/img/etc/passwd-      # detect new accounts via daemon backup diff
diff /mnt/img/etc/group  /mnt/img/etc/group-       # detect new groups
diff /mnt/img/etc/shadow /mnt/img/etc/shadow-      # detect new shadow rows
stat /mnt/img/etc/{passwd,group,shadow}            # placement timestamp - useradd writes all 3 atomically (<30ms)

backup files (passwd-, group-, shadow-) are managed by useradd/usermod - they hold the previous version. diff against them surfaces every account change in one pass.

shadow password field decoder:

Field 2 valueMeans
!! or !locked, no password ever set - cannot password-auth
*locked by admin
$6$<salt>$<hash>SHA-512 hash, can password-auth
$2y$<rounds>$<hash>bcrypt, can password-auth
awk -F: '$2 ~ /^\$/ {print $1}' /mnt/img/etc/shadow      # accounts with real passwords

SSH authorized_keys persistence

[T1098.004]  Account Manipulation: SSH Authorized Keys

find /mnt/img/home /mnt/img/root -name authorized_keys -ls
cat /mnt/img/home/*/.ssh/authorized_keys
cat /mnt/img/root/.ssh/authorized_keys

one find pass closes the family host-wide. Multiple keys in one file = multiple operators or an attacker keeping a backup channel.

Cron-based persistence

[T1053.003]  Scheduled Task/Job: Cron

cat /mnt/img/etc/crontab
ls /mnt/img/etc/cron.d/
cat /mnt/img/etc/cron.d/*
ls /mnt/img/etc/cron.{hourly,daily,weekly,monthly}/
ls /mnt/img/var/spool/cron/
cat /mnt/img/var/spool/cron/*

distro split for user crontabs:

FamilyPath
Debian / Ubuntu/var/spool/cron/crontabs/<user>
RHEL / CentOS/var/spool/cron/<user> (no subdir)
journalctl --root=/mnt/img -t CRON --since '<incident-date>'   # actual cron execution evidence

verify cron entries actually executed - cron fails silently when the command path does not exist. parse the command line as /bin/sh -c would and confirm the executable exists.

systemd persistence

[T1053.006]  Scheduled Task/Job: Systemd Timers
[T1543.002]  Create or Modify System Process: Systemd Service

find /mnt/img/etc/systemd /mnt/img/usr/lib/systemd /mnt/img/lib/systemd -name '*.service' -o -name '*.timer' | xargs stat -c '%y %n' | sort -r | head -20

stock CentOS 7 ships ~430 unit files. mtime-sort cuts the read to 20. Anything with mtime inside the adversary window is the lead.

grep -rE 'ExecStart=' /mnt/img/etc/systemd/system/ /mnt/img/usr/lib/systemd/system/

read every ExecStart= for non-distro-standard binaries (anything outside /usr/bin, /usr/sbin, /usr/lib, /sbin).

Init / boot script persistence

[T1037]  Boot or Logon Initialization Scripts

cat /mnt/img/etc/rc.local           # follow symlink on RHEL family - real file at /etc/rc.d/rc.local
cat /mnt/img/etc/rc.d/rc.local
ls -la /mnt/img/etc/init.d/
ls -la /mnt/img/etc/rc{0..6}.d/

look for SXX symlinks in rcN.d/ that don’t point to standard init.d files. Read rc.local body against the stock template - usually empty or contains a single exit 0.

Shell-config persistence

[T1546.004]  Event Triggered Execution: Unix Shell Configuration Modification

cat /mnt/img/etc/profile
cat /mnt/img/etc/bashrc
ls /mnt/img/etc/profile.d/
cat /mnt/img/etc/profile.d/*.sh
find /mnt/img/home /mnt/img/root \( -name '.bashrc' -o -name '.bash_profile' -o -name '.bash_logout' -o -name '.zshrc' \) -ls

stock skel sizes are stable across distro releases. Compare against expected:

FileStock size (CentOS 7 user)Stock size (root)
.bashrc231 B176 B
.bash_profile193 B176 B
.bash_logout18 B18 B

deviation in size or mtime past distro-build date is the lead. Watch for backgrounded commands ending in & that detach an attacker process.

Web shell persistence

[T1505.003]  Server Software Component: Web Shell

find /mnt/img/var/www /mnt/img/srv -name '*.php' -newer /tmp/before-incident -ls
find /mnt/img/var/www -type f \( -name '*.php' -o -name '*.jsp' -o -name '*.aspx' \) | xargs stat -c '%y %n' | sort -r | head -20

cross-reference web-shell hits with [[Web Log Triage]] access-log queries. A shell file on disk + access-log hit = confirmed deployment.

Privilege escalation taxonomy

sudo + sudoers.d drop-ins
[T1548.003]  Abuse Elevation Control Mechanism: Sudo and Sudo Caching

cat /mnt/img/etc/sudoers
ls /mnt/img/etc/sudoers.d/
cat /mnt/img/etc/sudoers.d/*
grep -E 'NOPASSWD|ALL.*ALL' /mnt/img/etc/sudoers /mnt/img/etc/sudoers.d/*

sudoers.d/ drop-ins are read by visudo at parse time but invisible from a cat /etc/sudoers alone. Always enumerate the directory.

Privileged group membership
[T1098]  Account Manipulation

grep -E '^(wheel|sudo|docker|adm|root):' /mnt/img/etc/group
diff /mnt/img/etc/group /mnt/img/etc/group-

distro split: Debian/Ubuntu use sudo group, RHEL/CentOS use wheel. docker group membership is a privesc path on hosts running Docker - members can mount host paths into containers as root.

Setuid / setgid baseline diff
[T1548.001]  Abuse Elevation Control Mechanism: Setuid and Setgid

find /mnt/img -xdev \( -perm -4000 -o -perm -2000 \) -type f | xargs stat -c '%y %n' | sort -r | head -30

stock CentOS 7 has 25 setuid binaries. Stock Ubuntu LTS has ~30. Any binary outside the baseline path set (/usr/bin, /usr/sbin, /bin, /sbin) or with mtime inside the adversary window is the lead.

Shadow access (credential dumping)
[T1003]  OS Credential Dumping

stat /mnt/img/etc/shadow                     # atime within adversary window = read by attacker

Birth on shadow tracks the last useradd/usermod/passwd (rename-on-write atomic replace). atime reflects last read - hits inside the adversary window mean the attacker copied the hashes for offline cracking.

Triage decision flow

graph TD
    mounted["image mounted ro at /mnt/img"] --> accounts["account diff<br/>passwd/group/shadow vs backups"]
    accounts -->|"new account"| stat_acc["stat for placement timestamp<br/>then SSH key + sudoers + group check"]
    accounts -->|"clean"| ssh["SSH authorized_keys host-wide find"]
    ssh --> cron["cron audit<br/>5 paths + journalctl -t CRON"]
    cron --> sysd["systemd unit mtime sort"]
    sysd --> init["init.d / rc.local diff"]
    init --> shell["shell config + skel size diff"]
    shell --> web["web shell hunt<br/>cross-ref Web Log Triage"]
    web --> privesc["privesc: sudoers / groups / setuid / shadow atime"]

Pitfalls

  • dormant accounts treated as “no finding”. account creation = persistence regardless of operational state. report dormant accounts with a “no current operational role” qualifier and the placement timestamp.
  • assuming a cron entry actually executes. parse the command line as /bin/sh -c does. verify the executable path exists. cross-check journalctl -t CRON for execution evidence.
  • stopping at the persistence artefact and not investigating the source. a public GitHub repo clone = T1588.002 Obtain Capabilities: Tool. Separate finding from the persistence + execution techniques.
  • -anewer against a noatime-mounted filesystem. atime is not updated, so -anewer returns nothing. Use -newer (mtime) or -cnewer (ctime) instead.
  • reference-file ctime trap. touch -t YYYYMMDDhhmm /tmp/marker updates marker’s mtime AND ctime. find -cnewer /tmp/marker then includes the marker creation event itself. Use a marker on a separate filesystem or pre-existing file with known timestamps.
  • find / without -xdev. recurses into /proc and /sys and produces enormous synthetic output. Always -xdev or explicit excludes.
  • /etc/passwd mtime ambiguity. shadow-utils use lock-read-write-rename, so /etc/passwd Birth tracks the last operation, not the file’s first appearance. Use the passwd- / group- / shadow- backup files for history.
  • stat-ing a symlink instead of its target. use stat -L or the target path directly. /etc/rc.local is commonly a symlink on RHEL.
  • regex on /etc/shadow hash field. [^!*] matches one character. Real hashes are 50+. Use awk -F: '$2 ~ /^\$/ {print $1}'.
  • sudoers.d/ drop-ins invisible from /etc/sudoers. always enumerate the directory.
  • sub-technique vs parent technique discipline. T1098.004 (SSH keys) is a sub-technique of T1098 (Account Manipulation). Tag the most specific applicable technique.

Field Manual | Mount Disk Images | Filesystem Metadata | Log Triage | Web Log Triage

Graph View

Backlinks