DFIR-Field-Manual

Log Triage

6 min read

Linux log triage for incident response and forensic analysis. /var/log catalogue, auth.log patterns, systemd journalctl flag walk, offline image analysis, boot history, shell history. Verified against systemd 259 and Kali userland.

Side: blue

/var/log catalogue

FileContents
/var/log/syslog (Debian/Ubuntu)general system + service messages
/var/log/messages (RHEL/CentOS)same role on Red Hat-family
/var/log/auth.log (Debian/Ubuntu)logins, sudo, su, SSH, PAM
/var/log/secure (RHEL/CentOS)equivalent to auth.log
/var/log/kern.logkernel messages
/var/log/apache2/access.log / error.logApache HTTP access + errors
/var/log/nginx/access.log / error.logNginx HTTP access + errors
/var/log/mysql/error.logMySQL errors, slow query log when enabled
/var/log/squid/access.logSquid proxy traffic
/var/log/audit/audit.logauditd records (when configured)
/var/log/wtmp / /var/log/btmpbinary login / failed-login records — read with last / lastb
/var/log/utmpcurrently logged-in users — read with who / w
/var/log/journal/<machine-id>/systemd binary journal — read with journalctl

On modern systemd hosts, journalctl is the primary interface. /var/log/syslog exists in parallel only when rsyslog is installed.

auth.log patterns

grep "Failed password" auth.log | wc -l                                  # brute-force volume signal
grep "Failed password" auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr   # rank attacker IPs
grep "Accepted " auth.log | tail -20                                     # last successful logins
grep "session opened" auth.log | tail                                    # PAM sessions opened
grep "sudo:" auth.log | awk -F'COMMAND=' '{print $2}' | sort | uniq -c | sort -nr     # sudo commands ranked

Line patterns to recognise:

PatternMeans
Failed password for <user> from <ip>SSH auth failure (mass = brute force)
Failed password for invalid user <x>username does not exist on host
Accepted password for <user> from <ip>successful interactive login
Accepted publickey for <user> from <ip>SSH key-based login, no password prompt
session opened for user <user>PAM session granted (login or sudo)
authentication failure ... rhost=<ip>PAM failure from remote host
sudo: <user> : TTY=... COMMAND=...exact sudo invocation

Brute force signature: many Failed password lines, single source IP, incrementing source ports, seconds-scale time gaps.

journalctl essentials

journalctl                                               # full journal, oldest first, paged
journalctl -ax                                           # show all fields + catalog explanations
journalctl -t sudo                                       # SYSLOG_IDENTIFIER filter
journalctl SYSLOG_IDENTIFIER=sudo                        # field-equality form (multiple = AND, + = OR)
journalctl -u ssh.service                                # filter by systemd unit
journalctl --since '2026-04-15' --until '2026-04-16'    # date window (also: '1 hour ago', 'yesterday')
journalctl -p err                                        # by syslog priority (emerg/alert/crit/err/warning/notice/info/debug or 0..7)
journalctl -g 'pattern'                                  # grep MESSAGE field (case-insensitive)
journalctl -k                                            # kernel messages this boot (alias --dmesg)
journalctl -f                                            # follow live, like tail -f
journalctl -o verbose                                    # show all fields per entry
journalctl --disk-usage                                  # how much history is preserved
Boot history
journalctl --list-boots                                  # IDX  BOOT-ID  FIRST-ENTRY  LAST-ENTRY
journalctl -b                                            # current boot only (= -b 0)
journalctl -b -1                                         # previous boot
journalctl -b <boot-id-32hex>                            # specific boot by ID

Forensic uses for --list-boots:

  1. given a known-bad timestamp, find the boot whose FIRST..LAST covers it, then read journalctl -b <id> for that window
  2. mismatched boot count vs uptime metadata = journal truncated, Storage=volatile discarded, or vacuumed
  3. one boot over a wide range = adversary may have been resident across all uptime
Offline image analysis
journalctl --root=/mnt/image                             # operate on a mounted image as if it were /
journalctl --root=/mnt/image --list-boots
journalctl --root=/mnt/image -t sudo --since '2026-04-17'
journalctl --root=/mnt/image -b <boot-id> -k
 
journalctl --image=/path/to/image.dd                     # operate directly on disk image (no LVM/encrypted)

--root= looks up <root>/etc/machine-id, finds the matching journal directory under <root>/var/log/journal/<machine-id>/, and reads from there. Equivalent to copying out individual *.journal files and using journalctl --file=... per file but vastly faster.

Verify the switch worked: journalctl --root=/mnt/image --list-boots should show the image’s boot history, not the analysis workstation’s.

Tampering detection
journalctl --verify                                       # check journal file consistency (FSS seals)
journalctl --root=/mnt/image --verify                     # against an image

Failures indicate truncation, replacement, or seal break.

Tamper-aware reading
  • Storage=volatile in /etc/systemd/journald.conf writes only to /run/log/journal/ — reboots wipe everything. Always check journald config before trusting the journal as a complete record.
  • Storage=auto (default) is persistent if /var/log/journal/ exists, volatile otherwise. An attacker who deletes the persistent dir causes silent fallback.
  • journalctl --rotate --vacuum-time=1s discards everything older than one second. Requires root. Evidence-destruction pattern.
  • Per-user journals at /var/log/journal/<machine-id>/user-<uid>.journal when Storage=persistent and the session was systemd-managed.

dmesg vs journalctl -k

Aspectdmesgjournalctl -k
Sourcekernel ring buffer (in-memory)journal files on disk
Persistencewiped on rebootsurvives reboots if Storage=persistent
Time formatseconds since boot (use dmesg -T for wall clock)wall clock with TZ offset
Cross-bootcurrent boot only-b -N or -b <id> for previous
On evidence imagen/a (kernel ring is volatile)works via --root=

For kernel events around an incident on a host that has rebooted since, the journal is the only source.

Login records (binary)

last                                                      # /var/log/wtmp - successful logins
last -f /mnt/image/var/log/wtmp                           # against an image
last -F                                                   # full date/time format
last -i                                                   # IPs, not hostnames
lastb                                                     # /var/log/btmp - failed logins (root only)
lastlog                                                   # last login per user (one row each)
who                                                       # currently logged-in users (utmp)
w                                                         # currently logged-in + activity

lastlog “Never logged in” rows are normal for system accounts; investigate when an account that never logged in suddenly has a login.

Shell history

cat /home/<user>/.bash_history                            # bash, no timestamps unless HISTTIMEFORMAT set
cat /home/<user>/.zsh_history                             # zsh, ': <epoch>:<duration>;<cmd>' if EXTENDED_HISTORY
cat /home/<user>/.local/share/fish/fish_history           # fish, YAML-ish blocks
cat /root/.bash_history                                   # always check root
 
find /home -name '.*history' -o -name '*_history' 2>/dev/null    # exhaustive sweep
stat /home/<user>/.bash_history                           # mtime = last clean shell exit, btime = file creation

Patterns worth flagging:

  • wget / curl from suspicious URLs (initial staging)
  • chmod +x against unusual paths
  • shell upgrades: bash -i, python -c "import pty; pty.spawn('/bin/sh')", nc -e, socat ... EXEC
  • privilege probes: sudo -l, sudo find ... -exec, sudo vim ...
  • history-cleanup: history -c, history -d <N>, unset HISTFILE, export HISTFILE=/dev/null, kill -9 $$
  • direct config edits: vi /etc/sudoers, nano /etc/passwd, vim ~/.ssh/authorized_keys
  • staging into world-writable: /tmp, /var/tmp, /dev/shm
Limitations to flag in reports
  • History files are editable by the owning user.
  • History flushes only at clean shell exit. kill -9 of the shell leaves recent commands in memory only.
  • Non-history shells (dash, ash, sh, busybox-sh) leave nothing.
  • Empty .bash_history on a long-active account is itself a finding, not an absence of finding.
  • Cross-check intent against journal sudo lines, auth.log SSH source IPs, and cron logs. History alone is the weakest source.

Standard text-log idioms

grep -F 'pattern' file                                    # fixed-string match (faster, no regex)
grep -E 'a|b|c' file                                      # extended regex
grep -v 'pattern' file                                    # invert match
awk -F: '$3>=1000 {print $1}' /etc/passwd                 # column-extract by delimiter
 
sort file | uniq -c | sort -nr                            # frequency rank
sort -u file                                              # unique values
cut -d',' -f1,3 file.csv                                  # CSV columns 1 and 3
 
journalctl -t sudo --since '2026-04-15' | grep -i 'COMMAND' | awk -F'COMMAND=' '{print $2}' | sort | uniq -c | sort -nr
grep "Failed password" auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr

Field Manual | Linux Log Analysis | Linux Forensics | Mount Disk Images

Graph View

Backlinks