Linux (Ubuntu)
Maintenance · #19 of 19

Troubleshooting & Logs

journalctl, dmesg, /var/log

Where to Look When Things Break

Linux logs everything. When something fails, the answer is usually in a log file.

journalctl — Systemd Logs

Modern logging system for systemd:

journalctl                      # All logs
journalctl -b                   # Since last boot
journalctl -b -1                # Previous boot
journalctl --since "1 hour ago" # Recent logs
journalctl --since today        # Today's logs
journalctl -p err               # Only errors
journalctl -f                   # Follow (live)

Service-Specific Logs

journalctl -u ssh               # SSH service logs
journalctl -u nginx -f          # Follow nginx logs
journalctl -u docker --since "10 min ago"

Priority Levels

Level Name Meaning
0 emerg System is unusable
1 alert Immediate action needed
2 crit Critical conditions
3 err Error conditions
4 warning Warning conditions
5 notice Normal but significant
6 info Informational
7 debug Debug messages 
journalctl -p warning           # Warnings and above
journalctl -p 0..4              # Emergency through error

dmesg — Kernel Messages

The kernel logs hardware events, driver messages, and boot info:

dmesg                           # All kernel messages
dmesg | tail -20                # Last 20 lines
dmesg -T                        # Human-readable timestamps
dmesg -T | grep -i error        # Find errors
dmesg -T | grep -i usb          # USB events
dmesg -w                        # Follow (watch for new)

Common Uses

/var/log — Traditional Logs

Classic log files in /var/log:

File Contents
syslog General system messages
auth.log Authentication (logins, sudo)
kern.log Kernel messages
dpkg.log Package installations
apt/history.log APT operations
boot.log Boot messages
nginx/ Web server logs
mysql/ Database logs

Reading Logs

cat /var/log/syslog             # Full file
tail -20 /var/log/syslog        # Last 20 lines
tail -f /var/log/syslog         # Follow live
less /var/log/syslog            # Scrollable viewer
grep error /var/log/syslog      # Find errors

Authentication Logs

# Who logged in?
tail /var/log/auth.log

# Failed login attempts
grep "Failed password" /var/log/auth.log

# sudo usage
grep sudo /var/log/auth.log

Common Troubleshooting Patterns

1. Service Won’t Start

sudo systemctl status myservice
journalctl -u myservice -n 50
# Look for error messages

2. Application Crashes

dmesg | tail -20                # Kernel messages
journalctl -b | grep -i "myapp"
# Check app's own logs if it has them

3. Disk Problems

dmesg | grep -i "error\|fail"
journalctl -p err               # All errors
df -h                           # Check disk space

4. Network Issues

journalctl -u NetworkManager
ping 8.8.8.8                    # Can we reach internet?
ip addr                         # Do we have an IP?

5. Login Problems

tail /var/log/auth.log
journalctl -u ssh

Disk Space for Logs

Logs can fill up your disk:

du -sh /var/log                 # Total log size
ls -lhS /var/log/*.log | head   # Largest log files

# Clean old journals
sudo journalctl --vacuum-time=7d   # Keep 7 days
sudo journalctl --vacuum-size=500M # Keep 500MB max

Useful Tools

less Navigation

Key Action
q Quit
/pattern Search forward
?pattern Search backward
n Next match
N Previous match
g Go to start
G Go to end
F Follow mode

grep Patterns

grep -i error file.log          # Case insensitive
grep -C 3 error file.log        # 3 lines context
grep -v DEBUG file.log          # Exclude DEBUG lines
grep -E "error|fail" file.log   # Multiple patterns
zgrep error file.log.gz         # Search compressed

Try It!

Practice log investigation in the terminal:

Exercises:

  1. View kernel messages: dmesg | head -20
  2. Check for errors: dmesg | grep -i error
  3. View systemd journal: journalctl -n 20
  4. Check auth log: cat /var/log/auth.log 2>/dev/null || echo "Permission denied"

Try it in the terminal

A sandboxed shell with this lesson’s commands. Type help to see what’s available.

bash — try it