System Logs

Logs are the first thing to check when something doesn't work. On modern Linux there are two systems: journald (journalctl) for service logs, and text files in /var/log/ for application logs.

02

journalctl: system and service logs

Most used commands

bash
# All logs (from oldest)
journalctl

# Logs in real time (like tail -f)
journalctl -f

# Last N messages
journalctl -n 50

# Log of a specific service
journalctl -u nginx
journalctl -u mariadb
journalctl -u sshd
journalctl -u php8.2-fpm

# Log of a service in real time
journalctl -u nginx -f

# Log of current boot
journalctl -b

# Log of previous boot (useful to understand why the server restarted)
journalctl -b -1

# Only errors and critical messages
journalctl -p err
journalctl -p err -b    # Only errors from current boot

# Logs in a time interval
journalctl --since "2024-03-01" --until "2024-03-28"
journalctl --since "1 hour ago"
journalctl --since "today"

Filter by keyword

bash
journalctl -u nginx | grep "error"
journalctl -u sshd | grep "Failed"
journalctl | grep -i "oom\|killed"
03

Traditional log files in /var/log/

FileContent
/var/log/syslogGeneral system log (Debian/Ubuntu)
/var/log/messagesGeneral log (CentOS/AlmaLinux)
/var/log/auth.logLogin, sudo, SSH (Debian/Ubuntu)
/var/log/secureLogin, SSH (CentOS)
/var/log/nginx/access.logHTTP requests received by Nginx
/var/log/nginx/error.logNginx errors
/var/log/apache2/error.logApache errors
/var/log/mysql/error.logMySQL/MariaDB errors
/var/log/fail2ban.logFail2ban blocks
/var/log/kern.logKernel log
/var/log/dpkg.logPackage installations (Debian/Ubuntu)

Useful commands for log files

bash
# Read a file in real time
tail -f /var/log/nginx/error.log

# Last 100 lines
tail -100 /var/log/nginx/access.log

# Search for a word in logs
grep "error" /var/log/nginx/error.log
grep -i "refused\|denied" /var/log/auth.log

# How many times an IP appears in the access log
grep "1.2.3.4" /var/log/nginx/access.log | wc -l

# IPs making the most requests (possible attack)
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20
04

Understand why the server restarted

bash
# Last reboot
last reboot | head -5

# OOM killer (process terminated due to lack of RAM)
journalctl -b -1 | grep -i "oom\|killed process\|out of memory"
dmesg | grep -i "oom"

# Kernel panic or crash
journalctl -b -1 -p 0..3

# Current uptime
uptime
05

Find suspicious SSH access

bash
# Failed login attempts
grep "Failed password" /var/log/auth.log | tail -20

# IPs attempting brute force
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -10

# Successful logins
grep "Accepted" /var/log/auth.log | tail -20

# Last accesses
last | head -20
06

Manage log size

bash
# How much space journald logs use
journalctl --disk-usage

# Delete logs older than 30 days
journalctl --vacuum-time=30d

# Limit logs to 500MB total
journalctl --vacuum-size=500M

# Configure limits permanently
nano /etc/systemd/journald.conf
ini
[Journal]
SystemMaxUse=500M
MaxRetentionSec=1month
bash
systemctl restart systemd-journald
07

Automatic log rotation (logrotate)

Log rotation prevents files from becoming enormous. Verify it is configured:

bash
# Test the configuration
logrotate --debug /etc/logrotate.conf

# Force manual rotation
logrotate -f /etc/logrotate.conf

# Nginx configuration (example)
cat /etc/logrotate.d/nginx

DeluxHost, fondata nel 2023, offre soluzioni di hosting di alta qualità per diverse esigenze digitali. Forniamo hosting condiviso, VPS e server dedicati con sicurezza avanzata e datacenter globali.

© DeluxHost, Tutti i diritti riservati. | Partita IVA: IT17734661006
Tutti i sistemi operativi