Operating system logs provide a wealth of diagnostic information about your computers, and Linux is no exception. Everything from kernel events đồ sộ user actions is logged by Linux, allowing you đồ sộ see almost any action performed on your servers. In this guide, we’ll explain what Linux logs are, where they’re located, and how đồ sộ interpret them.
Linux System Logs
Linux has a special directory for storing logs called /var/log
. This directory contains logs from the OS itself, services, and various applications running on the system. Here’s what this directory looks lượt thích on a typical Ubuntu system.
Some of the most important Linux system logs include:
/var/log/syslog
and/var/log/messages
store all global system activity data, including startup messages. Debian-based systems lượt thích Ubuntu store this in /var/log/syslog
, while Red Hat-based systems lượt thíchRHEL
orCentOS
use/var/log/messages
./var/log/auth.log
and/var/log/secure
store all security-related events such as logins, root user actions, and output from pluggable authentication modules (PAM). Ubuntu and Debian use/var/log/auth.log
, while Red Hat and CentOS use/var/log/secure
./var/log/kern.log
stores kernel events, errors, and warning logs, which are particularly helpful for troubleshooting custom kernels./var/log/cron
stores information about scheduled tasks (cron jobs). Use this data đồ sộ verify your cron jobs are running successfully.
Some applications also write log files đồ sộ this directory. For example, the Apache trang web server writes logs đồ sộ the /var/log/apache2 directory (on Debian), while MySQL writes logs đồ sộ the /var/log/mysql directory. Some applications also log via Syslog, which we’ll explain in the next section.
What’s Syslog?
Syslog is a standard for creating and transmitting logs. The word “syslog” can refer đồ sộ any of the following:
- The syslog service receives and processes syslog messages and listens for events by creating a socket located at
/dev/log
, which applications can write đồ sộ. It can write messages đồ sộ a local tệp tin or forward messages đồ sộ a remote server. There are different syslog implementations, including rsyslogd and syslog-ng. - The Syslog protocol (RFC 5424) is a transport protocol specifying how đồ sộ transmit logs over a network. It’s also a data format defining how messages are structured. By mặc định, it uses port 514 for plaintext messages and port 6514 for encrypted messages.
- A syslog message is any log formatted in the syslog message format and consists of a standardized header and message containing the log’s contents.
Since Syslog can forward messages đồ sộ remote servers, it’s often used đồ sộ forward system logs đồ sộ log management solutions such as SolarWinds® Loggly® and SolarWinds Papertrail™.
RFC 3164 vs. RFC 5424
While RFC 5424 is the current Syslog protocol, it’s not the only standard you’ll see in the wild. RFC 3164 (a.k.a. “BSD syslog” or “old syslog”) is an older syslog format still used by many devices. In practice, admins are likely đồ sộ see syslog messages that use both RFC 3164 and RFC 5424 formatting.
Good indicators of an RFC 3164 syslog message are the absence of structured data and timestamps using an “Mmm dd hh:mm:ss” format.
Here are some examples of what BSD messages look lượt thích, using section 5.4 of RFC 3164 as a reference:
<34>Nov 11 11:11:11 pepeggserver su: 'su admin' failed for user1 on /dev/pts/0
<13>Nov 11 11:11:11 198.51.100.11 Read the docs!
We’ll focus on the newer RFC 5424 protocol here, but keep RFC 3164 in mind if you see messages that don’t conform đồ sộ RFC 5424.
Syslog Format and Fields
Syslog messages contain a standardized header with several fields. These include the timestamp, the name of the application that generated the sự kiện, the location in the system where the message originated, and its priority. You can change this format in your syslog implementation’s configuration tệp tin, but using the standard format makes it easier đồ sộ parse, analyze, and route syslog events.
Here’s an example log message using the mặc định format. It’s from the SSH daemon (sshd), which controls remote logins đồ sộ the system. This message describes a failed login attempt:
Jun 4 22:14:15 server1 sshd[41458] : Failed password for root from 10.0.2.2 port 22 ssh2
You can also add additional fields đồ sộ your syslog messages. Let’s repeat the last sự kiện after adding a few new fields. We’ll use the following rsyslog template, which adds the priority (<%pri%>), protocol version (%protocol-version%), and the date formatted using RFC 3339 (%timestamp:::date-rfc3339%):
<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% %msg%n
This generates the following log:
<34>1 2019-06-05T22:14:15.003Z server1 sshd - - pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.0.2.2
Below, you’ll find descriptions of some of the most commonly used syslog fields when searching or troubleshooting issues.
Priority
The priority field or pri for short (“<34>” in the example) tells you how urgent or severe the sự kiện is. It’s a combination of two numerical fields: the facility and the severity. The facility specifies the type of process that created the sự kiện, ranging from 0 for kernel messages đồ sộ 23 for local applications. The severity ranges from 0 – 7, with 0 indicating an emergency and 7 indicating a debug sự kiện.
Pri can be output in two ways. The first is as a single number, prival, which is calculated as the facility field value multiplied by eight; then, the result is added đồ sộ the severity field value: (facility)(8) + (severity). The second is pri-text, which will output in the string format “facility.severity”. The latter format is often easier đồ sộ read and tìm kiếm but takes up more storage space.
Timestamp
The timestamp field (“2019-06-05T22:14:15.003Z” in the above example) indicates the time and date the message was generated on the system sending the message. The example timestamp breaks down lượt thích this:
- “2019-06-05” is the year, month, and day.
- “T” is a required element of the timestamp field, separating the date and the time.
- “22:14:15.003” is the 24-hour format of the time, including the number of milliseconds (003).
- “Z” indicates UTC time. Instead of Z, the example could have included an offset, such as -08:00, which indicates that the time is offset from UTC by eight hours.
Hostname
The hostname field (“server1” in the example) indicates the name of the host or system that originally sent the message.
App-name
The app-name field (“sshd:auth in the example) indicates the name of the application that sent the message.
Logging with systemd
Many Linux distributions ship with systemd—a process and service manager. Systemd implements its own logging service called journald, which can replace or complement Syslog. Journald logs in a significantly different manner than thở systemd, which is why it has its own section in the Ultimate Guide đồ sộ Logging. You can learn more about logging via systemd in the Systemd Logging section.
Additional Resources
How đồ sộ View and Configure Linux Logs on Ubuntu and CentOS (DigitalOcean)
Last updated: 2022