Logging is a critical aspect of Linux server management. Log messages are useful for root cause analysis and avoiding potential error occurrences in the future. Analyzing and debugging server errors is a core skill to have for both IT engineers and system administrators.

This guide will show you how to set up a remote logging server, also known as a log host, on Linux. A log host allows you to aggregate local Linux logs to a remote centralized server for ease of access and analysis.

Why Have a Dedicated Log Server?

The Linux operating system logs most activities on your server for auditing and debugging using the syslog (system logging protocol) daemon. So you might be wondering, why do I need a dedicated server for my logs? Here are some advantages to having a dedicated logging server:

  • Better security because the remote logging server only has a few ports open to the outside.
  • Improved server performance because the remote logging host does not run many services, except the ones used for logging.
  • Eases archiving and management of log messages.

Log messages are important for auditing your servers and base-lining and are a core part of preventive maintenance procedures on your server infrastructure.

Step 1: Installing rsyslog on Linux

This guide focuses on Ubuntu 20.04, but the process should be pretty much the same if you are using other mainstream Linux distros.

rsyslog is a remote logging service for Linux and comes preinstalled by default on most modern Linux distros, for example, Ubuntu and other Debian-based systems.

The rsyslog service is a modern and improved daemon to syslog, which only allows you to manage logs locally. With the rsyslog daemon, you can send your local logs to some configured remote Linux server.

If you do not have rsyslog installed on your PC, you can easily do so using the following command, on Debian-based distros:

        sudo apt install rsyslog
    

On Red Hat Linux, you can install it by typing:

        yum install rsyslog
    

On Fedora and its derivatives, run:

        dnf install rsyslog
    

To install rsyslog on Arch Linux:

        yay -S rsyslog
    

To check the status of rsyslog, run the following command:

        systemctl status rsyslog
    

Output:

rsyslog_status

Step 2: Configuring the Log Host Server

The log host is the server configured to receive log messages from other servers or PCs. The rsyslog configuration resides in the /etc/rsyslog.conf file.

You can open the /etc/rsyslog.conf file using any text editor of your choice. In this guide, we'll use Vim.

You'll need elevated privileges to make changes to the config file.

Before you start editing the config file, you should take a backup or copy of the file. To do so, run the command:

        sudo cp /etc/rsyslog.conf /etc/rsyslog_original.config
    

Next, open the /etc/rsyslog.conf file using a text editor.

        sudo vim /etc/rsyslog.conf 
    

There are two protocols you can use for sending/receiving log files with rsyslog: TCP and UDP. This guide shows you how to configure both.

You do not need to configure both UDP and TCP for remote logging to work. Only choose one of the two.

If you prefer to use UDP, look for and uncomment the following lines by removing the leading Pound (#) symbol preceding the lines. You can find these lines under the modules section of the config file.

        module(load="imudp")
input(type="imudp" port="514")

If you prefer to use TCP, then uncomment the following lines by removing the leading Pound (#) symbol located at the beginning of the lines:

        module(load="imtcp")
input(type="imtcp" port="514")

The following figure shows the rsyslog configuration file configured to use UDP communication:

udp_configuration_rsyslog

Next, configure the location where rsyslog will store your logs. For better organization, you should categorize incoming logs by their origin. Define a template in your rsyslog config file by adding the following lines:

        $template remote-incoming-logs, "/var/log/remote/%HOSTNAME%".log
*.* ?remote-incoming-logs

The aforementioned lines command rsyslog to store the logs in the folder /var/log/remote/hostname, where hostname is the name of the remote client that is sending log messages to the log host.

Now, save the changes you've made. If you are using Vim, here is how to save and quit a file.

Finally, restart the rsyslog services for the changes you've made to take effect.

        sudo systemctl restart rsyslog
    

Step 3: Configuring Your Firewall

If your firewall is enabled, make sure that the port you have configured above is able to communicate with the outside world. You'll need to edit your firewall rules to allow incoming logs.

For Debian-based distros, simply use the UFW tool, to enable either the UDP or TCP transfer protocol.

Related: How to Configure the Firewall in Ubuntu Using UFW

If you are using UDP, run the following command, where 514 is the configured port number:

        sudo ufw 514/udp
    

If you are using TCP on port 514, simply run:

        sudo ufw 514/tcp
    

On Fedora, you can use firewall-cmd to achieve similar results.

        firewall-cmd --zone=zone --add-port=514/udp
    

For Red Hat Linux, open the iptables file located at /etc/sysconfig/iptables using your text editor of choice, and add the following rule:

        -A INPUT -m state --state NEW -m udp -p udp --dport 514 -j ACCEPT
    

Restart the iptables service for the changes to take effect.

        service iptables restart
    

Step 4: Configuring the Logging Client

The client is the machine that sends its logs to a remote or centralized log host server. Open the rsyslog config file located at /etc/rsyslog.conf:

        sudo vim /etc/rsyslog.conf
    

Add the following line if you are using UDP, where 192.168.12.123 is the IP address of the remote server, you will be writing your logs to:

        *.* @192.168.12.123:514
    

If you are using TCP, add the following line instead. Note that the line has two @ symbols.

        *.* @@192.168.12.123:514
    

Save your changes and restart the rsyslog service on the client with the command:

        sudo systemctl restart rsyslog
    

Step 5: Viewing the Log Messages on the Server

You can use SSH to log in to your remote server and view the logs sent from the client servers. In this case, rsyslog is configured so that it stores the client logs in the /var/log/remote directory of the remote server.

        cd /var/logs/remote
    

Then list the contents of the directory using the ls command:

        ls -l
    

As you can see in the output, the directory contains log messages for the remote servers named andiwa and rukuru. Their log files are named andiwa.log and rukuru.log respectively.

logs_on_remote_log_host

You can then look at the log files using a text editor or with Linux file viewing tools such as cat or less.

Remote Logging Gives You More Control

This guide has looked at how to set up a remote logging server (log host) on Linux.

A log host offers you better organization and control when it comes to logging. Even in scenarios where a system is damaged or inaccessible, you can still view its logs from the log host and figure out what went wrong.