Configure Rsyslog Centralized Log Server on Ubuntu 20.04|18.04

Logs on any Linux system are critical for analyzing and troubleshooting any issues related to system and applications. With the help of tools like Graylog, you can easily ship these logs to a centralized platform for easy visualization. In this guide, we will look at how to Configure Rsyslog Centralized Log Server on Ubuntu 20.04|18.04 LTS.

On Linux, by default, all log files are located under /var/log directory. There are several types of log files storing varying messages, which can be cron, kernel, security, events, users e.t.c. Mostly these logs file are controlled by rsyslog service.

On recent systems with systemd, some logs are managed by journald daemon and they are written binary format. These logs are volatile since they are written to RAM and doesn’t withstand system reboot. They are often found on./run/log/journal/ But note that journald can also be configured to permanently store log messages by writing to file.

Configure Rsyslog Log Server on Ubuntu 20.04|18.04

We’re going to configure rsyslog server as central Log management system. This follows the client-server model where rsyslog service will listen on either udp/tcp port. The default port used by rsyslog is 514. On the client system, rsyslog will collect and ship logs to a central rsyslog server over the network via UDP or TCP ports.

When working with syslog messages, there is a priority/severity level that characterizes a log file. Namely:

• emerg, panic (Emergency ): Level 0 – This is the lowest log level. system is unusable
• alert (Alerts):  Level 1 – action must be taken immediately
• err (Errors): Level 3 – critical conditions
• warn (Warnings): Level 4 – warning conditions
• notice (Notification): Level 5 – normal but significant condition
• info (Information): Level 6 – informational messages
• debug (Debugging):  Level 7 – This is the highest level – debug-level messages

Rsyslog is installed by default on a freshly installed Ubuntu 20.04|18.04 system. If for any reason the package is not installed, you can install it by running:

sudo apt-get update
sudo apt-get install rsyslog

When installed, check service to see if it is running:

$ sudo systemctl status rsyslog
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-07-18 11:30:10 UTC; 4 days ago
Main PID: 903 (rsyslogd)
Tasks: 4 (limit: 4704)
CGroup: /system.slice/rsyslog.service
└─903 /usr/sbin/rsyslogd -n

Configure rsyslog to run in Server Mode

Now configure rsyslog service to run in server mode:

sudo vim /etc/rsyslog.conf

Uncomment the lines for udp and tcp port binding:

module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

If you would like to limit access from to specific subnet, IP or domain, add like below:

$AllowedSender TCP, 127.0.0.1, 192.168.10.0/24, *.example.com

You can add above line after input(type="imtcp" port="514") line. Remember to substitute given values with correct ones

Create a new template for receiving remote messages

Let’s create a template that will instruct rsyslog server how to store incoming syslog messages. Add the template just before GLOBAL DIRECTIVES section:

$template remote-incoming-logs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log" 
*.* ?remote-incoming-logs
& ~

The received logs will be parsed using the template above and stored inside directory/var/log/. The file naming follows the convention: %HOSTNAME% and %PROGRAMNAME% variables, i.e the client hostname, and client facility that produced the log message.

The & ~ instructs rsyslog daemon to store the log message only to a specified file.

Other variables that can be used include:

%syslogseverity%, %syslogfacility%, %timegenerated%, %HOSTNAME%, %syslogtag%, %msg%, %FROMHOST-IP%, %PRI%, %MSGID%, %APP-NAME%, %TIMESTAMP%, %$year%, %$month%, %$day%

Restart rsyslog service for the changes to take effect:

sudo systemctl restart rsyslog

Confirm if the service is listening on configured ports:

$ ss -tunelp | grep 514

udp    UNCONN   0        0                 0.0.0.0:514            0.0.0.0:*      users:(("rsyslogd",pid=15220,fd=5)) ino:441816 sk:2 <->                        
udp    UNCONN   0        0                    [::]:514               [::]:*      users:(("rsyslogd",pid=15220,fd=6)) ino:441817 sk:5 v6only:1 <->               
tcp    LISTEN   0        25                0.0.0.0:514            0.0.0.0:*      users:(("rsyslogd",pid=15220,fd=7)) ino:441820 sk:a <->                        
tcp    LISTEN   0        25                   [::]:514               [::]:*      users:(("rsyslogd",pid=15220,fd=8)) ino:441821 sk:11 v6only:1 <->

Configure Rsyslog firewall

If you have ufw firewall service running, allow rsyslog firewall ports:

sudo ufw allow 514/tcp
sudo ufw allow 514/udp

Configure Rsyslog as a Client

Once you’re done configuring rsyslog server, head over to your rsyslog client machines and configure them to send logs to remote rsyslog server.

sudo vim /etc/rsyslog.conf

Allow preservation of FQDN:

$PreserveFQDN on

Add remote rsyslog server at the end:

*.* @ip-address-of-rsysog-server:514

You can also use FQDN instead of Server IP Address:

*.* @fqdn-of-rsysog-server:514

The above line will enable sending of logs over UDP, for tcp use @@ instead of a single @

*.* @@ip-address-of-rsysog-server:514
#OR
*.* @@fqdn-of-rsysog-server:514

Also add the following for when rsyslog server will be down:

$ActionQueueFileName queue
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionQueueType LinkedList
$ActionResumeRetryCount -1

Then restart rsyslog service

sudo systemctl restart rsyslog

Also check:

Manage Logs with Graylog server on Ubuntu 18.04

How to Install Graylog 2.4 with Elasticsearch on CentOS 7.