Logmatic.io Blog

Real-time Log Centralization
with Rsyslog

Rsyslog Configuration to centralize and ship your Linux servers logs

Applications and servers generate tons of information through log events. However, the monitoring of heterogeneous logs coming from tens or hundreds of machines is hard, and sometimes impossible, without an efficient Log Management tool.

The good news is that centralizing log data to a remote place from almost any Linux distribution is actually quite easy. And yes, they almost all come with Rsyslog installed. This daemon is able to watch files and publish data through TCP or UDP connections, as well as gather all the system logs.
In this article, we’ll show you how to configure your Rsyslog in 5 minutes and transmit log events to a central syslog daemon (for specific logmatic.io details read documentation here).

Forward all the syslog messages

If you run Ubuntu, CentOs, Debian, Fedora, openSUSE or any other major linux distribution you should find a rsyslog.log file under your /etc/ directory. To forward all the syslog messages through a TCP socket we will do some edition:

sudo vim /etc/rsyslog.conf

Indicate that you want to send all the messages to a unified syslog serviceon TCP PORT 10514 for instance. The RFC-5424 format is very useful as it is automatically recognized and parsed by many log management solutions. So we want to format all logs as RFC-5424:

$template unifiedFormat," <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n" 
*.* @@unified.syslog.com:10514;unifiedFormat

The *.* before the TCP address is a Rsyslog’s selectors which means that you want to send all the messages.

Once you are happy with this configuration don’t forget to restart the service:

sudo service rsyslog restart

Watching your own files

If your applications do not publish their logs natively in syslog format you can also use Rsyslog as a file content forwarder. To do so, follow these steps:

Edit the rsyslog.conf file:

sudo vim /etc/rsyslog.conf

Insert the following configuration block for each file you want to follow:

# Prefix all the followed filed by these 4 lines
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog
# Input for FILE1
$InputFileName /path/to/your/log/file.log
$InputFileTag your-app-name
$InputFileStateFile be-sure-that-name-is-unique
$InputFileSeverity info
# Input for FILE2
$InputFileName /path/to/another/log/file.log
$InputFileTag your-secondary-app-name
$InputFileStateFile be-sure-that-name-is-unique-to-track-offset$InputFileSeverity info

If you need more details about this configuration, please refer to the imfile module documentation.

Again, once you’re done with this rsyslog configuration don’t forget to restart the service:

sudo service rsyslog restart

Going deeper into Rsyslog configuration

Enable TCP SSL security

You can set up encryption for Rsyslog. To do so, you will need to download your log management certificate first. Then install support TLS for Rsyslog. Usually this is done by installing rsyslog-gnutls via apt-get:

sudo apt-get install rsyslog-gnutls

If you followed the previous step, you can now enable TLS by replacing the following:

$template unifiedFormat," <%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n"

*.* @<@unified.syslog.com:10514;unifiedFormat


$template unifiedFormat," &lt;%pri%&gt;%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% - - - %msg%\n"
$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
*.* @@secure.unified.syslog.com:10515;unifiedFormat

Don’t forget to specify path_to_your_.crt_file, your_api_key and ensure that the port number is the right one (we used 10515 here).

Then you should restart the service.

Filter logs based on the application name

Sometimes, you don’t want to send all your logs to the centralized platform. In order to filter them based on the application name, you have to replace in your rsyslog.conf, the following line:

*.* @@unified.syslog.com:10514;unifiedFormat
if $programname == 'my-app-name' then @@unified.syslog.com:10514;unifiedFormat

Wrapping up

In this article, we discussed how to set your Rsyslog configuration to centralize and categorize any kind of linux server logs easily. Depending if you are a Java, Node.js or Ruby on Rails developer, if you use Apache or Nginx we encourage you to define a proper logging strategy so you can monitor better and troubleshoot 10x times faster.

Logging strategy and analytics is what this blog is all about! So if it is of interest to you, we will try to transmit everything we know and publish fresh articles every week. Check out our other articles on logging strategies, such as the one for Java logs or the Syslog-ng log management software.

See you soon!

Related Posts