Rsync and Log Processing

To manage and process logs of multiple webservers, Rsync provides the best method for transferring the logs from servers to centralized log processing server.

There are two major steps involved
a) Configuring the Log Processing Server (IP: 192.168.1.1)
b) Configuring the client (say, www.myserver.com) to transfer the logs to the central server.

Configuring the Log Processing Server

Let this server have IP address 192.168.1.1. We create a directory , /usr/local/logs, where the log files would be downloaded. We create a subdirectory for www.myserver.com, under /usr/local/logs. (mkdir /usr/local/logs/www.myserver.com)

a. create a group logman and add user logman to it. This will be the uid/gid for the log files
b. edit/create /etc/rsyncd.conf, with the following details:

uid = logman
gid = logman
use chroot = yes
max connections = 4
log file = /var/log/rsyncd.log
pid file = /etc/rsyncd.pid

[www.myserver.com_logs]
comment = here are the apache access logs from www.myserver.com downloaded
path = /usr/local/logs/www.myserver.com/
hosts allow = www.myserver.com
read only = no

c. Now, run rsync:
# rsync --daemon

We have now successfully configured our server to received log files.

Configuring the clients to transfer the logs

on the client system (www.myserver.com), run this command periodically, to transfer the logs:
rsync -azvu /usr/local/apache/logs/access_logs 192168.1.1::www.myserver.com_logs

This way, the logs would be transferred to 192.168.1.1, and would be updated (not deleted and recreated, differential transfer) everytime.