AWStats was useful for integrating access logs from multiple servers

This is Yamada from the Systems Department.
This time, I'll be talking about how to consolidate Apache access logs distributed across multiple web servers into one.

The beginning

The other day, I had to analyze the access logs of a certain site,
which had multiple web servers in a balanced environment.

Of course, the access logs are different for each server.
It's a very old site, and doesn't even have a fancy tool like fluentd installed.

Since the requirement was a bit urgent, I wanted to quickly integrate the data into one file for each chronological order.
I found it. I found it. A great tool.

"logresolvemerge.pl"

This is a script in an access analysis tool called "awstats," which has become less common recently.
Nowadays, "Google Analytics" is the most famous tool for access analysis, but I
remember that a while ago, it was common to use "awstats." (Yamada's research)

What you can do with this is,

This runs logresolvemerge in command line to open one or several
server log files to merge them (sorted on date) and/or to make a reverse
DNS lookup (if asked). The result log file is sent on standard output.

logresolvemerge runs on the command line and
merges one or several web server log files (sorted by date) and
(optionally) reverse-lookups the DNS. The resulting log is output to standard output.
Source: AWStats logfile analyzer 7.5 Documentation

Oh! This is it!

Usage: logresolvemerge.pl [options] file (date sort only one file) logresolvemerge.pl [options] file1 ... filen (date sort and merge multiple files) logresolvemerge.pl [options] *.* (date sort and merge all logs in a directory) perl logresolvemerge.pl [options] *.* > newfile (date sort and merge all logs in a directory and write to newfile) Options: -dnslookup make a reverse DNS lookup on IP adresses (reverse lookup IP addresses from DNS) -dnslookup=n same with an parallel threads instead of serial requests (run dnslookup with n parallel threads) -dnscache=file make DNS lookup from cache file first before network lookup (perform DNS lookup from cache file "file" before network lookup) -showsteps print on stderr benchmark information every 8192 lines (output benchmark information on stderr every 8192 lines) -addfilenum if used with several files, file number can be added in first. -addfilename if used with several files, file name can be added in the first field of output file. This can be used to add a cluster ID when log files come from several load balanced computers. -stoponfirsteof Stop processing when any logfile reaches end-of-file. -printfields For IIS or W3C logs, prints the latest field header for the current log file when switching between log file entries so that the parser can automatically determine which fields are available. -ignoremissing will not fail if a log file is missing 

As a note, it states that "there is no guarantee," "the sorting is not strict," and "it is not a tool for sorting a single file."
It is a free tool, so please don't get angry if there are any problems.

Let's try it

First, install

Luckily, this server had "awstats" installed, but
if it's not installed on your server, please install it using the following command.

cd /usr/src/ wget http://prdownloads.sourceforge.net/awstats/awstats-7.3.tar.gz tar zxvf awstats-7.3.tar.gz

If you want to install it with yum, click here. (To be honest, if you only want to use logresolvemerge.pl, I think you only need the source.)

yum install --enablerepo=epel awstats

Now let's actually use the tool

Log Consolidation

The working directory is set to [/var/tmp], but please change it as appropriate.
*The installation directory for awstats is set to "/usr/src/awstats-7.3" if you installed it from source.
For now, we will assume that the necessary access logs are placed in the working directory.

Let's integrate access logs!

cd /var/tmp/ perl /usr/src/awstats-7.3/tools/logresolvemerge.pl web01-access_log web02-access_log > merged-access_log less merged-access_log

the
way, there seem to be many other tools under the AWStats tool directory.

maillogconvert.pl ... Converts postfix, sendmail, and qmail logs into a human-readable format. urlaliasbuilder.pl ... Generates a URL alias file from a list file of URLs

Please feel free to use this as a reference!
You can leave the tedious server management and log analysis to