Here’s a little howto on merging multiple apache log files when you are migrating servers or want to consolidate your logs into one place. This works on ubuntu and debian, but your mileage may vary on other OS’s.
Here’s the howto:
Login to old server
scp old log files over, to the admin home directory, in /logs
# scp -r /home/username/logs/ <username@yournewserver>:
If copying multiple directories, and you want to call them something other than logs do this:
# scp -r /home/username/logs/ <username@yournewserver:/home/username/logs_domainname/>
Login to new server
Go to /home/username/logs (or /home/username/logs_domainname)
Make a backup of current logs
#mkdir new_logs #cp access_log error_log ./new_logs
Make a few directories
# mkdir old_logs # mkdir concat_logs
Enter the old logs directory, and copy the old log files over
# cd old_logs # cp /home/admin/logs/* .
Now we’re going to concatenate all these to a new single file. Logresolvemerge.pl is part of the awstats distribution and has been copied to /usr/local/bin.
# logresolvemerge.pl ./access_log* > access_log_overall_old # logresolvemerge.pl ./error_log* > error_log_overall_old
Now we’ll leave this directory, and enter the concat_logs directory
# cd .. # cd concat_logs
Now we’ll copy our newly created log files over to this directory
# cp ../old_logs/error_log_overall_old ../old_logs/access_log_overall_old .
Now we’ll copy in the “new” log files from the new production server. This step can be omitted if there are no stats on this server already, or they are not needed.
# cp ../new_logs/* . # ls -l -rw-r–r– 1 root root 554 Sep 2 19:38 access_log -rw-r–r– 1 root root 3702368 Sep 2 19:37 access_log_overall_old -rw-r–r– 1 root root 115 Sep 2 19:38 error_log -rw-r–r– 1 root root 907085 Sep 2 19:37 error_log_overall_old
Here we’ll combine the old logfile, with the new logfile. This again, can be omitted if not using old stats and new stats.
# logresolvemerge.pl ./access_log ./access_log_overall_old > access_log_final # logresolvemerge.pl ./error_log ./error_log_overall_old > error_log_final
Now we’ll either have in our directory, a final old log file, or a final combined file from this server, and the old server up to the minute.
# cp ./access_log_overall_old ../access_log # cp ./error_log_overall_old ../error_log # /etc/init.d/apache restart
If you did do step’s 12 and 13, then do this:
# cp ./access_log_final ../access_log # cp ./error_log__final ../error_log # /etc/init.d/apache restart
You can now delete the old_logs, new_logs, concat_logs directories if wanted.
# cd /home/domain/logs/ # rm -rf concat_logs # rm -rf new_logs # rm -rf old_logs
To update the logs, and make them visible on the awstats interface run this:
# /usr/lib/cgi-bin/awstats.pl -config=yourdomain.tld -update wq
Visit them on the web at; http://www.domainname.com/awstats/awstats.pl
Add the update line to your crontab to schedule it. You can also use updateall.pl that comes with the awstats distribution to update all sites at one time if vhosting.
Go have a rum and coke to celebrate!