Here’s a little howto on merging multiple apache log files when you are migrating servers or want to consolidate your logs into one place. This works on ubuntu and debian, but your mileage may vary on other OS’s.

Here’s the howto:

  1. Login to old server

  2. scp old log files over, to the admin home directory, in /logs

     # scp -r /home/username/logs/ <username@yournewserver>:
  3. If copying multiple directories, and you want to call them something other than logs do this:

     # scp -r /home/username/logs/ <username@yournewserver:/home/username/logs_domainname/> 
  4. Login to new server

  5. Go to /home/username/logs (or /home/username/logs_domainname)

  6. Make a backup of current logs

     #mkdir new_logs  
     #cp access_log error_log ./new_logs
  7. Make a few directories

     # mkdir old_logs  
     # mkdir concat_logs
  8. Enter the old logs directory, and copy the old log files over

     # cd old_logs  
     # cp /home/admin/logs/* .
  9. Now we’re going to concatenate all these to a new single file. is part of the awstats distribution and has been copied to /usr/local/bin.

     # ./access_log* > access_log_overall_old  
     # ./error_log* > error_log_overall_old
  10. Now we’ll leave this directory, and enter the concat_logs directory

    # cd ..  
    # cd concat_logs
  11. Now we’ll copy our newly created log files over to this directory

    # cp ../old_logs/error_log_overall_old ../old_logs/access_log_overall_old .
  12. Now we’ll copy in the “new” log files from the new production server. This step can be omitted if there are no stats on this server already, or they are not needed.

    # cp ../new_logs/* .  
    # ls -l
    -rw-r–r– 1 root root 554 Sep 2 19:38 access_log  
    -rw-r–r– 1 root root 3702368 Sep 2 19:37 access_log_overall_old  
    -rw-r–r– 1 root root 115 Sep 2 19:38 error_log  
    -rw-r–r– 1 root root 907085 Sep 2 19:37 error_log_overall_old
  13. Here we’ll combine the old logfile, with the new logfile. This again, can be omitted if not using old stats and new stats.

    # ./access_log ./access_log_overall_old > access_log_final  
    # ./error_log ./error_log_overall_old > error_log_final
  14. Now we’ll either have in our directory, a final old log file, or a final combined file from this server, and the old server up to the minute.

       # cp ./access_log_overall_old ../access_log  
       # cp ./error_log_overall_old ../error_log  
       # /etc/init.d/apache restart
  15. If you did do step’s 12 and 13, then do this:

    # cp ./access_log_final ../access_log  
    # cp ./error_log__final ../error_log  
    # /etc/init.d/apache restart
  16. You can now delete the old_logs, new_logs, concat_logs directories if wanted.

    # cd /home/domain/logs/  
    # rm -rf concat_logs  
    # rm -rf new_logs  
    # rm -rf old_logs
  17. To update the logs, and make them visible on the awstats interface run this:

    # /usr/lib/cgi-bin/ -config=yourdomain.tld -update wq
  18. Visit them on the web at;

  19. Add the update line to your crontab to schedule it. You can also use that comes with the awstats distribution to update all sites at one time if vhosting.

  20. Go have a rum and coke to celebrate!