Click to enlarge

Here’s a quick breakdown of how to get the usage of your different Amazon CloudFront Edge locations within AWStats. I confess that I get some crazy pleasure from collecting statistics, still though, bandwidth usage is of interest in this case, as well as hits and that it can be interesting to see which regions most of the visitors come from.

I use the cluster functionality in AWStats to display the CloudFront Edges, as can be seen in the screenshot to the right.

I’m running AWStats on an Ubuntu server and this guide assumes that you are already collecting your CloudFront logs to a server and processes them with AWStats. If not, check out my other article dealing with that.

Let’s work with our hypothetical CloudFront distribution named cdn.example.com.

In Ubuntu AWStats stores the configurations files in /etc/awstats/ so I will assume there is a file located there named
/etc/awstats/awstats.cdn.example.com.conf
which processes the CloudFront log already.

If you have configured the above .conf file properly for CloudFront, %cluster is collected in the LogFormat which contains all information needed to be able to display the edges. So now we only need to tell AWStats to start using that information.

First in the awstats.cdn.example.com.conf, enable cluster statistics by finding ShowClusterStats=0 and change it to:
ShowClusterStats=PHB
And then continue down to the plugin section and uncomment
LoadPlugin="clusterinfo"

Now we only need to tell AWStats the name of the edges, so they make sense when checking the statistics later on. In AWStats DirData directory, create a file named clusterinfo.[your awstats config name].txt

Which in our example would be
/var/lib/awstats/clusterinfo.cdn.example.com.txt

This file shall contain a tab separated list of cluster numbers and then a text to display for each cluster. At the time of writing this article this is the complete list to paste into that file for all available CloudFront Edges. If Amazon adds more edges in the future they will have to be added at that time.

DFW3    USA: Dallas/Fort Worth, TX
EWR2    USA: Newark, NJ
IAD2    USA: Ashburn, VA
JAX1    USA: Jacksonville, FL
JFK1    USA: New York, NY
LAX1    USA: Los Angeles, CA
MIA3    USA: Miami, FL
SEA4    USA: Seattle, WA
SFO4    USA: Palo Alto, CA
STL2    USA: St. Louis, MO
AMS1    Europe: Amsterdam
DUB2    Europe: Dublin
FRA2    Europe: Frankfurt
LHR3    Europe: London
HKG1    Asia: Hong Kong
NRT4    Asia: Tokyo
SIN2    Asia: Singapore

Okay, that’s it, the next time AWStats updates the statistics output you should start getting Edge usage outputted in the Cluster section, like in the screenshot above.

To make this happen I’ve written a script in Python with the boto module that downloads all generated log files to a local folder and then deletes them from the Amazon S3 Bucket when done. The log files downloaded to the local folder can then be further processed with logresolvemerge and AWStats.

You need to have the boto module installed for this to work. Personally I’m working with Ubuntu 10.04, where boto can be easily installed by executing:

sudo apt-get install python-boto

The script takes some command line arguments that are listed in the doc header. All of these can have a default value set in the head of the get_logs class. If you set default values in the script, the command line arguments are useful if you need to override a default value on some occasions.

get-aws-logs.py

#! /usr/bin/env python
"""Download and delete log files for AWS S3 / CloudFront

Usage: python get-aws-logs.py [options]

Options:
  -b ..., --bucket=...    AWS Bucket
  -p ..., --prefix=...    AWS Key Prefix
  -a ..., --access=...    AWS Access Key ID
  -s ..., --secret=...    AWS Secret Access Key
  -l ..., --local=...     Local Download Path
  -h, --help              Show this help
  -d                      Show debugging information while parsing

Examples:
  get-aws-logs.py -b eqxlogs
  get-aws-logs.py --bucket=eqxlogs
  get-aws-logs.py -p logs/cdn.example.com/
  get-aws-logs.py --prefix=logs/cdn.example.com/

This program requires the boto module for Python to be installed.
"""

__author__ = "Johan Steen (http://www.artstorm.net/)"
__version__ = "0.5.0"
__date__ = "28 Nov 2010"

import boto
import getopt
import sys, os

_debug = 0

class get_logs:
	"""Download log files from the specified bucket and path and then delete them from the bucket.
	Uses: http://boto.s3.amazonaws.com/index.html
	"""
	# Set default values
	AWS_BUCKET_NAME = '{bucket}'
	AWS_KEY_PREFIX = '{prefix}'
	AWS_ACCESS_KEY_ID = '{access key}'
	AWS_SECRET_ACCESS_KEY = '{secret key}'
	LOCAL_PATH = '{local path}'
	# Don't change below here
	s3_conn = None
	bucket_list = None

	def __init__(self):
		s3_conn = None
		bucket_list = None

	def start(self):
		"""Connect, get file list, copy and delete the logs"""
		self.s3Connect()
		self.getList()
		self.copyFiles()

	def s3Connect(self):
		"""Creates a S3 Connection Object"""
		self.s3_conn = boto.connect_s3(self.AWS_ACCESS_KEY_ID, self.AWS_SECRET_ACCESS_KEY)

	def getList(self):
		"""Connects to the bucket and then gets a list of all keys available with the chosen prefix"""
		bucket = self.s3_conn.get_bucket(self.AWS_BUCKET_NAME)
		self.bucket_list = bucket.list(self.AWS_KEY_PREFIX)

	def copyFiles(self):
		"""Creates a local folder if not already exists and then download all keys and deletes them from the bucket"""
		# Using makedirs as it's recursive
		if not os.path.exists(self.LOCAL_PATH):
			os.makedirs(self.LOCAL_PATH)
		for key_list in self.bucket_list:
			key = str(key_list.key)
			# Get the log filename (L[-1] can be used to access the last item in a list).
			filename = key.split('/')[-1]
			# check if file exists locally, if not: download it
			if not os.path.exists(self.LOCAL_PATH+filename):
				key_list.get_contents_to_filename(self.LOCAL_PATH+filename)
				if _debug:
					print "Downloaded from bucket: "+filename
			# check so file is downloaded, if so: delete from bucket
			if os.path.exists(self.LOCAL_PATH+filename):
				key_list.delete()
				if _debug:
					print "Deleted from bucket:    "+filename

def usage():
	print __doc__

def main(argv):
	try:
		opts, args = getopt.getopt(argv, "hb:p:l:a:s:d", ["help", "bucket=", "prefix=", "local=", "access=", "secret="])
	except getopt.GetoptError:
		usage()
		sys.exit(2)
	logs = get_logs()
	for opt, arg in opts:
		if opt in ("-h", "--help"):
			usage()
			sys.exit()
		elif opt == '-d':
			global _debug
			_debug = 1
		elif opt in ("-b", "--bucket"):
			logs.AWS_BUCKET_NAME = arg
		elif opt in ("-p", "--prefix"):
			logs.AWS_KEY_PREFIX = arg
		elif opt in ("-a", "--access"):
			logs.AWS_ACCESS_KEY_ID = arg
		elif opt in ("-s", "--secret"):
			logs.AWS_SECRET_ACCESS_KEY = arg
		elif opt in ("-l", "--local"):
			logs.LOCAL_PATH = arg
	logs.start()

if __name__ == "__main__":
	main(sys.argv[1:])

Have in mind that I’m pretty new to Linux and to Python, so I bet things can be solved better, easier or in a more beautiful way than what I’ve done, as well as making it more fail safe.

Feel free to suggest improvements that can be made to the code.

This led me to think that it would be real handy to get the log messages sent out by email as well, so you don’t risk to have a problem going on at the site unnoticed for days or even weeks.

So I wrote an extension to the native Log class in CodeIgniter which adds this functionality. Below is the code for MY_Log.php which extends the CI_Log class. MY_Log calls the native Log functions so the log file still gets generated, and if a log message was executed it also gets sent out by email afterward.

Change the email address in the code to the one you want to receive the log messages to, and then save the file in your application/libraries/ folder as MY_Log.php. This assumes that your Class Extension Prefix is set to MY_ in your config file.

I hope some of you find this class extension useful, and feel free to improve upon it. Cheers!

<?php
/**
 * MY_Log Class
 *
 * This library extends the native Log library.
 * It adds the function to have the log messages being emailed when they have been outputted to the log file.
 *
 * @package		CodeIgniter
 * @subpackage		Libraries
 * @category		Logging
 * @author		Johan Steen
 * @link		http://wpstorm.net/
 */
class MY_Log extends CI_Log {
	/**
	 * Constructor
	 *
	 * @access	public
	 */
	function MY_Log()
	{
		parent::CI_Log();
	}

	/**
	 * Write Log File
	 *
	 * Calls the native write_log() method and then sends an email if a log message was generated.
	 *
	 * @access	public
	 * @param	string	the error level
	 * @param	string	the error message
	 * @param	bool	whether the error is a native PHP error
	 * @return	bool
	 */
	function write_log($level = 'error', $msg, $php_error = FALSE)
	{
		$result = parent::write_log($level, $msg, $php_error);

		if ($result == TRUE && strtoupper($level) == 'ERROR') {
			$message = "An error occurred: \n\n";
			$message .= $level.' - '.date($this->_date_fmt). ' --> '.$msg."\n";

			$to = 'someone@example.com';
			$subject = 'An error has occured';
			$headers = 'From: Example Name <no-reply@example.com>' . "\r\n";
			$headers .= 'Content-type: text/plain; charset=utf-8\r\n';

			mail($to, $subject, $message, $headers);
		}
		return $result;
	}
}
?>