Getting Listed On Search Engines Part I - Where Am I

You can almost always track where your visitors are coming from. To keep a good eye on your traffic there are some excellent tools you can use. There are the roll your own variety which are hosted on your server, and the pre-made, already hosted, ready to use 3rd party variety. My favorite is to use 1 of each if you can.

Self Hosted Website Traffic Analysis
My favorite tool in this genre is AWStats. This is a free tool that with closely look at your log files for apache, IIS, webstar, mail server, wap, proxy, streaming servers and some ftp servers. Based on the data collected from these log files, graphs, charts and a plethora of other data will be generated for your servers.

AWStats provides number of unique visitors, pages, hits, bandwidth statistics broken down by month, day of month, day of week and hour! AWStats will also tell you which country the visitors are from, which spiders have been to your site, when the spider was last seen, and how many hits they generated. This tool also provides duration of visits, entry/exit statistics by page, visitor operating system and browser, screen sizes, keywords, keyphrases. There is also plugins you can add to AWStats or you can write your own. The possibilities are truly endless.

AWStats can also be a bit cumbersome to setup and requires you to know what you are doing in apache configuration files. You need to be able to setup Allow, Deny rules in apache for the awstats folder and have apache generate custom log files that awstats can read and use.

The statistics that can be collected server side are very pure since we don't have to depend on client side scripts to say "Hey I'm at your site and this is how I found you!". We still do have to depend on client side scripts to retrieve the screen sizes.

There are other tools out, Webalizer, WebTrends, W3Perl, all of these are free but I have personally not had any experience with them.

3rd Party Website Traffic Analysis
My favorite tool in this genre is Google Analytics. If you don't have a Google account yet, go sign up for one! Once you sign up for this free service, and create a analytics profile for your site, Google will give you a little piece of javascript to put somewhere on all the pages you would like to track. Google is really good about providing easy to use and understand step by step instructions to getting this done.

You can have up to 50 different profiles setup, which means you can track up to 50 different web sites from one account. There are approximately 80 different reports each that can be shaped to your likings. You can view traffic by geographic region, how long they have stayed on a specific page, number of visits, number of pages per visit. Most importantly you can tell where the traffic came from, directly typing in the URL, search engines, or referring sites. If they came from a search engine it will list which keywords were used to find you.

Google analytics is not without it's drawbacks though. The results may be slightly skewed, some ad filtering programs and browser extensions block the Urchin javascript used by Google analytics which will cause the visitor to not be counted. Also if they visitor doesn't have javascript turned on in their browser then they aren't tracked. Privacy applications such as Tor will show users coming from IP addresses that aren't theirs, which will equate to a Tor user in California showing up as coming from Germany.

Do not let the drawbacks of Google Analytics scare you away, it is an excellent tool to use especially if you do not have access to the server log files.


Am I Listed?
Once you have made your website a candy emporium for search engines and have the ability to see where your traffic is coming from, you need to get listed on the big 3.

• To check and see if you are listed on Google do a search at their site for site:YOURFULLURL
  
• To check and see if you are listed on yahoo do a search at their site for your partial URL, everything after http:// will work or everything after http://www.
• To check and see if you are listed on msn do a search at their site for your full URL

So you have now determined if you are listed or not, if you are then great, get to optimizing and stop reading now. Go create some candy for your search engine friends! If you aren't listed then we have a bit of work to do.

Questions, comments and criticism are always welcome, search engine optimization and web development are life long learning adventures.