I have a database logging all website visits. How can I separated between real visitors and crawlers?
You can use Google Analytics for the best results, and the most data regarding your visitors. To have an idea what information they can offer you: http://www.icantinternet.org/2009/01/stats-show-you-what/
If you want to track it yourself, then you should use the “user agent string”, which holds the identification of the browser (or the spider in that case). I've written an article on how to use this string to identify an iPhone visitor, but by simply changing the string to look for, it also works for spiders: http://www.icantinternet.org/2009/01/recognizing-the-iphone-for-webpages/
powered by Yahoo Answers
April 8th, 2009 at 12:40 am
You should use the Google analytics to help you on it.
References :
http://www.neuracom.com
April 8th, 2009 at 1:04 am
You can use Google Analytics for the best results, and the most data regarding your visitors. To have an idea what information they can offer you: http://www.icantinternet.org/2009/01/stats-show-you-what/
If you want to track it yourself, then you should use the “user agent string”, which holds the identification of the browser (or the spider in that case). I've written an article on how to use this string to identify an iPhone visitor, but by simply changing the string to look for, it also works for spiders: http://www.icantinternet.org/2009/01/recognizing-the-iphone-for-webpages/
References :
http://www.icantinternet.org/2009/01/stats-show-you-what/
http://www.icantinternet.org/2009/01/recognizing-the-iphone-for-webpages/