Controlling spiders on your site can be difficult. Now you find they’re accessing pages you never intended because they view the robots.txt file as case sensitive - even if they know your site is not case sensitive…
Read More...If your server is not properly configured to give a proper response for a robots.txt file your site will not be indexed by Google.
Read More...4D AdSense Advertising AJAX Ask Bad Bots Blogging Brand Management Cell Phones Community Content Development CSS Databases Design Domains Duplicate Content Feeds Firefox Flex Google Google Analytics Hackers Image Search IndexTools Internet Explorer Link Building Marketing Meta Tags Mobile Browsers MS Gatineau MSN/Live OS X PageRank PHP Podcasting Safari Search Engines Search Marketing SEO/SEM Server Admin SOAP Social Media Spam Spiders/Bots Technology Uncategorized Video Web 2.0 Web Analytics Web Application Development Web Browsers Web Design Web Site Configuration Web Site Traffic Yahoo!
