Our test to see if Google would index 166,000 new pages (when people at Google say to only put up 10,000 new pages at a time), is going well. Google is crawling the pages and they’re starting to show up in the SERPs.
One of the rules in search engine optimization is to use permanent redirects to guide search engines from old URLs to new URLs, but Ask.com doesn’t observe them – or more precisely their implementation of them is completely wacked.
Controlling spiders on your site can be difficult. Now you find they’re accessing pages you never intended because they view the robots.txt file as case sensitive – even if they know your site is not case sensitive…
When crafting your robots.txt file, don’t forget that the search engines have specialized spiders that crawl for image search. These spiders need to see not only the image file, but the page that it is used on.