blog

Spiders/Bots - Category Archive

Friday, May 11th, 2007

166,000 Page Test – 2 1/2 Week Followup

Our test to see if Google would index 166,000 new pages (when people at Google say to only put up 10,000 new pages at a time), is going well. Google is crawling the pages and they’re starting to show up in the SERPs.

Read More...
Thursday, May 3rd, 2007

Ask.com Doesn’t ‘Get’ 301 Permanent Redirects…

One of the rules in search engine optimization is to use permanent redirects to guide search engines from old URLs to new URLs, but Ask.com doesn’t observe them – or more precisely their implementation of them is completely wacked.

Read More...
Saturday, April 28th, 2007

I feel sorry for Google…

I managed to botch the launching of over 166,000 pages, and GoogleBot just dealt with it…

Read More...
Tuesday, April 17th, 2007

Be Careful: Robots.txt Is Case Sensitive

Controlling spiders on your site can be difficult. Now you find they’re accessing pages you never intended because they view the robots.txt file as case sensitive – even if they know your site is not case sensitive…

Read More...
Friday, April 13th, 2007

Media/Image Crawlers Need to See HTML Pages

When crafting your robots.txt file, don’t forget that the search engines have specialized spiders that crawl for image search. These spiders need to see not only the image file, but the page that it is used on.

Read More...
HOME · CREATIVE · WEB · TECH · BLOG