Microsoft announced that it’s MSNBot now supports “if-modified-since” HTTP headers and GZip compression… About 10 years too late…

Against the advice of the folks at Google, we put up a medical thesaurus with over 166,000 pages (they say to limit the number of new pages to about 10,000) - and things went really well. The pages got indexed and we’ve started seeing an uptick in traffic.

Google Webmaster Tools has an option to set the crawl rate for your site. I’d never seen “faster” enabled as an option until I was working on a web site for a non-profit animal shelter.

Our test to put up 166,000+ pages all at once is looking good… Quite a few pages have been indexed by Google.