my Site been crawled by googlebot – about 28000 hits and msnboot about 1000 hits. Is this possible that indexing caused a high cpu usage? I been advised to setup a robots.txt to limit googlebot hits, but I am not sure how to do this.
I think there is a lot of traffic on your site and typically Google is pretty light weight about this stuff
.robots.txt won’t help you here. You can sign up for a free account at Google Webmaster Tools and in there you can ask them to crawl your site less often. But robots.txt is all or nothing – you can say don’t ever crawl these set of pages, but that’s NOT what you want!
ask your web host, which scripts are responsible for high cpu load?
Leave a Reply
You must be logged in to post a comment.