Seo: Are Bots Able To Crawl Password Protected Pages Or Secure Pages?
SEO November 10th, 2009If you have a site and it requires a username/password, will bots be able to index the contents?
Also for secure sites, will it be able to index?
If you have a site and it requires a username/password, will bots be able to index the contents?
Also for secure sites, will it be able to index?
November 10th, 2009 at 4:34 am
Password protected: No.
Secure: Yes, as long as the content isn’t behind a password. (Google, for example, is able to crawl at least the initial page, before the login.)
November 10th, 2009 at 4:34 am
Secure: yes. Password protected: no. Also, as a rule, I add a robots noindex/nofollow to the login page so as to let the search engine spiders know they shouldn’t even try.
November 10th, 2009 at 4:34 am
Hello Keep reading my bloghttp://www.praveenkodur.com/blog/http://www.praveenkodur.com/
November 10th, 2009 at 4:34 am
The use of robots.txt will exclude bots that crawl for major search engines. However, there are many ‘guerilla bots’ that dont observe these.
November 10th, 2009 at 4:34 am
If they could, what would be the sense in the pw protection?