Comments on: Seo: Are Bots Able To Crawl Password Protected Pages Or Secure Pages?

By: strayinm

strayinm — Tue, 10 Nov 2009 04:34:46 +0000

Password protected: No.
Secure: Yes, as long as the content isn’t behind a password. (Google, for example, is able to crawl at least the initial page, before the login.)

By: trentroc

trentroc — Tue, 10 Nov 2009 04:34:46 +0000

Secure: yes. Password protected: no. Also, as a rule, I add a robots noindex/nofollow to the login page so as to let the search engine spiders know they shouldn’t even try.

By: kpraveen

kpraveen — Tue, 10 Nov 2009 04:34:46 +0000

Hello Keep reading my bloghttp://www.praveenkodur.com/blog/http://www.praveenkodur.com/

By: Megan C

Megan C — Tue, 10 Nov 2009 04:34:46 +0000

The use of robots.txt will exclude bots that crawl for major search engines. However, there are many ‘guerilla bots’ that dont observe these.

By: thomy8s

thomy8s — Tue, 10 Nov 2009 04:34:46 +0000

If they could, what would be the sense in the pw protection?