Articles - Robots who Readby Michael Bloch
Posted on January 20, 2004
|
|
Before you submit your site to a search engine, you may want to consider what you want the search engine "bot" (the program that indexes your site) to "spider" (follow). You may have pages with sensitive information, or a scrap directory full of pages in progress that you would not like to see listed.
This can be achieved 2 ways. The first way is with a robots.txt file placed in the root directory of your web, but you must have full domain privileges in order for this to work. For an article relating to this subject:
http://tamingthebeast.net/ articles2/search-engine-spiders.htm
- A quick note on the robots.txt file - do not leave it empty. This will indicate to some search engines that you do not want any part of the site indexed.
The other way to stop a number of bots from searching a page is to use META exclusion tags.
The following META tags can be used:
<META NAME="ROBOTS" CONTENT="NOINDEX">
Putting this line between the <head> and </head> tags in your HTML will prevent the bot from indexing that page.
An alternative is:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
The page will be indexed, but any hyperlinks in that page will not be spidered by the bot.
Or a combination of the two:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
Page will not be indexed, and other links will not be followed by the bot. This tag may also prevent some web-stripper (mirroring) software programs from downloading the page.
Further learning resources:
Optimization and Search engines
Search engine optimization tutorials. Increase your web site traffic by gaining better page rankings.
http://www.tamingthebeast.net/ articles2/optimization-tutorials.htm
Meta Tags and Zen (revised)
Easy to understand tutorial on the use of Meta Tags for boosting search engine rankings. An optimized set of meta-tags is an important component of any overall search engine strategy.
http://www.tamingthebeast.net/ articles/metatags.htm
Michael Bloch
Taming the Beast
http://www.tamingthebeast.net
Tutorials, web content, tools and software.
Web Marketing, Internet Development & Ecommerce Resources
____________________________
Copyright information.... This article is free for reproduction but must be reproduced in its entirety & this copyright statement must be included. Visit http://www.tamingthebeast.net for free Internet marketing and web development articles, tutorials and tools! Subscribe for free to our popular ecommerce/web design ezine!
This can be achieved 2 ways. The first way is with a robots.txt file placed in the root directory of your web, but you must have full domain privileges in order for this to work. For an article relating to this subject:
http://tamingthebeast.net/ articles2/search-engine-spiders.htm
- A quick note on the robots.txt file - do not leave it empty. This will indicate to some search engines that you do not want any part of the site indexed.
The other way to stop a number of bots from searching a page is to use META exclusion tags.
The following META tags can be used:
<META NAME="ROBOTS" CONTENT="NOINDEX">
Putting this line between the <head> and </head> tags in your HTML will prevent the bot from indexing that page.
An alternative is:
<META NAME="ROBOTS" CONTENT="NOFOLLOW">
The page will be indexed, but any hyperlinks in that page will not be spidered by the bot.
Or a combination of the two:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
Page will not be indexed, and other links will not be followed by the bot. This tag may also prevent some web-stripper (mirroring) software programs from downloading the page.
Further learning resources:
Optimization and Search engines
Search engine optimization tutorials. Increase your web site traffic by gaining better page rankings.
http://www.tamingthebeast.net/ articles2/optimization-tutorials.htm
Meta Tags and Zen (revised)
Easy to understand tutorial on the use of Meta Tags for boosting search engine rankings. An optimized set of meta-tags is an important component of any overall search engine strategy.
http://www.tamingthebeast.net/ articles/metatags.htm
Michael Bloch
Taming the Beast
http://www.tamingthebeast.net
Tutorials, web content, tools and software.
Web Marketing, Internet Development & Ecommerce Resources
____________________________
Copyright information.... This article is free for reproduction but must be reproduced in its entirety & this copyright statement must be included. Visit http://www.tamingthebeast.net for free Internet marketing and web development articles, tutorials and tools! Subscribe for free to our popular ecommerce/web design ezine!
Print This Article|
Send To A Friend|
RSS Feeds|Read More Related ArticlesCOMMENT ON THIS ARTICLE...
No comments yet. Be the first one to comment.
Post them now in our forums for quick, helpful advice from thousands of members!
Giving search engine spiders direction with a 301 redirect
Search Engine Basics: Title And META-Tags
Effective Search Engine Use
Search Engine Basics: Title And META-Tags
Effective Search Engine Use
SEO Articles
Internet Marketing Articles
Development Articles
General Articles
And also in our Archives
Internet Marketing Articles
Development Articles
General Articles
And also in our Archives
Drive traffic to your business and get recognized as an industry leader by sharing your knowledge on Site-Reference. Authors are given a wide range of exclusive benefits here at SR; so checkout what we can offer to those that…
We’re always on the lookout for new writting talent so even if haven’t written for the web yet, feel free to contact us anytime
We’re always on the lookout for new writting talent so even if haven’t written for the web yet, feel free to contact us anytime
Home|Forums|SEO Blog|Marketing Blog|Resources|Advertise Here|RSS Feeds|Contact Us|Affiliates
Copyright 2004 - 2008 Site-Reference.com|Privacy Policy
