When a web robot visits a site, it goes to the "/robots.txt" page on a site, which is protocol, from where it receives instructions on which areas of the domain it can or cannot visit.

If there is no "/robots/txt" file within a domain, the robot gathers that it can traverse all areas within the domain. The file for robots should always be placed immediately after your domain name, and not in another folder, ie. "www.yoursite.com/robots.txt"

To exclude a robot from visiting certain areas within your domain, like the /cgi-bin/ for example, simply include those details in your file. To disallow all robots to traverse your site, the robots.txt page should look like this

User-agent *
Disallow: /

To allow robots to traverse your site

User-agent *
Disallow:

or just create an empty robots.txt file.

To tell the robots to exclude certain areas within your domain, simply add a file as follows :

User-agent: *
Disallow: /cgi-bin/
Disallow: /Admin/

In the process, you can also variate for your own reasons whether you want to exclude a particular robot and allow others.

User-agent: LousyBot
Disallow: /

No comments yet. Be the first one to comment.