Robots.txt

3.3out of 10
  • Type
  • Shade
  • DifficultyEasy
  • Time
  • Weight ?

Google

Yahoo

MSN

  1. 589 views
  2. 0 votes

Description

The Robots.txt file can be a great way to make the most out of your link equity and keep the search engines on the most popular and relevant pages. For most large sites there are usually sections that get very little attention, yet have hundreds of thousands of pages.

Things such as galleries or sorting links may be new indexable pages that have little relevant or original content to users. Directories that hold these pages could easily be thrown in a robots file as pages to exclude.

By excluding low PR and irrelevant pages you are basically passing your link juice in a much more dense area. These search engines will also spend more time crawling useful pages with a properly configured robots.txt file.