The Robots META Tag is meant to provide users who cannot upload or control the /robots.txt file at their websites, with a last chance to keep their content out of search engine indexes and services.
<meta name="robots" content="robots-terms">
The content=”robots-terms” is a comma separated list used in the Robots META Tag that may contain one or more of the following keywords without regard to case: noindex, nofollow, all, index and follow.
noindex = Page may not be indexed by a search service.
nofollow = Robots are not to follow links from this page.
You can find more about the robots meta tag over here: http://www.seoconsultants.com/meta-tags/robots/
There is an easy way in Magento to adjust robots meta tags. You can simply choose from admin settings.
Go to System -> Configuration -> Design -> HTML Head -> Default Robots
You have the following options to choose:
– index, follow
– noindex, follow
– index, nofollow
– noindex, nofollow
If you have a live site (production site) then choose ‘index, follow‘. If you have a test site (development site) then choose ‘noindex, nofollow‘.
Hope it helps. Thanks.