Magento: Show/Hide website to search engines by adjusting robots meta tag

The Robots META Tag is meant to provide users who cannot upload or control the /robots.txt file at their websites, with a last chance to keep their content out of search engine indexes and services.

<meta name="robots" content="robots-terms">

The content=”robots-terms” is a comma separated list used in the Robots META Tag that may contain one or more of the following keywords without regard to case: noindex, nofollow, all, index and follow.

noindex = Page may not be indexed by a search service.
nofollow = Robots are not to follow links from this page.

You can find more about the robots meta tag over here: http://www.seoconsultants.com/meta-tags/robots/

There is an easy way in Magento to adjust robots meta tags. You can simply choose from admin settings.

Go to System -> Configuration -> Design -> HTML Head -> Default Robots

You have the following options to choose:

– index, follow
– noindex, follow
– index, nofollow
– noindex, nofollow

If you have a live site (production site) then choose ‘index, follow‘. If you have a test site (development site) then choose ‘noindex, nofollow‘.

Hope it helps. Thanks.