How to: Robots.txt

How to: Robots.txt: "There has been some noise around the seo industry today about Google not obeying the NoIndex meta tag. Google is indexing pages that use this tag when they shouldn't be. I have actually seen this first hand myself when our company received an email earlier this week asking for opinions on the seo of a website. We noticed this site was trying to keep pages from being indexed in the search engines by using this meta tag. However, when we searched through Google we noticed these pages were still being indexed. There is an alternate way to block the search engines from indexing your pages. It's called a Robots.txt file. The Robots.txt file is a plain text file that can be created and uploaded to the root folder on your server. If you really would like to keep the search engines from crawling and indexing certain pages on your site this is what you should be using. Here is how to properly create the Robots.txt file. Step 1 - Open a blank .txt file.Step 2 - Define the user agent. In the first line of the text file you will add a User-agent: followed by the bot n"