Email Standards Project

The Web Robots Pages In a nutshell Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say Before it does so, it firsts checks for and finds: User-agent: * Disallow: / The "User-agent: *" means this section applies to all robots. There are two important considerations when using /robots.txt: robots can ignore your /robots.txt. So don't try to use /robots.txt to hide information. See also: The details The /robots.txt is a de-facto standard, and is not owned by any standards body. In addition there are external resources: The /robots.txt standard is not actively developed. The rest of this page gives an overview of how to use /robots.txt on your server, with some simple recipes. How to create a /robots.txt file Where to put it The short answer: in the top-level directory of your web server.

Initializr - Start an HTML5 Boilerplate project in 15 seconds! HTML5 Reset :: A simple set of templates for any project