How to Configure Your Blog Robots.txt File
― ∙ Using robots on any website or blog instructs web robots on what to do when they visit this site. When a robot wants to visit a website URL, say http://yourdomain.com/index.html; before it does so, it would have to first of all check for http://yourdomain.com/robot.txt. If it finds something like; ∙ User-agent: * Disallow: / ∙ Read more>> .







