# Robots Tutorial: http://www.searchengineworld.com/robots/robots_tutorial.htm User-Agent: * # Disallow the retrieval of scripts and admin related pages Disallow: /cgi-bin/ Disallow: /_ User-Agent: nicebot Disallow: / User-Agent: microsoft-webdav-miniredir Disallow: /