User-agent: * Disallow: / # disallow all User-agent: * Disallow: / Crawl-delay: 600 # but allow only important bots User-agent: Googlebot User-agent: Googlebot-Image User-agent: Mediapartners-Google User-agent: msnbot User-agent: msnbot-media User-agent: Slurp User-agent: Yahoo-Blogs User-agent: Yahoo-MMCrawler # Directories Disallow: /wp-includes/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /berlin/ Disallow: /ftp/ Disallow: /images/