It's already a while ago that AskApache published the post "WordPress robots.txt SEO", but we think it is still quite relevant and many times a 'forgotten' or not completely understood SEO-tool.

They say in their post that adding a good robots.txt file to your WordPress blog or website this leads to higher ranks in the search engines. This can also (eventually) lead to more traffic to your site and - if that is your aim - ads that pay more per click.

In their post AskApache has listed their own robots.txt file (at the time of publishing). If you want to use it, we have pasted it below. Obviously you should change the location of the sitemap.xml to your own location!

The article goes on with a lot of additional information from mostly Google (with links) and more tips. Good stuff!

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content
Disallow: /tag
Disallow: /author
Disallow: /wget/
Disallow: /httpd/
Disallow: /i/
Disallow: /f/
Disallow: /t/
Disallow: /c/
Disallow: /j/

User-agent: Mediapartners-Google
Allow: /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: Googlebot-Mobile
Allow: /

User-agent: ia_archiver-web.archive.org
Disallow: /

Sitemap: http://www.askapache.com/sitemap.xml

#               __                          __
#   ____ ______/ /______ _____  ____ ______/ /_  ___
#  / __ `/ ___/ //_/ __ `/ __ / __ `/ ___/ __ / _
# / /_/ (__  ) ,< / /_/ / /_/ / /_/ / /__/ / / /  __/
# __,_/____/_/|_|__,_/ .___/__,_/___/_/ /_/___/
#                   /_/
#

Popularity: 33% [?]