This is the first post in our two-part AdSense Crawler Errors series.

There are many ways that publishers can go about optimizing their site for AdSense; opting-in to text/image ads, upgrading to our preferred ad formats, and increasing ad coverage across a site are just a few of the more well-known ones. But did you know that there’s another straightforward optimization tip that many publishers often overlook?

A bit of context
Your site’s robots.txt file essentially acts as a gatekeeper that determines which web crawlers, web robots, and search engines have access to your site and which do not. Those that are granted permission can do things like view your pages and index your site. Those that don’t have permission are not able to view or index specific sections of your site, depending on what you’ve specified.

AdSense ads are displayed through the use of an AdSense web crawler. That crawler scans your page’s content and determines which ads to display, according to specific keywords. If our AdSense crawler is being blocked by your robots.txt file, we’re going to have a difficult time displaying relevant ads on your site. As a result, your users may see less relevant ads, which can lead to a lower CTR.

How you can help yourself
View the contents of your robots.txt file by going to [yourdomain.com]/robots.txt.  (If you have a subdomain, it likely has a robots.txt file as well, located at [sub.yourdomain.com]/robots.txt.) Be sure that the file is configured to allow our AdSense ad crawler to view your site. You can do that by simply adding the following two lines to the very top of the file:

User-agent: Mediapartners-Google
Disallow:

This will ensure that our AdSense ad crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. Please note that making this change will not impact your Google search rankings. Adding these two lines to your robots.txt file will only help to deliver better, more relevant ads to pages with AdSense code already on them. Pages that don’t have AdSense ad code will not be affected.

If you have URLs with any errors, you can see what they are by logging into your AdSense account and clicking on ‘Account Settings’ from the home page. From there, click on ‘View errors’ under ‘Access and Authorization.’

Stay tuned for the second post in our AdSense Crawler Errors series, where we’ll cover crawler login issues and how you can solve them.

Posted by Andrew Boni - Inside AdSense Team