Online Marketing Blog

Fear The Bots For Your Quality Score Sanity

Since Google added a new layer of complexity to the AdWords system this year in the form of Quality Score, not a lot has been published about how it works or why it does a certain thing. In fact, most advertisers are still in the dark about quality score and how they can work with it rather than fight it or work around it. One aspect of the quality score algorithm that has had me frustrated from the get go is how can Google determine the landing page quality score of a page that you tell it it cannot access (by means of robots.txt exclusion) ? After speaking with our Google rep, we finally got our answer.

Are You Still Credible?

While Michael Gray gave a great explanation on working with the AdWords quality robot there were still some unanswered questions. We all know that some of the factors required to get a great quality score are creating credibility indicators like a privacy policy or a terms of service document. We also know that these documents have little value in the main search index and many people like to exclude these documents rather than wasting link authority and to be honest, we are among those people. However, our biggest concern was that if we excluded these documents from the main index to preserve our link equity, would the quality score bot remove whatever quality points it gave for these documents because it could not see them? Getting no answers from anyone, we decided to simply risk it for ourselves. We created a brand new P3P privacy policy for this very site and excluded it using our robots.txt file. Then we waited.

The Results ?

After over a month, our main index rankings have all increased (Although this is more than likely due to other factors). The point is that there was no noticeable decrease in any areas that we focus on in the organic results. It might still be a little early to tell (and I may eat these words) but it appears that this has more of a positive effect than negative.

What About AdWords?

This is where we expected to see the biggest change. We expected the Quality Score for the campaigns that we run for this site to suffer. We might as well use ourselves as guinea pigs to know for sure. After over a month our quality score has not changed at all. Our quality score for all campaigns is “great” across the board. It is important to note at this point that the privacy page was denied access by all robots. However, we noticed that despite this exclusion the AdWords bot (AdsBot-Google) was still visiting. After contacting our Google rep about this, we got the following reply:

In order to avoid increasing CPCs for advertisers who do not intend
to restrict AdWords visits to their pages, the system will ignore blanket
exclusions (User-agent: *) in ‘robots.txt’ files.

So basically, the AdWords bot does not obey the rules set out in your robots.txt unless you explicitly state that you didn’t want that particular bot snooping around. Fair enough, I actually think this is a good thing and I am sure it has already saved a lot of advertisers a lot of heartache. So in our small example, we have saved ourselves some link equity and still had the benefit of having the AdWords bot be (im)polite and reviewing our privacy page.
So The Quality Score Robot Still See You?

So far, this is the perfect setup and plays nice with the organic side of things. But that got us thinking. Is there really a need for a privacy policy? While the use of Google Analytics actually requires you to display a privacy policy, we thought we would tempt fate. We decided to see how powerful a credibility indicator the privacy policy actually was. Last week we added to following lines to our robots.txt file:

User-agent: AdsBot-Google
Disallow: /privacy-policy/
Disallow: /w3c/policy.html

Two days later, most of our campaigns keywords went to either “OK” or “POOR”. Immediately after discovering this we removed the lines from the robots.txt. The next day things returned to normal. While it is possible that there were other factors at play here (the test was far from lab tested) I think it is fair to assume that a privacy policy carries a significant weight when it comes to quality score calculation. We also performed this “test” by excluding ALL PPC landing pages with similar or worse results. Our Google rep explained:

We feel that a non-participating advertiser does detract from the user’s
search experience and from the overall quality of the AdWords programme.
While you can exclude your pages from review, this tells us little about
the quality and relevance of your page. Therefore, if you restrict AdWords
from visiting your pages, you will experience a drop in Quality Scores for
your related keywords. (This will cause higher minimum bid requirements
for any landing page for which you have restricted access.)

In Conclusion.

We got exactly what we wanted from this little test. We wanted to know if excluding pages from the main index would effect our quality scores. The answer is no. Everything is now back to normal, our quality scores are “Great” across the board and our organic rankings have far from suffered. I hope that has answered that question for all that have called and mailed in to ask.


  1. avatar
    markus941 Says:
    July 16th, 2007 at 5:35 pm

    Nice insight. Thanks for writing this up. Sphunn.

    Reply

  2. avatar
    Dave Davis Says:
    July 17th, 2007 at 5:08 pm

    THanks Markus and thanks for the Sphinn :)

    Nice not to have to worry if your content appeals or not to the SEO lynch mob!

    Reply

  3. avatar
    Lar Says:
    July 20th, 2007 at 2:24 pm

    Hi Dave,
    on an algorithm-related subject, your namesake (almost) has published a post on EVOLT. Not sure if you’ve noticed anything
    PS Love the Google Global Add-on. Was doing some non-SEO related work today which it really came in handy for.
    Lar

    Reply

  4. avatar
    Dave Davis Says:
    July 20th, 2007 at 2:33 pm

    Hi Lar, thanks for stopping bye. I had a read over the article (More and more people are confusing us. Of all the obscure names in all the obscure specialized markets!) and to be honest, I have not noticed anything major with our clients. I will however note that a client of ours who was suffering the “-950 penalty” has suddenly been green flagged.

    I do not think this is a coincidence that it happened days before webmaster central released the webmaster message center. As far as predictions go, I do not think Dave is in a position to accurately predict what he thinks will happen with any algorithm.

    I do agree however that domain age is being given a lot more weight lately.

    Also, thanks for your comments on the plugin. It is far from perfect, but I think it adds some value. We are already testing the new version with some pretty nifty city targeting and ZIP code targeting.

    Reply

  5. avatar
    john blankey - domain name information Says:
    July 22nd, 2007 at 9:02 pm

    This is an interesting take on adwords. nice that you guys tested out this little trick. So what your saying is to include a privacy policy but have it blocked by a robots.txt command. the adwords bot will ignore the command saving link popularity while still getting a great quality score from adwords. User of the adsense program should not exclude any pages for that bot. Other wise you will get a bad quality score.

    Reply

  6. avatar
    Amit Says:
    September 10th, 2007 at 11:59 am

    Hey Dave another great article! But I have a question. You said that you blocked the privacy policy page from the Google ad bot and it resulted in poor to OK quality scores, right?
    But then you again removed the exclusion and allowed the ad bot to visit your privacy policy page.
    Now I want to know how does Google ad bot know that you’ve made changes to your website? Does it visit everyday to your pages? I think there was a thread going at both DigitalPoint and WebmasterWorld about how often the ad bot visit the landing pages and websites advertised!
    Thanks in advance!

    Reply

  7. avatar
    Dave Davis Says:
    September 12th, 2007 at 4:29 pm

    Hi Amit, that is a very good and common question. In fact, I have a blog post written about it that I will publish shortly. In short, you can create a new ad or landing page URL and the quality score bot HAS to come by and visit again.

    Reply

  8. avatar
    Web Design Glagow Says:
    November 10th, 2008 at 3:17 pm

    Hi Dave, I’m enjoying your articles on PPC. Do you have any ideas as to how Google determines whether you have a privacy policy or not? I ask because it’s not guaranteed the site will link to the page with the text ‘Privacy Policy’, or use any standard format for the page – I wouldn’t want to miss out on the benefit because I didn’t include a meta tag or some such!

    Cheers

    Reply

  9. avatar
    Estate internet marketing Says:
    December 2nd, 2009 at 11:12 am

    Nice test! A month ago I noticed that on a site (pharmaceutical) was no privacy policy included. We started advertising, and the scores rapidly decreased. After a privacy policy was included the scores improved over night.

    Reply