The Problem

When working on a site for a customer under a development domain I noticed that the ‘Last 5 Search Terms’ and the ‘Top 5 Search Terms’ entry on the dashboard had thousands of hits.

SearchStatsPic

That was impossible because the site wasn’t in production and I had only done a few catalog searches myself. So after a little research, it turned out that these large numbers were generated because the Google bot had been indexing the ‘Search Terms’ page in the footer of the website. The Search Terms page displays a tag cloud of the most popular search terms. Following a link in the tag cloud is the equivalent of issuing an internal store search. The Google bot and other search engine crawlers have been following links in the tag cloud repeatedly (maybe once for every page in the website) and so the popular search term numbers went up.

This is a problem, because it taints the internal search metrics for the store which are really important for analytics and marketing.

The Solution

The easiest fix is to tell the search bots not to index the Search Terms page. Add the following line to the robots.txt file found in the root of your magento app.

Disallow: /catalogsearch/term/

If you don’t have a robots.txt file there, create one and add the following lines. Note: Normally you would want a bunch of other entries in robots.txt, to make the indexing of your store more efficient and to avoid duplicate content, but that’s another post waiting to be written.

User-agent: *
Disallow: /catalogsearch/term/
Sitemap: http://www.domain.co.uk/sitemap.xml

To reset the search term count go to Catalog > Search and delete all the entries.

This article has 5 comments

  1. Matt

    If you block all URLs that contain a question mark using:

    Disallow: /*?

    You will prevent the indexing of the catalogsearch pages and also links from the layered navigation which can produce duplicate content.

  2. Chris

    I added this to my site, but i am still getting high levels on one term ? Also /catalogsearch/term/ doesn’t actually lead to anywhere on the site – is this correct? Or do you mean to actually put in the search term where you have /term/ ?

  3. Tim

    Chris and Justin:

    It’s possible that Magento changed the URL. I’m trying this:

    Disallow: /catalogsearch/result/

Leave a Reply

Your email address will not be published. Required fields are marked *