Unfortunately for very large sites, getting these pages cached could take a matter of years! This of course is way too long to see the results of your hard work. So how can you get pages into the SERPs as quickly as possible when your site is massive?

We’ve tried most things and recently we tried to see if certain google bots are faster at crawling pages than the standard bot. Matt Cutts mentioned in 2006 that to save bandwidth, if one bot crawls a page other google bots (such as the adsense spider and the google news bot) can use those cached results rather than crawling the page again.

Before the introduction of the adsense code, we would be typically getting a few thousand pages cached every few days, but after it was added, around 60,000 extra pages have been cached in just 4 days. Of course, it’s early days to say whether this experiment is the result of the adsense code being on the page or whether it is other changes that we have made recently, but it definitely invites further scrutiny.

Bookmark It

Hide Sites

Of course this is inefficient but we can solve this problem by helping google and other search engines to find our content with an XML sitemap. An XML sitemap can tell a search engine more information about a page such when it was last modified, how important the page is as well as how often it is updated. By providing this information to the search engines, they can crawl your site more efficiently, neither wasting their resources or yours.

Creating an XML sitemap is extremely simple. Here is an example:

<?xml version='1.0' encoding='UTF-8'?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9
                            http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd">
    <url>
        <loc>http://www.redsauce.com/</loc>
        <lastmod>2009-11-04</lastmod>
        <changefreq>daily</changefreq>
        <priority>1</priority>
    </url>
</urlset>

For each individual page, you can specify the URL between the <loc></loc> tags, when the page was last updated between the <lastmod></lastmod> tags, how often the page is updated between the <changefreq></changefreq> tags and how important the page is between the <priority></priority> tags.

The priority of a page can be between 0.1 and 1; 1 being your most important page, e.g. your home page and the deeper you go into your site the less the priority score should be. By providing all this information, search engines can crawl your pages more intelligently without wasting your server’s resources.

Once you have finished creating your XML file, save it to the root of your server and name it sitemap.xml for example. You can let the search engines know it exists by adding this line to a robots.txt file (placed at the root of your domain):

Sitemap: http://www.redsauce.com/sitemap.xml

or by notifying google in webmaster tools about the location of your XML sitemap file.

An XML sitemap file can contain a maximum of 50,000 URLs and cannot exceed a maximum of 10MB file size. You can compress XML sitemap files to save your bandwidth but still mustn’t exceed 10MB once uncompressed.

You can write a script to create your sitemap from the pages on your site, or if your site doesn’t contain too many pages, you can use an online service such as xml-sitemaps.com to generate the sitemap file for you.

Bookmark It

Hide Sites