After creating a truly huge site recently, we were faced with the problem of getting all the pages cached as quickly as possible. As many of you will know, Google will cache pages that are the least amount of clicks away from your most important pages (e.g your home page) first. After that though, the search engine is likely to take its time about caching the rest for inclusion in the SERPs.

Unfortunately for very large sites, getting these pages cached could take a matter of years! This of course is way too long to see the results of your hard work. So how can you get pages into the SERPs as quickly as possible when your site is massive?

We’ve tried most things and recently we tried to see if certain google bots are faster at crawling pages than the standard bot. Matt Cutts mentioned in 2006 that to save bandwidth, if one bot crawls a page other google bots (such as the adsense spider and the google news bot) can use those cached results rather than crawling the page again.

Before the introduction of the adsense code, we would be typically getting a few thousand pages cached every few days, but after it was added, around 60,000 extra pages have been cached in just 4 days. Of course, it’s early days to say whether this experiment is the result of the adsense code being on the page or whether it is other changes that we have made recently, but it definitely invites further scrutiny.

Bookmark It

Hide Sites