Does the Adsense crawler speed up page caching? December 23, 2009 No Comments
After creating a truly huge site recently, we were faced with the problem of getting all the pages cached as quickly as possible. As many of you will know, Google will cache pages that are the least amount of clicks away from your most important pages (e.g your home page) first. After that though, the search engine is likely to take its time about caching the rest for inclusion in the SERPs.
XML Sitemap Basics – How To Get Your Site Crawled More Efficiently November 4, 2009 No Comments
As a developer at redsauce.com, I am always looking for the most efficient way of doing things, whether it be a PHP script or a database schema. Working on a truly huge site recently, it became obvious that google’s crawling was causing performance issues with the server the site was hosted on. If google doesn’t already know what pages your site contains (i.e. if its a new site), the google bot will have to go through every link on your site, possibly visiting certain pages several times.
