You can increase the loading speed of your Magento webshop using Yireo Search Engine Page Cache extension. Search engines and some other sites use web crawling or spidering to update their web content or indexes of your web content. Using your sitemap the crawlers are informed wich pages they have to visit and when to return to that page. This can result in a heavy load on your server when they visit and generate these pages. Blocking search engines to your site to prevent heavy load is out of the question. Using Yireo Search Engine Page Cache the search engines are allowed to crawl your site while preventing high load of the server at the same time. Read on to learn how you can accomplish this ideal situation.
Search engines just read fast content
When a search engine crawls your website it will follow your provided sitemap and follows the web links on every page it’s allowed to follow. They don’t add a product to a cart nor will it add a product to a wish list. Search engines don’t mind the functionality of your website. They just visits your website to read your content.
Since search engines don’t care much about dynamic content you should provide them with content as quick as possible. Which means…. provide them with a page from cache if available. And that’s where Yireo Search Engine Page Cache comes in handy. According to their website this extension allows full-page caching only for search engines.
Using this extension the search bots get served with cached pages if available. Plain HTML output directly from cache. A positive side-effect is that these pages are loaded very fast resulting in a positive impact on your SEO-ranking. Search engines tend to like fast websites.
About the extension
The Yireo extension Search Engine Page Cache is an open source commercial extension and can be bought from their website yireo.com. (http://www.yireo.com/software/magento-extensions/search-engine-page-cache) After purchase you can download the compressed package-file and upload the extracted files to your Magento root. Refresh the Magento cache and you’re done. Using the configuration in Magento backend you can deactivate / activate the extension. By default Yireo added the some User Agents to receive a cached page. A complete list can be found on there website. You can add more user agents within Magento configuration. And last but not least you can set the cache lifetime (default set to 1 day).
Reduce visits search bot using robots.txt
On top of this extension you can add a crawl delay to your robots.txt. Crawl delay is a special directive recognised by Yahoo, Live Search, and Ask that instructs a crawler on the number of seconds to wait between crawling pages. Set the crawl-delay somewhere below 10 seconds.
- Google does not support the crawl delay command directly, but you can lower your crawl priority inside Google Webmaster Central (Configuration >> Settings >> Set crawl rate).
- Bing does support the crawl delay in your robots.txt and on top of that you can set a time interval inside Bing Webmaster (Configure my site >> Crawl Control).