Magento Varnish

Varnish is an amazingly fast reverse proxy server. It operates in front of a Magento shop and caches frequently retrieved pages such as the frontpage and productpages. For selected pages, this results in 2 to 20 times (!!) faster loading times.

Concepts

How much improvement will I see for cached pages?

The more complex ("dyanamic") a page is, the longer it takes for the server to compute its content. Magento is an example of an extremely complex application. Pages generated by Magento take relatively long, even on a very fast server. In our daily work we see that uncached Magento pages take generally between 2 and 10 seconds.

Now there are several ways to improve this. These methods yield better response times:

• 0.5 to 1.5 seconds when internal block caching is in use
• 0.2 to 0.6 seconds when a full page cache (FPC) extension is in use (such as Lightspeed or the FPC included with Magento Enterprise).
• 0.01 to 0.03 seconds when the page is fetched from Varnish cache.

The speed increase is incredible. In the best case, the time to first byte is decreased by a thousand fold. However, the loading time of a webpage depends on more than just server calculation time, such as media and javascript retrieval and rendering. Therefore, to an end user, the perceived speed gain for Varnish cached pages will be 2 to 20 times, still quite remarkable!

How does it work?

Caching only works for static content (that does not change very often). If you would cache dynamic content, your users would be served outdated pages.

In a standard setup, Magento is unable to have Varnish cache its output. Content generated by Magento is explicitly marked as "dynamic". This assumption is not always true, in fact most Magento pages are static. The solution is to make Magento mark some output as "cacheable". This is achieved by a clever Magento extension that was developed by Byte and is available to our customers.

What part of Magento can be cached?

Not everything is cacheable. Items like shopping cart and login status are always dynamic and can't be easily cached. Our Byte extension doesn't cache by default to prevent false positives, and only marks pages cacheable that are explicitly marked as such. At the moment (changes my apply), the following types of pages are cached:

• Product pages
• Category pages
• Blog pages
• CMS pages

When a visitor has started a valid session, for example when something has been added to a shopping cart, or is selecting products to compare, caching is always disabled.

Some Magento extensions produce dynamic output, such as a "random product" on the frontpage. If the random product is picked on the server, it can't be cached. Instead, the random selection should be implemented by javascript in the browser.

Are there other advantages?

1. A faster site will increase the conversion and revenue of your shop (see for example this research and this blogpost).
2. Google likes faster sites, so your shop will end up higher in the search results (Google announcement).
3. If pageviews are handled more efficiently, your webshop will be more resilient towards traffic spikes. The chance that your shop becomes unresponsive because of a sudden traffic surges is decreased. So you will be able to serve all your customers during this prime time television ad!
4. Site visitors that are handled by the Varnish cache, are near-free. Because Varnish can handle all visitors that have not yet selected anything to purchase, your hosting costs will scale linearly with the number of real, purchasing customers. So it is essentially free to attract large amounts of potential customers to your site, because a low conversion rate will not bite your wallet.

What hosting plans @ Byte do support Varnish?

At the moment, we only support Varnish for dedicated server and clusters. This is due to technical limitations. Varnish for shared customers is on the roadmap, but due to its complexity we cannot give an ETA on this. Also, Varnish for shared customers will likely be less flexible.

Who is eligible to participate in the pilot?

At the moment, we run a pilot with selected customers whose site characteristics match our technical requirements. Unfortunately, we already have enough participants for the pilot, so we have to decline any new requests.

Technical

How do I enable my shop to use Varnish?

If you have been invited for the private pilot, a Varnish server will be installed for you and you will be given the Byte specific Magento extension. After installation, Varnish will know what to cache and what not. Try it out by browsing your site. The first visit to a page will be slow, but you will notice quite a difference for the second visit and onward. To measure it, you can use the developer tools in Chrome (F12, network tab) or the Firebug extension for Firefox.

What happens when someone edits a product in the Magento backend?

If you have installed the Byte Varnish extension, then it will automatically notify the right Varnish server to invalidate the relevant pages from the cache.

There is an exception to this rule, where you are using an external module that outputs content about products. Because the Varnish extension doesn't know what URLs belong to this module, it cannot tell the Varnish server what URLs to invalidate. So you have to do this yourself (either by pressing control-shift-F5 or programmatically).

How large is the cache and what happens when it's full?

At the moment, we are experimenting with various cache sizes to determine its optimal size. When it's full, it uses the Least Recently Used (LRU) algorithm to purge the least relevant items from the cache.

How do I know when the cache memory is full?

TODO

How long are items cached?

Static media (images, javascript and such) are cached for 1 month, all other content expires after a week.

How can I control the expiry time?

Magento product pages and category pages are cached for a week. This is a fixed period. They are purged on edit, right from the backend. These do not need manual refreshing.

Static resources like images and CSS are cached for 30 days by default, but you can control the TTL from .htaccess using Expires headers. These caching headers are respected.

My new stylesheet / image / javascript does not show!

You can refresh the resource forceably by:

• Refreshing the page containing the resource by pressing ctrl-shift-r or ctrl-F5.
• Opening the one resource on a browser tab and then pressing ctrl-shift-r or ctrl-F5

How can I automatically refresh items more often?

If your server is relatively idle during the night, you could use this spare capacity to refresh the most used parts of your site (generally, the frontpage and its 1 level deep links). A handy utility is wget and runs on Linux, MacOSX and our shell server. It crawls your site automatically. A sample command line that you can put in a nightly cron:

wget --delete-after \
     --no-directories \
     --level 1 \
     --page-requisites \
     --recursive \
     --server-response \
     --no-cache \
     http://www.YOURSITE.nl 2>&1 | egrep -i '(^--|x-cache:)'

If you want the crawler to reach more pages, increase the level. Warning: if you have many products (500+) or multiple storefronts, this will take a long time AND it will deplete your cache memory. So it is advisable to only refresh the "busiest" parts of your site.

Does it help to use other caching mechanisms, along with Varnish?

This depends on the mechanism:

• We would advice to NOT USE full page cache systems, such as Lightspeed, Fooman Speedster and the one included in Magento Enterprise. Those FPC systems have the same caching scope as Varnish, and do it less efficiently. More importantly, you will have multiple caching layers, which makes it hard to debug.
• Block cache, memcache, Byte Smartcache: please use this, as it speeds up pages that are uncacheable by Varnish (for example, when somebody has logged in).

How can I refresh a page?

In any browser, you can press control-shift-F5 or ctrl-shift-r and Varnish will serve you a fresh page.

TODO: does Magento with block/fullpage cache enabled serve a fresh page as well? This depends on your setup, but probably not.

How can I purge everything?

We wouldn't recommend that, as rebuilding your cache takes a lot of time, especially if you have many storefronts and/or products. But if you are absolutely sure, you can purge the complete Varnish cache from within Magento using the "Clear Cache Storage" button. Note that this will not rebuild your cache, so after this action, your visitors will experience slow pages. It is advisable to automatically rebuild the most popular part of your site, see the "wget" example elsewhere on this page.

What cache ratio will I achieve?

On production sites we've see cache hit ratio's from 70-95%.

What happened to my cookies?

Varnish doesn't cache anything if the browser sends cookies, because in that case it will assume a dynamic session. So we configured Varnish to strip all cookies from incoming HTTP requests, unless a shop or login session is in effect. This might be a problem if you have a module that requires cookies from any visitor, because it will render your complete shop uncacheable. Solution: adjust the module so that it will only send cookies for shop sessions.

How can I see if a page is cached?

You can tell by watching the HTTP headers. We like the Chrome developer tools (under the F12 key, Network tab). Uncached server response headers will look like this:

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection: close
Date: Tue, 07 Feb 2012 16:51:26 GMT
Pragma: no-cache
Via: 1.1 varnish
Age: 0
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html; charset=UTF-8
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Client-Date: Tue, 07 Feb 2012 16:51:28 GMT
Client-Peer: 82.94.56.105:80
Client-Response-Num: 1
X-Cache: MISS
X-Cacheable: NO
X-Cacheable-Reason: The application indicates that the page should not be cached.
X-Varnish: 1770497029

A cached server response will have these extra lines:

Age: 11988
X-Cache: HIT (146)
X-Cacheable: YES
X-Cacheable-Reason: The application idicates that the page should be cached.

Age is the age (in seconds) of the cached version of the page.

How can I disable Varnish?

You can disable the Magento module to disable the Varnish functionality. TODO: how?

How do I cache a page that displays a random item?

If the random item is selected on the server, it will not be random anymore after it is cached ;) The solution here is to make the randomness client side using javascript. For example, you could have the server send an array of SKUs in JSON, then let javascript pick a random item to display.

What about SSL?

At the moment, our Varnish servers don't cache HTTPS URLs. For most shops, this doesn't matter, because HTTPS is only engaged when a customer wants to checkout, and that is dynamic anyway. We are working on a solution for those shops that run https-only.

How do I make my Magento module Varnish-compatible?

TODO

What happened to the access logs?

For performance reasons, pageviews that are handled by Varnish (using its cache) are not recorded into your regular access logs. If you want to see statistics of all your visitors (you probably do) we strongly advise you to implement Google Analytics.

De huidige waardering is: 66/100 (13 stemmen)

 You need to enable JavaScript to vote