HTTP Caching

By:    Updated: March 1,2017

Fetching something over the network is both slow and expensive: large responses require many roundtrips between the client and server, which delays when they are available and can be processed by the browser, and also incurs data costs for the visitor. As a result, the ability to cache and reuse previously fetched resources is a critical aspect of optimizing for performance. Great news, every browser ships with an implementation of an HTTP cache! All we have to do is ensure that each server response provides correct HTTP header directives to instruct the browser on when and for how long the response can be cached by the browser.

What is HTTP Caching

An HTTP cache is a local store of response messages and the subsystem that controls storage, retrieval, and deletion of messages in it. A cache stores cacheable responses in order to reduce the response time and network bandwidth consumption on future, equivalent requests.

Cache-Control

The "Cache-Control" header field is used to specify directives for caches along the request/response chain. Such cache directives are unidirectional in that the presence of a directive in a request does not imply that the same directive is to be given in the response.

  • Each resource can define its caching policy via Cache-Control HTTP header
  • Cache-Control directives control who can cache the response, under which conditions, and for how long.

The best request is a request that does not need to communicate with the server: a local copy of the response allows us to eliminate all network latency and avoid data charges for the data transfer. To achieve this, the HTTP specification allows the server to return a number of different Cache-Control directives that control how, and for how long, the individual response can be cached by the browser and other intermediate caches.

Cache-Control = "Cache-Control" ":" 1#cache-directive

cache-directive = cache-request-directive | cache-response-directive

cache-request-directive = "no-cache"
                        | "no-store"
                        | "max-age" "=" delta-seconds
                        | "max-stale" [ "=" delta-seconds ]
                        | "min-fresh" "=" delta-seconds
                        | "no-transform"
                        | "only-if-cached"
                        | cache-extension

cache-response-directive = "public"
                         | "private" [ "=" <"> 1#field-name <"> ]
                         | "no-cache" [ "=" <"> 1#field-name <"> ]
                         | "no-store"
                         | "no-transform"
                         | "must-revalidate"
                         | "proxy-revalidate"
                         | "max-age" "=" delta-seconds
                         | "s-maxage" "=" delta-seconds
                         | cache-extension

cache-extension = token [ "=" ( token | quoted-string ) ]

HTTP Caching

Backwards compatibility with HTTP/1.0

HTTP/1.0 caches might not implement Cache-Control and might only implement "Pragma: no-cache" and "Expires: 0". Clients SHOULD include both header fields ("Pragma: no-cache" and "Cache-Control: no-cache") when a no-cache request is sent to a server not known to be HTTP/1.1 compliant. In HTTP/1.1, When the Cache-Control header field is also present and understood in a request, Pragma is ignored.

 

"no-cache" VS "no-store"

"no-cache" indicates that the returned response cannot be used to satisfy a subsequent request to the same URL without first checking with the server if the response has changed. As a result, if a proper validation token (ETag) is present, no-cache will incur a roundtrip to validate the cached response, but can eliminate the download if the resource has not changed.

By contrast, "no-store" is much simpler, as it simply disallows the browser and all intermediate caches to store any version of the returned response - e.g. one containing private personal or banking data. Everytime the user requests this asset, a request is sent to the server and a full response is downloaded each and every time.

 

"public" VS "private"

If the response is marked as "public" then it can be cached, even if it has HTTP authentication associated with it, and even when the response status code isn't normally cacheable. Most of the time, "public" isn't necessary, because explicit caching information (like "max-age") indicates that the response is cacheable anyway.

By contrast, "private" responses can be cached by the browser but are typically intended for a single user and hence are not allowed to be cached by any intermediate cache - e.g. an HTML page with private user information can be cached by that user's browser, but not by a CDN.

Calculating Freshness Lifetime

A cache can calculate the freshness lifetime of a response by using the first match of the following:

  • If the cache is shared and the s-maxage response directive is present, use its value, or
  • If the max-age response directive is present, use its value, or
  • If the Expires response header field is present, use its value minus the value of the Date response header field, or
  • Otherwise, no explicit expiration time is present in the response. A heuristic freshness lifetime might be applicable.
if exists Cache-Control: s-maxage
{
    freshness_lifetime = s-maxage
}
else if exists Cache-Control: max-age
{
    freshness_lifetime = max-age        
}
else if exists Expires and Date
{
    freshness_lifetime = expires_value - date_value
}
else if none of above is present, but have a Last-Modified (Using heuristic freshness lifetime)
{
    heuristic_freshness_lifetime <= (Date - Last-Modified)/10
    freshness_lifetime = heuristic_freshness_lifetime
}

Validating cached responses with ETags

What is ETag?

The "ETag" header field in a response provides the current entity-tag for the selected representation, as determined at the conclusion of handling the request. An entity-tag is an opaque validator for differentiating between multiple representations of the same resource, regardless of whether those multiple representations are due to resource state changes over time, content negotiation resulting in multiple representations being valid at the same time, or both.

  • Validation token is communicated by the server via the ETag HTTP header;
  • Validation token enables efficient resource update checks: no data transfer if the resource has not changed.

HTTP Caching Etag

Let's assume 30 seconds have passed since our initial fetch and the browser has initiated a new request for the same resource. First, the browser checks the local cache and finds the previous response, unfortunately it cannot use it as the response has now "expired". At this point it could simply dispatch a new request and fetch the new full response, but that's inefficient because if the resource has not changed then there is no reason to download the exact same bytes that are already in cache!

 

That's the problem that validation tokens, as specified in the ETag header, are designed to solve: the server generates and returns an arbitrary token which is typically a hash or some other fingerprint of the contents of the file. The client does not need to know how the fingerprint is generated, it only needs to send it to the server on the next request: if the fingerprint is still the same then the resource has not changed and we can skip the download.

HTTP Caching Etag

In above example the client automatically provides the ETag token within the "If-None-Match" HTTP request header, the server checks the token against the current resource, and if it has not changed returns a "304 Not Modified" response which tells the browser that the response it has in cache has not changed and can be renewed for another 30 seconds. Note that we do not have to download the response once more - this saves time and bandwidth.

As a web developer, how do you take advantage of efficient revalidation? The browser does all the work on our behalf: it will automatically detect if a validation token has been previously specified, it will append it to an outgoing request, and it will update the cache timestamps as necessary based on received response from the server. The only thing that's left for us to do is to ensure that the server is, in fact, providing the necessary ETag tokens: check your server documentation for necessary configuration flags.

Cache Validation

When a cache has one or more stored responses for a requested URI, but cannot serve any of them (e.g., because they are not fresh, or one cannot be selected), it can use the conditional request mechanism in the forwarded request to give the next inbound server an opportunity to select a valid stored response to use, updating the stored metadata in the process, or to replace the stored response(s) with a new response. When sending a conditional request for cache validation, a cache sends one or more precondition header fields containing validator metadata from its stored response(s), which is then compared by recipients to determine whether a stored response is equivalent to a current representation of the resource.

 

One such validator is the timestamp given in a Last-Modified header field which can be used in an If-Modified-Since header field for response validation, or in an If-Unmodified-Since or If-Range header field for representation selection (i.e., the client is referring specifically to a previously obtained representation with that timestamp).

Another validator is the entity-tag given in an ETag header field. One or more entity-tags, indicating one or more stored responses, can be used in an If-None-Match header field for response validation, or in an If-Match or If-Range header field for representation selection (i.e., the client is referring specifically to one or more previously obtained representations with the listed entity-tags).

server response client request
Last-Modified If-Modified-Since or If-Unmodified-Since or If-Range
Etag If-None-Match or If-Match or If-Range

How to invalidate and update cached responses

All HTTP requests made by the browser are first routed to the browser cache to check if there is a valid cached response that can be used to fulfill the request. If there is a match, the response is read from the cache and we eliminate both the network latency and the data costs incurred by the transfer. However, what if we want to update or invalidate a cached response?

 

For example, let's say we've told our visitors to cache a CSS stylesheet for up to 24 hours (max-age=86400), but our designer has just committed an update that we would like to make available to all users. How do we notify all the visitors with what is now a "stale" cached copy of our CSS to update their caches? It's a trick question - we can't, at least not without changing the URL of the resource. Once the response is cached by the browser, the cached version will be used until it is no longer fresh, as determined by max-age or expires, or until it is evicted from cache for some other reason - e.g. the user clearing their browser cache. As a result, different users might end up using different versions of the file when the page is constructed; users who just fetched the resource will use the new version, while users who cached an earlier (but still valid) copy will use an older version of its response.

 

So, how do we get the best of both worlds: client-side caching and quick updates? Simple, we can change the URL of the resource and force the user to download the new response whenever its content changes. Typically, this is done by embedding a fingerprint of the file, or a version number, in its filename - e.g. style.20170301.css.

 

The ability to define per-resource caching policies allows us to define "cache hierarchies" that allow us to control not only how long each is cached for, but also how quickly new versions are seen by visitor:

  • The HTML is marked with "no-cache", which means that the browser will always revalidate the document on each request and fetch the latest version if the contents change. Also, within the HTML markup we embed fingerprints in the URLs for CSS and JavaScript assets: if the contents of those files change, than the HTML of the page will change as well and new copy of the HTML response will be downloaded.
  • The CSS is allowed to be cached by browsers and intermediate caches (e.g. a CDN), and is set to expire in 1 year. Note that we can use the "far future expires" of 1 year safely because we embed the file fingerprint its filename: if the CSS is updated, the URL will change as well.
  • The JavaScript is also set to expire in 1 year, but is marked as private, perhaps because it contains some private user data that the CDN shouldn't cache.
  • The image is cached without a version or unique fingerprint and is set to expire in 1 day.

The combination of ETag, Cache-Control, and unique URLs allows us to deliver the best of all worlds: long-lived expiry times, control over where the response can be cached, and on-demand updates.

  • Locally cached responses are used until the resource 'expires'
  • Embedding a file content fingerprint in the URL enables us to force the client to update to a new version of the response (or appended query strings to the end of URL, the query strings can be file content fingerprint ,or current timestamp,or others, e.g. jquery.min.201703012100.js or jquery.min.js?v=201703012100)
  • Each application needs to define its own cache hierarchy for optimal performance
More in Development Center
New on Valinv
Sources
  • https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching
  • http://tools.ietf.org/html/rfc2616
  • http://tools.ietf.org/html/rfc7234
Related Articles
Sponsored Links