Unnecessary HTTP headers

HTTP headers are important to control how caches and browsers handle your content. But many of them are used incorrectly or meaninglessly, spending extra resources at the critical moment of loading the page, and they may not work the way you think. In a series of articles on best practices, we first look at unnecessary headlines.

Most developers are aware of important and relevant HTTP headers. The most famous are Content-Type and Content-Length , which are almost universal headers. But recently, headers such as Content-Security-Policy and Strict-Transport-Security have begun to be used to increase security, and for better performance - Link rel=preload . Despite widespread support in browsers, few use them.

At the same time, there are many extremely popular headlines that are not at all new and not very useful. We can prove it with the help of HTTP Archive , a project running by Google and sponsored by Fastly, which every month using WebPageTest downloads 500,000 sites and uploads the results to BigQuery .

Here are the 30 most popular response headers for the HTTP Archive data (based on the number of domains in the archive that each header gives), and an approximate assessment of the usefulness of each:

Headline	Requests	Domains	Status
date	48779277	535621	Required by protocol
content-type	47185627	533636	Usually required by browser
server	43057807	519663	Not required
content-length	42388435	519118	Useful
last-modified	34424562	480294	Useful
cache-control	36490878	412943	Useful
etag	23620444	412370	Useful
content-encoding	16194121	409159	Required for compressed content
expires	29869228	360311	Not required
x-powered-by	4883204	211409	Not required
pragma	7641647	188784	Not required
x-frame-options	3670032	105846	Not required
access-control-allow-origin	11335681	103596	Useful
x-content-type-options	11071560	94590	Useful
link	1212329	87475	Useful
age	7401415	59242	Useful
x-cache	5275343	56889	Not required
x-xss-protection	9773906	51810	Useful
strict-transport-security	4259121	51283	Useful
via	4020117	47102	Not required
p3p	8282840	44308	Not required
expect-ct	2685280	40465	Useful
content-language	334081	37927	Controversial
x-aspnet-version	676128	33473	Not required
access-control-allow-credentials	2804382	30346	Useful
x-robots-tag	179177	24911	Irrelevant to browsers
x-ua-compatible	489056	24811	Not required
access-control-allow-methods	1626129	20791	Useful
access-control-allow-headers	1205735	19120	Useful

Let's look at the optional headers, why we do not need them and what to do with it.

Vanity (server, x-powered-by, via)

You can be very proud of your choice of server software, but most people don't care. In the worst case, these headers may disclose sensitive data, which simplifies the attack on your site.

Server: apache
X-Powered-By: PHP/5.1.1
Via: 1.1 varnish, 1.1 squid

RFC7231 allows the Server header to be included in the response of the server, indicating specific software on the server that is used to issue content. Most often it is a string like "apache" or "nginx". Although the title is allowed, it is optional and not particularly valuable for developers or end users. However, today it is the third most popular HTTP server header on the Internet.

X-Powered-By - the most popular header from those that are not defined by any standard, and it has a similar goal: it usually indicates the application platform on which the server is running. Top replies include "ASP.net", "PHP" and "Express". Again, it does not carry any tangible benefits and simply takes its place.

Perhaps more controversial is Via . He is obliged ( by RFC7230 ) to add to the request each proxy through which the request passes - to identify the proxy. This may be the proxy host name, but more often a common identifier like "vegur", "varnish" or "squid". Deleting (or not adding) such a header may result in a proxy redirect cycle . But it is interesting that it is also added in response to the way back to the browser - and here it performs just an informational function: no browsers do anything with it, so it’s safe enough to get rid of it if you want.

Outdated standards (P3P, Expires, X-Frame-Options, X-UA-Compatible)

Another category of headings are those that actually cause an effect in the browser, but (already) are not the best way to achieve this effect.

P3P: cp="this is not a p3p policy"
Expires: Thu, 01 Dec 1994 16:00:00 GMT
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=edge

P3P is a fun little thing. I had no idea what it was. Even more amusing, one of the most common P3P header content is “This is not a P3P rule.” Well, is this it or not?

Here the story goes back to trying to standardize machine-readable privacy rules . There was disagreement about how to display data in browsers, and only one browser implemented support for this header - Internet Explorer. But even in it, P3P had no visual effect for the user; he just had to be present to allow access to third-party cookies in frames. Some sites have even established non-compliance rules for P3P, as in the example above, although it is very doubtful to do so.

Needless to say, reading third-party cookies is generally a bad idea, so if you don’t, then you don’t need to set the P3P header!

Expires is incredibly popular given that Cache-Control has had an advantage over Expires for 20 years. If the Cache-Control header contains the max-age directive, then any Expires header in the same response is ignored. But a huge number of sites establishes both of these headers, and Expires most often put on the date Thu, 01 Dec 1994 16: 00: 00 GMT , so that the content is not cached. Of course, because the easiest way to copy-date is from specifications .



But there is simply no need to do this. If you have an Expires header with a date from the past, simply replace it with:

Cache-Control: no-store, private

( no-store is too strict a directive not to write content to persistent storage, so you may prefer no-cache for better performance, for example, to navigate backward / forward or resume sleeping tabs in the browser)

Some site validation tools will advise you to add an X-Frame-Options header with a value of 'SAMEORIGIN'. He tells browsers that you refuse to send content to a frame on another site: as a rule, this is a good protection against clickjacking . But the same effect can be achieved with another heading with more consistent support and more reliable behavior:

Content-Security-Policy: frame-ancestors 'self'

There is an additional benefit here, because you still have to give the CSP header for other reasons (more about them later). Probably, in our time, you can do without X-Frame-Options .

Finally, back in IE9, Microsoft introduced the “Compatibility Mode”, which displayed the page using IE8 or IE7. Even in normal mode, the browser thought that an earlier version of the engine might be needed for proper rendering. These heuristics did not always work correctly, and developers could override them using the X-UA-Compatible title or meta tag. In fact, it has become a standard part of many frameworks like Bootstrap. Now this title is almost useless: the proportion of browsers that understand it is very small. And if you actively support the site, it is very unlikely that it uses technologies that will launch compatibility mode.

Debugging Information (X-ASPNet-Version, X-Cache)

In some ways, it is surprising that some of the most popular headlines are not mentioned at all in any standard. In essence, this means that in some way, thousands of websites suddenly agreed to use a certain title in a certain way.

X-Cache: HIT
X-Request-ID: 45a336c7-1bd5-4a06-9647-c5aab6d5facf
X-ASPNet-Version: 3.2.32
X-AMZN-RequestID: 0d6e39e2-4ecb-11e8-9c2d-fa7ae01bbebc

In fact, these "unknown" headers are not invented by the developers. Usually these are artifacts of using certain server-side frameworks, software, or services of specific vendors (for example, the last header is typical of AWS).

The X-Cache header adds Fastly (and other CDNs), along with other specific headers, such as X-Cache-Hits and X-Served-By . When debugging is enabled, more are added, for example, Fastly-Debug-Path and Fastly-Debug-TTL .

These headers are not recognized by any browser, and their removal does not affect the display of pages. But since they can provide you, the developer, with useful information, you can save them.

Misunderstandings (Pragma)

I did not expect that in 2018 I would have to mention the Pragma header, but according to the HTTP Archive it is still in the top (11th place). It was not only declared obsolete as early as 1997, but it was never thought of as a response header, but only as part of a request.

Pragma: no-cache

Nevertheless, it is so widely used as a response header that some browsers even recognize it in this context. But today there is almost zero chance that someone understands Pragma in the context of the answer, but does not understand Cache-Control . If you want to disable caching, all you need is Cache-Control: no-store, private .

Non-browsers (X-Robots-Tag)

One title in our top 30 is not a browser title. X-Robots-Tag designed for crawlers, such as Google or Bing bots. Since it is useless for the browser, you can set such an answer only to the requests of crawlers. Or you decide that it makes testing difficult or violates the terms of use of a search engine.

Bugs

Finally, it is worth ending with an honorable mention of simple mistakes. The Host header makes sense in the request , but if it is found in the response, then your server is probably configured incorrectly (and I would like to know exactly how). However, the 68 domains in the HTTP Archive return the Host header in their responses.

Deleting headers on a CDN edge server

Fortunately, if your site works with us on Fastly, then removing the headers is quite simple using VCL . If you want to save the development team with really useful data for debugging, but hide it from the general public, this is easily done over the cookie or the incoming HTTP header:

unset resp.http.Server;
unset resp.http.X-Powered-By;
unset resp.http.X-Generator;

if (!req.http.Cookie:debug && !req.http.Debug) {
unset resp.http.X-Amzn-RequestID;
unset resp.http.X-Cache;
}

In the next article I will talk about best practices for really useful headers and how to activate them on the CDN edge server.