Content Compression in Microsoft IIS Server
How to make IIS deliver compressed content to HTTP 1.1-compliant browsers.
by Konstantin Balashov

Posted April 26, 2004

About This Article
This article is adapted from the book Speed Up Your Site: Web Site Optimization by Andrew B. King. Copyright © 2003 by New Riders Publishing. Reproduced by permission of Person Education Inc. Publishing as New Riders Publishing. All rights reserved. Peachpit Press: www.peachpit.com.

Compression technology seems to offer something for nothing. Compression saves bandwidth and speeds up Web sites by removing redundancy to reduce the amount of data sent. Although the cost of compression is certainly not zero, over networks such as the Internet, transmission time and not rendering is usually the bottleneck. This article shows you how to compress the text in your content to minimize bandwidth costs and maximize speed.

Compression algorithms trade time for space by preprocessing files to create smaller versions of the files. A compressed file is later decompressed to reconstruct the original, or an approximation thereof. The compression process naturally includes two components: encoding and decoding. Encoding compresses the data, and decoding decompresses the data, which usually happens at a faster rate. Because bandwidth is more precious than CPU cycles, network concerns usually trump server load considerations.

Text Compression Algorithms
There are three major approaches to text compression:

• Dictionary-based (LZ stands for Lempel and Ziv).
• Block sorting-based (BWT, or Burrows-Wheeler Transform).
• Symbol probability prediction-based (PPM, or Prediction by Partial Matching).

Most file-compression formats like arj, zip, gz, lzh, and rar (including GIF and PNG) are based on dictionary algorithms of previously occurring phrases. By substituting the distance to the last occurrence and the length of the phrase, they save space. The LZW (Lempel, Ziv, and Welch) algorithm used in the GIF format is different—it substitutes a dictionary index for the phrase. LZ-based algorithms are very fast, with moderate compression ratios and modest memory requirements (< 100K).

BWT algorithms make a block-sorting transform over the text. The result of the transform is text of the same size, but letters are magically grouped. It can then be efficiently compressed with a very fast and simple coding technique. Block-sorting transforms make sense when they are applied to a big data block. BWT algorithms are fast, with high compression ratios and moderate memory requirements (1 MB or more).

PPM algorithms calculate the probability distribution for every symbol and then optimally encode it. Most PPM implementations are slow, sophisticated, and memory intensive (8 MB or more).

The efficiency of these lossless algorithms is measured in bits per character (bpc), or the number of bits needed to encode each character. The English language has an effective bpc of 1.3, so theoretically if a compression algorithm "knew" all the idioms and structure of this language, it could approach this figure. To give you an idea of how effective the current Web compression algorithms are, Table 1 illustrates lossless compression algorithms and their efficiencies.

Other than multimedia- or plug-in-based formats, the compression algorithms currently used on the Internet are pretty old and far from state-of-the-art. Will they be replaced with newer and better ones? Not likely anytime soon. Even if somebody came up with the world's best compression algorithm and the algorithm was patent-free, it will be hard to convince the major browser developers to add this algorithm to their browser code. But even if they do, it will be hard to convince Webmasters to use compression that is not supported by older browsers.

We've got to work with what we have. Let's see how we can squeeze the maximum advantage out of these algorithms.

Content Compression
Most browsers released since 1998-1999 support the HTTP 1.1 standard known as IETF "content-encoding" (although content encoding was included in the HTTP 1.0 specification: RFC 1945.3). Content encoding (also known as HTTP compression) is a publicly defined way to compress (that is, deflate) HTTP content transferred from Web servers to browsers using public domain compression algorithms, such as gzip.

Here's how it works. Browsers tell servers they would prefer to receive encoded content with a message in the HTTP header, like this:

Accept-Encoding: gzip

The server should then deliver the content of the requested document using an encoding accepted by this client. If the client isn't lying (like early versions of Netscape 4.x can), the compressed data is decompressed by the browser. Modern browsers that support HTTP 1.1 content-encoding support ZLIB inflation of deflated documents and benefit from HTTP compression. Older browsers that don't send the Accept-Encoding header automatically receive the uncompressed version of your files.

The net effect is dramatically smaller files, faster page response, and lower bandwidth bills. In fact, Webmasters who have employed content encoding on their servers have realized bandwidth savings of 30 to 50 percent. The compression ratio depends on the degree of redundancy in your site's content and the ratio of text to multimedia. Most importantly, compressed pages display much faster because the browser downloads less data. As browsers decompress compressed content with ZLIB, a dictionary-based algorithm, decompression speed is very fast.

GZIP Compression
The easiest way to employ content encoding on your server is to use software specifically designed for this purpose. Mod_gzip, mod_hs, and mod_deflate-ru are software modules that automate the entire process for Apache, as does PipeBoost and Hyperspace i for Microsoft's IIS server. These server add-ons can work with static or dynamic content, and the predefined installation files take care of most common server and browser configurations. We'll compare the strengths and weaknesses of IIS servers later.

How good is gzip compression? Some say that the improvement can be up to 90 percent; others are happy with 25 percent. They just measure it in a different way. HTML files are typically compressed by 80 percent (5:1 ratio), whereas JavaScript and CSS files average 70 and 80 percent compression, respectively.

A typical HTML page consists of an HTML file, several image files, sometimes a CSS file, and a couple of JavaScript files. Images are already compressed, so it doesn't make sense to apply content compression to them. Because images and multimedia objects take more than half of the total Web page size, even if you compress the HTML file by 90 percent, the total compression ratio will be less than 50 percent.

To get an idea of how effective compression can be, take a look a Table 2. This table shows the potential compression ratios for five of the most popular high-tech companies, five online newspapers, five Web directories, and five sports resources. On average, the text portion of these sites was compressed by 75 percent. Overall, compression would save 37 percent in total file size.

Most dial-up modems use V42bis or V44bis compression based on the LZW algorithm. If modems already compress data, you might ask, why do we need any additional compression? First, because of the speed and limited memory of modems, modem compression has relatively low compression ratios. V42b provides a compression ratio of about 2:1 on most text files, and gzip gives much higher compression ratios of 3:1 or higher. Second, modems do not compress SSL-encrypted files. Most importantly, dial-up connections are only one way to connect to the Internet. DSL, cable, and T1 modems as well as network cards do not have compression onboard. That is why HTTP content compression is so important.

Browsers and servers have brief conversations about what kind of content they would prefer to accept and deliver. The browser tells the server that it can accept content encoding, and if the server is capable, it will then compress the data and transmit it. The browser decompresses the data and then renders the page. Clients that don't understand compressed content don't request encoded files and, thus, receive files uncompressed (assuming that the content is offered conditionally). By definition, HTTP 1.1-compliant browsers support gzip compression. Most modern browsers support gzip content encoding.

What is the difference between gzip and deflate? Both are based on the same compression algorithm, deflate, implemented in the compression library zlib. Deflate encoding assumes that you are sending only the compressed data. Gzip adds a 10-byte header to the compressed data. It also adds a CRC32 checksum and the length of the compressed data (4+4=8 bytes) to the end of compressed file. The image of transferred data is a valid .gz file.

There is one more content-encoding compression algorithm: compress, which utilizes a compression algorithm implemented in the UNIX compress utility. This algorithm is supported only by Lynx, Netscape, and Mozilla.

HTML and Compression
HTML and other text files can be compressed on the server and automatically decompressed with HTTP 1.1-compliant browsers. Because HTML files must be downloaded before your content appears, fast delivery of this page framework is critical to user satisfaction.

Because HTML text files can be highly redundant (especially tables), compression rates for HTML files can be dramatic, with savings up to 90 percent. Most modern browsers support decompression of HTML files compressed with gzip.

In theory, you can also compress external style sheets using content encoding. In practice, Webmasters have found that browsers inconsistently decompress .css files. Apparently, style sheets were hacked into some browsers in a non-HTTP-compliant way. So when these browsers receive a Content-Encoding: gzip header in the response for a .css file, they don't realize that they are supposed to decompress it first.

This is not always the case, however, and no one to my knowledge has been able to nail down which browsers can actually handle the decompression of style sheets and under what circumstances. The problem seems to involve a mixture of variables. Therefore, I recommend that you exclude compression of .css files in any configuration files for programs such as mod_gzip:

mod_gzip_item_exclude         file       \.css$

Most .css files are smaller than .js files anyway, so the need for compression is usually greater for .js files. In fact, CSS files are usually so small that the two HTTP headers needed to request and respond can add up to a significant portion of the total traffic (up to 750-1000 bytes). So for smaller CSS files on high-traffic pages, it may be more efficient to embed them directly into your (X)HTML files or use SSI, where they can then be compressed.

Like HTML files, external JavaScript files can be compressed with IETF content encoding. Unlike external .css files, support for decompressing compressed JavaScript files is good in modern HTTP 1.1-compliant browsers, as long as they are placed within the head of your HTML documents. Although it is possible, I don't recommend using proprietary compression methods to deliver external JavaScript files (.jar, CHM/ITS, etc.). The standards-based method described in this chapter requires at most only one additional file—not four—to maintain.

External scripts must be referenced in the head element of (X)HTML documents to be reliably decompressed by modern browsers. The story goes like this. Netscape's original specification for JavaScript 1.1 implied that the inclusion of JavaScript source files should take place in the head section, because that is the only place where they are preloaded and preprocessed. For some reason, browser manufacturers stopped decompressing any compressed files after leaving the head.

As scripts grew larger, developers started moving script elements down into the body to satisfy impatient users. In HTML, the "head-only" rule was then relaxed to allow script elements within the body, but the die was cast. Developers subsequently discovered that certain JavaScript inclusion operations must be in the head section or problems can occur.

Browsers continue to decompress scripts only when they are located within the head element. Some companies get around this limitation by adding "_h" to the names of JavaScript include files in the head section of HTML documents. Using this technique, a script author can use server-side filtering logic to determine whether a request for a certain JavaScript file is coming from the head section of a HTML document (where it is OK to send it compressed) versus somewhere in the body (where it is not OK to send the script compressed). You can optionally use the defer attribute to compensate for this requirement.

Content Compression: Server Side
So it sounds like all you need to do is compress your content on the server and deliver it, right? Unfortunately, it is not that simple. You have three options for choosing the type of compression you want to use:

• Static precompressed content.
• Dynamic realtime server-side compression.
• Proxy-based compression.

Static precompressed content makes sense for heavily loaded Web sites with static content. Some news Web sites keep all their news in a database and convert it to static HTML files once every several minutes. These sites send responses faster than sites with ASP, JSP, or PHP pages. The best thing to do here is to keep a compressed copy of the pages ready to send. If an HTTP request contains the Accept-Encoding line, send a compressed file; otherwise, send an HTML file.

Dynamic realtime server-side compression compresses files before sending them to the client. It can be implemented as a plug-in to a Web server or as a part of standalone proxy server.

Proxy-based compression compresses files between the server and the client, alleviating the need to compress files on the server. This reduces server load and gives ISPs and enterprise operations more power and flexibility.

Content Compression in Microsoft's IIS Server
With some help, Microsoft's Internet Information Services (IIS) server can deliver compressed content to HTTP 1.1-compliant browsers. The best way to enable content encoding on IIS is to use an ISAPI filter specifically designed for this purpose. For precompressed content, IIS doesn't have any built-in mechanism similar to Apache's Multiviews.

Microsoft's IIS server has content compression onboard. To turn it on, follow these steps:

1. Open the Computer Management window.
2. Select Internet Information Services.
3. Right-click and select Properties.

To set up compression for IIS, do the following:

1. In Internet Information Services Properties, click Edit in the Master Properties section.
2. In WWW Service Master Properties, check the Compress application files option and click OK
3. Restart the IIS server.

Ironically, this works with all browsers except Microsoft Internet Explorer. It is a caching problem. Internet Explorer displays the compressed file properly for the first time, but the refresh button ruins it. The file is there, and you can save it to disk, but you cannot see it properly.

A number of dynamic content compression solutions allow you to work around this limitation. All these tools are implemented as an ISAPI filter. The difference is flexibility.

PipeBoost
PipeBoost is a powerful, flexible, and easy-to-use commercial content compression solution for Microsoft IIS.

In PipeBoost, you can configure literally everything. You can set individual configurations for every Web site and every folder. You can assign individual content type handling rules for each browser.

You can even set custom compression levels for each file type (see Figure 1). The other good thing about PipeBoost is that it includes a powerful analyzing and monitoring system (see Figure 2), a cache performance analyzer, and some other useful tools.

Inner Media SqueezePlay
SqueezePlay from Inner Media Inc. is another commercial content compression solution for Microsoft IIS. It consists of Configurator and Accelerometer. Configurator is a GUI tool with a lot of dialogs and settings. You can configure browser types and content types. The Accelerometer is a graphics analysis tool with column diagrams. It shows sent data size and savings.

In addition to the HTTP content compression, SqueezePlay provides image optimization. An LZW license to optimize GIF files is not included; you have to buy it separately.

Vigos IIS Accelerator
The Vigos IIS Accelerator ISAPI filter has all the configuration options from the Vigos Website Accelerator, excluding the proxy specific ones. The filter can remove white space before compression, auto-detects browsers and MIME-types to avoid incompatibilities, and includes graphics optimization. It is configured with a text file, but comes with a handy configuration tool for editing this file.

IIS ISAPI Compression Filters
As long as all this software uses the same compression algorithm and the same ISAPI technology, you'll find no relevant difference in performance or compression ratio with IIS ISAPI Compression Filters. Just keep in mind that the tools will slow down server response about 1.5 times (for example, 1.5-second versus 1-second latency). On the majority of Web sites, however, this added overhead is more than made up with the time saved sending compressed content. Therefore, I recommend that you make a decision based on the functionality you need, the added maintenance you are ready to take on, and the price you are willing to pay.

Proxy Server-Based Content Compression
Compression does not necessarily need to be done by the Web server. It can be implemented as a proxy server installed on the same or on a standalone server. Such a server can work with any Web server. It can run on a different platform under a different operating system.

If you can install a compression server on the server side and decompression on the client side, you can use better compression algorithms. Proxy-based solutions give you real power, but they usually are more sophisticated. I recommend that you use them only in the following cases:

• To move compression to separate servers to reduce Web server CPU load.
• To compress an Internet line with traffic when Web servers are not available for administration.
• To use with a Web server that does not support content compression.

There are several proxy-based solutions. Most of these companies have solutions for ISPs and enterprise-size networks (see Resources).

For maximum compatibility, consider one of these proxy solutions. They avoid problems with SSL and dynamic content. Vigos Website Accelerator is a reverse-proxy software solution that automatically removes white space, optimizes graphics, compresses content, and includes Vigos' SmartShrink database, which the company says eliminates HTTP compression compatibility problems.

For maximum speed, consider a hardware compression proxy. Packeteer's AppCelera ICX is a hardware/software appliance that automatically adjusts the optimization level based on the user's browser type and connection speed. It uses compression, image optimization, and conversion from GIFs to PNGs or JPEGs, caching, and SSL processing to minimize page size. AppCelera reports on user and server response times, load, and connection profiles. Packeteer also incorporates delta encoding in their latest products.

About the Author
Andrew B. King is the founder and former managing editor of WebReference.com and JavaScript.com, two developer sites acquired by Jupitermedia. Since 1993, King has worked full time as a Web professional applying and teaching Web optimization and creation techniques.