CONTENT
 PRODUCTS
 ARTICLES
 DOWNLOAD
 SOURCE CODE
 BOOK
 ABOUT
 HOME

MVP

ADS




Web Service Compression with .NET CF
We'll look at how to use standard web server compression with XML Web Services to make Pocket PC applications use less bandwidth, and thereby save communication costs. This is extra interesting as mobile devices tend to use bandwidth that is more expensive (GPRS, etc).

Most enterprise Pocket PC solutions involve communication with a back-end server. With the introduction of the SOAP standard (or the more commonly used term "XML Web Services"), the industry had an option of accessing back-end resources in a standard way. Further, as Pocket PCs got the powerful (and nicely wrapped) support for calling those XML Web Service, the mobile solutions could benefit from easier integration and faster solution implementation. However, as SOAP is based on XML, and XML is a text-based format with extensive meta-data, the network traffic can be quite heavy. For a normal network connection (one that you normally have to your desktop), this is normally to a problem. But for most mobile solutions, that use network connection with lower bandwidth, this can create performance problems. Even worse, as those mobile network connections are more expensive to use, this can also create high communication costs.

WSCompress Anyplace Sample
WSCompress Anyplace Sample
As shown in this article, standard compression of web content can be used not only to increase the performance of Pocket PC applications that use XML Web Services, but also significantly reduce communication costs for running such applications. Before we dive into how to achieve this with all necessary configurations and cool code, let's look at the reasons why you'd want to do this.

Get the sample source code!

Tests
To find out how much can be achieved in terms of performance, bandwidth, and consequently saved communication costs, here are some test results. The absolute numbers may not be as interesting as the relations between the measured values. Let's start by looking at the actual win in terms of bandwidth, and the first measures are for a small payload:

Compression of small payload
Figure 1. Compression of small payload

The uncompressed small payload was almost 2 KB (1878 bytes), and when compressed it was reduced to about half that size (for gzip to 955 and for deflate to 935). Note that all measured payload values are including the HTTP headers that are never compressed. Then, the large payload:

Compression of large payload
Figure 2. Compression of large payload

The uncompressed large payload was almost half a megabyte (456804 bytes), and when compressed it was reduced to about a tenth of that size (for gzip to 45322 and for deflate to 45303). The conclusion is that the larger the payload, the more we gain in terms of saved bandwidth with compression, and also that there is not much difference if we use "gzip" or "deflate" to compress. Therefore, the fact that the preferred format is "gzip" is no problem.

When we look at performance, these are the measures for the small payload:

Performance for small payload
Figure 3. Performance for small payload

The uncompressed small payload was retrieved in a tenth of a second (100 ms). When the same payload was compressed (to about half the size, see Figure 1 above), retrieved, and uncompressed, the total time increased about a fifth (for both gzip and deflate to 120). This means that for small payloads, we save about 50% of the bandwidth, but have to pay with a performance penalty of about 20%.

What about the large payload? The measures look like this:

Performance for large payload
Figure 4. Performance for large payload

The uncompressed large payload was retrieved in a little more than 9 seconds (9336 ms). When the same payload was compressed (to about tenth of the size, see Figure 2 above), retrieved, and uncompressed, the total time decreased to about half (for gzip to 5299 and for deflate to 5052). This means that for larger payloads, we save about 90% of bandwidth, and we also gain a performance win of about 50%.

A few words about how the tests were done: The small payload was created from a DataSet with the Shippers table, and the large payload with the Orders table. Both tables come from the standard Northwind database. The tests were made with the emulator which means optimal network conditions and a fast processor, but other tests were made on physical devices (PXA255 processor) and with real-world network conditions (802.11b WLAN with my son gaming heavily online) that show very similar results (the relations between the values are almost identical). For example, a PXA255 on a WLAN retrieved the small payload in 330 ms, and compressed (retrieved and uncompressed) it took 360 ms (only a performance hit of about 10%). For the large payload, the uncompressed took 24 seconds to retrieve, and compressed it took 13 seconds (a performance win of about 45%).

See the conclusion at the end of the article for a summary of the tests.

Background
The HTTP 1.1 Specification (RFC 2616) includes a specification for content compression, and in the latest version of IIS (Internet Information Services, 6.0), there is built-in support for this compression. This means that you can enable compression for all your ASP.NET and ASP.NET Web Service applications without changing any code. However, to use those applications from a client, you need to make some changes to the Web Reference (the generated proxy) to make it work. The details for doing that is nicely explained in the article Retrieving Data from Web Services using Standard HTTP 1.1 Compression by Jacek Chmiel. When trying to do the same thing with .NET CF, you encounter that you get a compilation error that can be fixed by the advice expressed in the excellent blog post Getting Web Services using HTTP 1.1 Compression to work in Compact Framework by Alex Yakhnin.

Although parts of the solution can be found on the links mentioned, this article includes all the steps necessary to make a Pocket PC .NET CF application make calls to XML Web Services (written in ASP.NET and running on IIS 6.0) that uses HTTP 1.1 Compression.

HTTP 1.1 Compression
As described in the HTTP 1.1 Specification, content from a web server can be compressed using a number of algorithms. The client request compressed content by including a header named "Accept-Encoding" with the compressed formats it supports in a comma-separated list. The mostly used formats are "gzip" (RFC 1952) and "deflate" (RFC 1951), where "gzip" is the preferred format. If the server supports compression, the response will be compressed according to the format(s) specified in the request header ("Accept-Encoding").

To make all this work, there are some necessary preparations. First, HTTP 1.1 Compression must be enabled in IIS 6.0, and then the .NET CF files on the development PC must be updated.

HTTP 1.1 Compression in IIS 6.0
To enable HTTP 1.1 Compression on IIS 6.0, you need to make some updates to the web server configuration. First, you need to enable compression for the file extensions that you want to use. As ASP.NET Web Services has the file extension ".asmx", that extension need to be enabled. First, we enable this extension (together with the default ones) for "gzip" compression by opening a Command Prompt, and running the following command (on one line):
  cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs
    set W3Svc/Filters/Compression/gzip/HcScriptFileExtensions "asp" "dll" "exe" "aspx" "asmx"
Note that compression is now also enabled for ASP.NET applications (using the file extension ".aspx"). Staying in the same folder, we can enable the file extensions for "deflate" compression as well with the following command (on one line):
  cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs
    set W3Svc/Filters/Compression/deflate/HcScriptFileExtensions "asp" "dll" "exe" "aspx" "asmx"
Then, the default compression level (0) should be increased to make the compression rate higher (9) without increasing the CPU usage too much. Here's what Scott Forsyth says in his article IIS Compression in IIS6.0:

"HcDynamicCompressionLevel has a default value of 0. Basically this means at if you did everything else right, the compression for dynamic contact is at the lowest level. The valid range for this is from 0 to 10. I had the opportunity of receiving an internal testing summary from Chris Adams from Microsoft regarding the compression level -vs- CPU usage which showed that the CPU needed for levels 0 - 9 is fairly low but for level 10 it hits the roof. Yet the compression for level 9 is nearly as good as level 10. I write all this to say that I recommend level 9 so make sure to change HcDynamicCompressionLevel to 9. Do this for both deflate and gzip."

This level is changed (still in the same folder) for "gzip" by entering the command:
  cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs
    set W3SVC/Filters/Compression/gzip/HcDynamicCompressionLevel "9"
The same change is done for "deflate" with the command:
  cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs
    set W3SVC/Filters/Compression/deflate/HcDynamicCompressionLevel "9"
To make all these changes active, IIS needs to be restarted, and that can be done with the following command:
  iisreset /restart
The above changes can also be done directly in the file C:\WINNT\system32\inetsrv\Metabase.xml (or C:\Windows\system32\inetsrv\Metabase.xml), but be sure to make a backup copy before you change it! Editing the Metabase.xml file also requires the IIS to be stopped before, and restarted only after the changed file has been saved. As this also means that you easier risk messing up your web server configuration, the above way (using the adsutil.vbs script) is clearly the recommendation. Just for reference, the two updated sections of the Metabase.xml file should look like this:
  <IIsCompressionScheme Location ="/LM/W3SVC/Filters/Compression/deflate"
    HcCompressionDll="C:\WINNT\system32\inetsrv\gzip.dll"
    HcCreateFlags="0"
    HcDoDynamicCompression="TRUE"
    HcDoOnDemandCompression="TRUE"
    HcDoStaticCompression="TRUE"
    HcDynamicCompressionLevel="9"
    HcFileExtensions="htm
      html
      txt"
    HcMimeType=""
    HcOnDemandCompLevel="10"
    HcPriority="1"
    HcScriptFileExtensions="asp
      dll
      exe
      aspx
      asmx"
  >
  </IIsCompressionScheme>
  <IIsCompressionScheme Location ="/LM/W3SVC/Filters/Compression/gzip"
    HcCompressionDll="C:\WINNT\system32\inetsrv\gzip.dll"
    HcCreateFlags="1"
    HcDoDynamicCompression="TRUE"
    HcDoOnDemandCompression="TRUE"
    HcDoStaticCompression="TRUE"
    HcDynamicCompressionLevel="9"
    HcFileExtensions="htm
      html
      txt"
    HcMimeType=""
    HcOnDemandCompLevel="10"
    HcPriority="1"
    HcScriptFileExtensions="asp
      dll
      exe
      aspx
      asmx"
  >
  </IIsCompressionScheme>
Now, you need to enable compression for dynamic content (application files) by following two steps:
  1. In IIS Manager, expand the local computer, click to select, then right-click the Web Sites folder, and then click Properties.
  2. Click the Service tab, select the Compress application files check box, and click OK.
This will enable compression for all dynamic content of all virtual directories. If you want to enable compression only for a specific virtual directory, you first have to find out the identifier for the site where the virtual directory is located (you can check it in IIS Manager by selecting the Web Sites folder and look in the Identifier column on the right). Then, you use the following command to enable compression for a specific virtual directory (in this case the site with identifier 1, and the virtual directory WSCompression):
  cscript.exe C:\Inetpub\AdminScripts\adsutil.vbs
    set W3SVC/1/Root/WSCompression/DoDynamicCompression True
Ok, now HTTP 1.1 compression should be enabled on your web server, so let's go on and update the development tools.

Update Development Tools
To allow the sample (see below) to compile in your development environment, you need to replace a file on the development PC. This is because the Service Pack 2 of .NET CF 1.0 support overriding some needed methods, but the SP2 installation does not update the development environment. But the files can be updated manually by following this procedure:
  1. Download the .NET CF 1.0 SP2 Developer Redistributable.
  2. Unpack the netcf.core.ppc3.x86.cab from the distribution (with a tool like WinRAR).
  3. Rename the SYC6B2~1.014 file to System.Web.Services.dll.
  4. Copy it to the C:\Program Files\Microsoft Visual Studio .NET 2003\CompactFrameworkSDK\v1.0.5000\Windows CE folder (could be located elsewhere depending on chosen installation location).
That's it! Now you should be able to compile the below sample, and obviously the SP2 bits also need to be installed on the device (or emulator) where the sample is run.

WSCompress Anyplace Sample
Building on the samples mentioned (see the background above), the sample show how to call XML Web Services that use HTTP 1.1 Compression in your Pocket PC applications. This sample application is built for .NET CF with Visual Studio .NET 2003 and it looks like this:

WSCompress Anyplace Sample
Figure 5. WSCompress Anyplace Sample

The sample takes an entered table name (from the Northwind sample database) and sends it off to an XML Web Service on a server. That XML Web Service queries the Northwind database (SELECT * FROM ) and returns the result as a DataSet to the Pocket PC. The time it takes to make the call is measured and shown (in milliseconds) after the call is completed. After the DataSet is returned, its content is displayed as XML in a text box. In the tests above, the sample is modified to (1) not use compression, (2) use "gzip" compression, and (3) to use "deflate" compression.

Code Walkthrough
To start on the server side, the following is the code for the ASP.NET Web Service method:
  [WebMethod]
  public DataSet GetDataSet(string table)
  {
    SqlConnection cn = new SqlConnection("Data Source=(local);" +
      "Initial Catalog=Northwind;Integrated Security=SSPI;");
    cn.Open();
    SqlDataAdapter da = new SqlDataAdapter("SELECT * FROM " + table, cn);
    DataSet ds = new DataSet();
    da.Fill(ds, table);
    return ds;
  }
If compression has been enabled in IIS for all virtual directories or for this specific virtual directory (as described above), this new XML Web Service support HTTP Compression as soon as it is compiled.

On the client side, a normal Web Reference (named WebServices) was created with the following declaration in the form's class:
  private WebServices.DataService ws = new WebServices.DataService();
Then, the following code was entered in the button's Click event:
  Stopwatch sw = Stopwatch.StartNew();
  DataSet ds = ws.GetDataSet(tableTextBox.Text);
  sw.Stop();
  xmlTextBox.Text = ds.GetXml();
  MessageBox.Show(sw.ElapsedMilliseconds.ToString() + " ms");
The XML Web Service proxy instance (ws) is used to call the method (GetDataSet), and the returned DataSet (ds) is inserted as XML into the TextBox (xmlTextBox). To measure performance, the Stopwatch class from OpenNETCF (http://www.opennetcf.org/) was used (it will be included in version 1.3 of their Smart Device Framework). On the first line a new Stopwatch instance is created (sw), and after the XML Web Service method call, it is stopped. When the TextBox is filled, the resulting call time is presented.

When the code is run at this point, it will not use compression, and therefore this is the code executed for the first test run (see the test results above).

Now, to enable compression, we need to make some changes to the generated Web Service proxy (normally called Reference.cs), and at the end of the proxy class (DataService) the following overrides are added:
  protected override System.Net.WebRequest GetWebRequest(Uri uri)
  {
    System.Net.WebRequest request = base.GetWebRequest(uri);
    request.Headers.Add("Accept-Encoding", "gzip, deflate");
    return request;
  }
  protected override System.Net.WebResponse
    GetWebResponse(System.Net.WebRequest request)
  {
    HttpWebResponseDecompressed response = new
      HttpWebResponseDecompressed(request);
    return response;
  }
The first override (GetWebRequest) simply adds the HTTP header to the request, and the second (GetWebResponse) uses a custom class (HttpWebResponseDecompressed) to decompress the response. This class need to be added to the project, and it looks like this:
  internal class HttpWebResponseDecompressed : WebResponse 
  {
    private HttpWebResponse response;

    public HttpWebResponseDecompressed(WebRequest request) 
    {
      response = (HttpWebResponse)request.GetResponse();
    }
    public override void Close() 
    {
      response.Close();
    }
    public override Stream GetResponseStream()
    {
      Stream compressedStream = null;
      if(response.ContentEncoding == "gzip")
      {
        compressedStream = new
          GZipInputStream(response.GetResponseStream());
      }
      else if(response.ContentEncoding == "deflate")
      {
        compressedStream = new
          ZipInputStream(response.GetResponseStream());
      }
      if(compressedStream != null)
      {
        // Decompress
        MemoryStream decompressedStream = new MemoryStream();
        int totalSize = 0;
        int size = 2048;
        byte[] writeData = new byte[2048];
        while(true) 
        {
          size = compressedStream.Read(writeData, 0, size);
          totalSize += size;
          if(size > 0) 
            decompressedStream.Write(writeData, 0, size);
          else 
            break;
        }
        decompressedStream.Seek(0, SeekOrigin.Begin);
        return decompressedStream;
      }
      else
        return response.GetResponseStream();
    }
    public override long ContentLength 
    {
      get { return response.ContentLength; }
    }
    public override string ContentType 
    {
      get { return response.ContentType; }
    }
    public override System.Net.WebHeaderCollection Headers
    {
      get { return response.Headers; }
    }
    public override System.Uri ResponseUri
    {
      get { return response.ResponseUri; }
    }
  }
This class in turn uses part of the SharpZipLib library that implement the decompression streams (GZipInputStream for "gzip" and ZipInputStream for "deflate").

When the code is run at this point, it will use the compression specified by the HTTP header ("Accept-Encoding"), and in the second test (see the test results above), the following line of code was used (in the override of the GetWebRequest method, see above):
  request.Headers.Add("Accept-Encoding", "gzip");
In the third and final test, the following line was used:
  request.Headers.Add("Accept-Encoding", "deflate");
For a more complete example, see the WSCompress Anyplace source code.

Conclusion
The potential wins in terms of bandwidth and communication costs are obvious when using HTTP 1.1 Compression. On small loads (like the Shippers table with 3 rows) the saving is at least 50%, and on larger loads, more than 90% compression, and cost saving, can be achieved. When it comes to performance, the tests show that for small payloads, we have to pay a performance penalty of about 20%, but for larger payloads, we actually get a performance win of about 50%. The advice (as always) is not to always use new possibilities, but you should really take a look at what this means for your application.

Any comments?


©2001-2009 Christian Forsberg & Andreas Sjöström