Web caching architecture

[DESIRE] [DESIRE Web cache]

This document describes the design of a Web caching system. A short checklist is provided to help the system administrator who sets up a Web cache server as a part of a Web caching system.

1 Introduction
2 Web cache Server Software
- Squid
- Netscape
3 Clients
4 Location of your Web caches
5 Mesh, cooperating servers
- ISP-wide neighbors/parents
- International neighbors/parents
- Institutional neighbors/parents
- Configuring Squid in a mesh
6 Statistics and logs
7 Caching Policy
8 Hardware
9 Examples
11 References

The entire document (warning: internal links are broken)

Ingrid Melve - DESIRE Web caching team

Last modified: Thu Mar 20 14:10:16 1997 Web caching System Design Checklist [DESIRE] [DESIRE Web cache] [Web cache Architecture]

Web caching System Design Checklist

This is a short checklist version of a longer detailed Web caching Architecture Document

Your software

We recommend that you

use Harvest-based software
- Squid
- Cached-2.*
We have tested Squid extensively and found it very good
if in Norway, use our Squid-package for Linux and SAMSON

Clients

We recommend that you

configure all clients to use your first level webcache
if you use Netscape, set Automatic Proxy Configuration to avoid single point of failure

Be aware that only Netscape offers a fallback mechanism (yet).

Your location

We recommend that you

always place webcaches close to where the traffic flows
place caches on your side of network bottlenecks

First level cache

We recommend that you place your first-level Web cache

as close as possible to your external Internet connection
close to users
if you have several first-level: where networks come together

Upper-level Web cache

We recommend that you place your upper-level Web caches

where networks join
on or close to a national Internet exchange point
on or close to an international Internet exchange point

Your mesh

We recommend that you

allow access to your Web cache server only for those you have set up an agreement with
always ask the administrator before you set a Web cache as you parent or neighbor
go to NLANR registration database, find potential neighbors and parents in your area
contact you Internet Service Provider

Your statistics

We recommend that you

keep logs unavailable unless anonymized
rotate and process logfiles once a day (or more) due to their large size
keep the large size of the log files in mind if you write programs to analyze them
keep the privacy of your users in mind when you publish results of log file analyzing
find out what kind of statistics you want from your server, or you'll drown in numbers

Your policy

We recommend that you

register your server
contact your Internet Service Provider before setting up parents outside your network
observe your Acceptable Use Policy (if you have one)
keep logs and statistics unavailable unless anonymized

If you are an Internet Service Provider, we recommend that you

establish peering agreements with other providers, this is very important within a country/language zone
investigate your routing agreements with connection providers

Your hardware

Toplevel server

We recommend that you either have

more than one big strong UNIX server (dedicated), if you have one, be prepared to buy another
or a farm of smaller specialized servers (dedicated)

Local server

Depending on your user community we recommend

Linux, if you have a reasonable amount of users (low cost equipment)
any UNIX-system with
- good I/O
- lots of disk if you have many users
- lots of memory if you have lots of disk

Ingrid Melve

Last modified: Mon Mar 24 17:33:19 1997 Web caching architecture, conclusions [DESIRE] [DESIRE Web cache] [Web cache Architecture]

10 Executive summary

Drawing on the experiences in the EC 4th Framework Project DESIRE, funded by the European Union, this document identifies some of the pitfalls and possibilities of building Web caching meshes. This document gives guidelines and outlines strategies for building a functional Web caching system with cooperating Web caches.

Institutions and Internet Service Providers should provide Web cache servers for their users; and teach users to use caching. Web caching halves your Web related network traffic and reduces download time for popular Web pages.

In order to have a high quality of service, a mesh of Web cache servers using cooperative Web caching that supports the Internet Cache Protocol is the best solution.

The architecture of a Web caching mesh is highly depending on the network topology and the cost factors of the network.

The emphasis is on building Internet Service Provider wide caching systems, but architectural considerations for inter-network agreements on Web caching is discussed and examples of policies for international Web caching agreements are given. The impact of Web caching as a routing issue is discussed.

A summary of the advice given in the document:

a single caching server is not a good solution (not scalable), and therefore the use of a mesh of cooperating caching servers (a caching mesh) is necessary
the operation of a caching mesh must be resistant to the failure of one of its caching servers, so therefore a robust protocol such as the ICP and the use of apc scripts is advisable
more than 3 levels of caches in a caching mesh should be avoided
Web clients situated in a LAN, using a local caching server on this LAN should preferably turn off local disk caching; Web clients situated at an dial-up line should preferably use their local disk cache
a proper architecture for a caching mesh is highly dependent on network topology and care should be taken not to violate routing policies
place Web cache servers on the users side of a bottleneck
place Web cache servers close to the flow of traffic
place a local Web cache server close to your internet connection
place Web cache servers where more networks join together
upper level Web caches should be placed where more networks join together
the caching servers in a caching mesh should tune the lifetimes of the objects they cache, as improper settings may result in less performance of the caching mesh

Ingrid Melve

Last modified: Tue Feb 4 12:54:08 1997 Web caching architecture, introduction [DESIRE] [DESIRE Web cache] [Web cache Architecture]

1 Introduction

1.1 Terminology
1.2 Introduction to Web caching
1.3 Recommended reading
1.4 Requested background

The architecture of a Web caching mesh is highly dependent on the network topology and the cost factors of the network.

The emphasis is on building Internet Service Provider (ISP) wide caching systems, but architectural considerations for inter-network agreements on Web caching is discussed and examples of policies for international Web caching agreements are given. The impact of Web caching as a routing issue is discussed.

We look into several criteria for choosing Web caching server software and discuss why Squid has been chosen as the primary server software. Then the importance of Web browser (client) configuration is discussed. Criteria for location of Web caching servers and the importance of network topology and traffic flow are discussed. Several basic requirements for hardware used on Web caching servers are included, giving general advice and pointers to several hardware suppliers. The main part of the paper is focused on how to build and join an ISP-wide Web caching mesh, providing examples and listing alternative approaches to the architecture of a mesh.

1.1 Terminology

In this document a number of terms are used to refer to the roles played by participants in, and objects of, the HTTP communication. The following definitions are used in the HTTP/1.1 specification [HTTP/1.1] :

client: An application program that establishes connections for the purpose of sending requests. These are often browsers, editors, spiders (Web traversing robots), or other end user tools.
server: An application program that accepts connections in order to service requests by sending back responses.
origin server: The server on which a given resource resides or is to be created.
proxy: An intermediary program which acts as both a server and a client for the purpose of making requests on behalf of other clients. Requests are serviced internally or by passing them, with possible translation, on to other servers. A proxy must interpret and, if necessary, rewrite a request message before forwarding it. Proxies are often used as client-side portals through network firewalls and as helper applications for handling requests via protocols not implemented by the user agent.
cache: A program's local store of response messages and the subsystem that controls its message storage, retrieval, and deletion. A cache stores cacheable responses in order to reduce the response time and network bandwidth consumption on future, equivalent requests. Any client or server may include a cache, though a cache cannot be used by a server while it is acting as a tunnel.

To these definitions we add definitions specifically concerned with Web cache systems

caching mesh: a set of cooperating caching servers
local Web cache server: caching server running on the same LAN as a client
first level Web cache server: the Web cache server an end user client connect to
upper level Web cache server: seen from the clients view, all caches participating in the caching mesh that are not the clients first level cache are upper level caches
top-level Web cache server: one or more servers in a hierarchical caching mesh, normally few requests are made to other caching servers from the top level, serves first level Web cache servers
neighbor/sibling Web cache server: caching server participating in caching mesh, sends/receives requests to other cache servers
parent Web cache server: neighbor/sibling caching server through which misses are resolved

1.2 Introduction to Web caching

The introduction of World Wide Web has changed the view of the user community to the network in respect to accessibility and user friendliness. With the Web the user is able to look for and retrieve all kind of information from the network without having any knowledge of the network. From the user point of view it does not matter if the information he is looking for, e.g. a video clip with sound effects, is on a computer in the next room or on the other side of the world. This leads to an enormous growth of traffic on the local, national and international network backbone. Because the use of the Web is growing exponentially it is to be expected that the WWW traffic on the national and international networks also will grow exponential with raising latency. Nevertheless the user expects a high quality of service with modest response times.

The quality of service and the response times can be improved by decreasing the network load. One way to achieve this is to install a Web caching service. Caching effectively migrates copies of popular documents from Web servers closer to the Web clients. In general, Web client users see shorter delays when requesting a URL, network managers see less traffic and Web servers see lower request rates. An origin Web server might not only see lower requests rates but primarily will experience a lower server load because files will be fetched with an If-Modified-Since GET HTTP request. Web clients request documents from Web servers, either directly or through a Web cache server or proxy. A Web cache server has the same functionality as a Web server, when seen from the client and the same functionality as a client when seen from a Web server. The primary function of a Web cache server is to store Web documents close to the user, to avoid pulling the same document several times over the same connection, reduce download time and create less load on remote servers.

There is a certain urgency which may be deduced from network usage graphs, which show HTTP traffic increasing with a faster exponential growth rate than total Internet traffic. In 1995 the HTTP traffic has become a significant proportion of the total, and so Web growth is pushing network upgrades, causing performance degradation, and itself being slowed by network limitations [Berners-Lee].

However, the use of caches introduces two main problems: how do we ensure that the caches return the correct values, and how do we obtain the best possible performance. In general, there may be a trade-off between how correct the returned values are, and how effectively the caches perform.

Caching is used in two forms in the Web. The first is a local cache, which is built into a Web browser. A Web client with caching stores not only the documents currently displayed in browser windows, but also documents requested in the past. This local cache is also used for (temporary) storage of the history. There are two forms of client caches: persistent and nonpersistent. A persistent client cache retains its documents between invocations of the Web browser; Netscape uses a persistent cache. A nonpersistent client cache (used in Mosaic) deallocates any memory or disk used for caching when the user quits the browser. Per-client caches may maintain consistency of cached files with server copies by issuing an optional If-Modified-Since GET to the HTTP server or proxy server.

The second form of caching is in the network used by the Web (i.e., the caching proxy mentioned earlier). The cache is located on a machine on the path from multiple clients to multiple servers. Normally a caching proxy is not on a machine that runs a Web client or an HTTP server. The caching proxy caches URLs generated by multiple clients. It is possible to use caching proxies hierarchically, so that caching proxies cache URLs from other caching proxies. In this case the caches can be identified as first level caches, second level caches, and so on. A hierarchical arrangement is only one possible configuration; Smith [Smith96] gives a nonhierarchical arrangement of large national and international caching proxies.

Cache network

Web caching servers may cooperate in a mesh where they share load and distribute requests. A Webcaching server in a mesh may pass a unfulfilled request on to another Web caching server.

The simplest mesh is two servers strung after another, if a request is not answered on the first cache server, it is passed on to the second and then on to the Web server. Another way is to build a network of servers that have different relations to one another, this may be done by the help of the Internet Cache Protocol, ICP [ICP-draft]. ICP passes messages among Web cache servers.

1.3 Recommended reading

Quick Introduction to WWW Caching, Martin Hamilton
DESIRE survey on caching, Henny Bekker, Ingrid Melve, Ton Verschuren: DESIRE project, requirements and recommendations for Web caching
WWW Cache Briefing, Alan J. Flavell: What is Web caching, why use it, how to start
A Hierarchival Internet Object Cache, Peter Danzig: Harvest cache and the basic ideas for cache cooperation
A Caching Relay for WWW, Steven Glassman: One of the first papers on Web caching, an introduction to the topic
Caching Proxies: Limitations and Potentials, Marc Abrams, Charles R. Standridge, Ghaleb Abdulla, Stephen Williams, Edward A. Fox: Potential of Web caching
The Harvest Object Cache in New Zealand, Donald Neal: Includes overview of the Internet Cache Protocol
UK Advisory Committee on Networking report on Web Caching, Andrew Cormack: State of the art on Web caching
Evolution of the NLANR Cache Hierarchy: Global Configuration Challenges, Duane Wessels and k claffy: Discussion of cache meshes and configuration
Caching, introduction for libraries, Jon Knight and Martin Hamilton: Laymans introduction to Web caching

More material is available at

Woj's links to caching info, the Polish W3CACHE
Recommended Reading, NLANR

1.4 Requested background

In this document we assume that you have a working knowledge of Web caching and the primary target group is system administrators that are running a Web cache server or planning to start one. Executives are referred to the executive summary.

Examples are provided using the Squid software. Documentation on Squid is available from the Web site and is not included in this document.

Ingrid Melve, Henny Bekker

Last modified: Tue Mar 25 10:11:49 1997 Web caching architecture, software [DESIRE] [DESIRE Web cache] [Web cache Architecture]

2 Web caching software

2.1 Cooperating servers
2.2 Netscape Proxy
2.3 Squid
2.4 Other proxy servers
2.5 Configuring a Web cache server for heavy traffic

Checklist

We recommend that you

use Harvest-based software
- Squid
- Cached-3.*
We have tested Squid extensively and found it very good
if in Norway, use our Squid-package for Linux and SAMSON. If in the Netherlands, refer to the SURFnet caching pages

2.1 Cooperating servers

Any Web caching software that is unable to cooperate with several servers is questionable, because the single server approach does not scale sufficiently when one is experiencing exponential growth. One monolithic national server has severe scaling problems as well as routing policy problems. One of the things we know about Internet traffic is that it increases exponentially, a single server is not a good solution. The approach has been tested in UK [Smith96], and even with a cache distributed over 6 computers located at 2 different places (in order to provide redundancy), there is heavy load.

Criteria number 1: The server software must be able to cooperate with several other servers.

The Web Consortium Problem Statement on Propagation, Replication and Caching states: "There is an urgent need for making the Web more mature in order to scale to a number of at least 100 times the current size, and efficient techniques for replication and caching is a corner stone in achieving this goal." Any Web caching solution must be able to scale to at least 100 times today's use.

The scaling issue is illustrated by the fact that on the 16th September 145 GB of traffic went in/out of UNINETT, a rough estimate indicates that half of this was Web traffic, 72 GB makes about 15 million connections.

Criteria number 2: The server solution must be scalable with a factor of at least 100

A Web caching system should be transparent for the user, the only result he should notice is faster response [Bekker96]. This implies that servers must fail gracefully.

Criteria number 3: The server system must fall back gracefully in case of failures

2.2 Netscape Proxy Server

The Netscape proxy server belongs to the second generation of proxy servers. It is a fast, stable and reliable server. Due to its advanced process model (using pre-forked processes) and the dynamic and sophisticated Resource Manager, it should be well suitable for creating a hierarchical caching service.

A big disadvantage, however, is the way it deals with unreachable or misconfigured proxy servers higher in the caching hierarchy. There are no mechanismes for detecting failures and route around them.

Even with the use of the Automatic Proxy Configuration facility of the Netscape browser, the URLs, which are served by the parent of the proxy server, are unreachable. A chain of Netscape proxy servers is as strong as the weakest link. The misbehaviour of the chain of proxy servers is at least the sum of the individual components. Due to this shortcoming, the authors discourage the use of the Netscape proxy server for creating a hierarchical caching service.

The Netscape proxy does not have support for the Internet Cache Protocol (ICP).

2.3 Squid

The Squid software is being developed as a free version of the Harvest software. A commercial version (Cached-3.*) is available. Given the requirements of the academic community to save money, Squid has been chosen as the first test case. Cached-3.* needs to be further investigated.

Squid is available for most flavors of Unix, and has also been ported to OS/2 recently.

The Squid proxy server belongs to the second generation of proxy servers. It is a fast single process server (implements its own "threads" in a select-loop). Squid is a public domain server based on the Harvest v1.4 code. It is developed "on the net", and the 1.1 version has been in a beta development state with frequent changes up until the release of Squid-1.1.0 on December 6, 1996 . Some of the 1.1 beta versions have been very stable, others have been quite unusable, but the next improved version usually comes whithin a day or two (this is to be expected in beta development). Squid-1.1.0 and above has proved to be stable and reliable.

Squid use the Internet Cache Protocol (ICP) version 2 to cooperate with other proxy servers. There are several slightly different versions of ICP around (one in Harvest 1.4, another in Squid, and a third in Harvest Cached-3.*). Although Squid communicates best with its own ICP version it can cooperate with the other versions. Due to the ICP protocol, the ability to ignore unreachable proxy servers, and the ability to use other (non-ICP) servers as parents, Squid is an excellent choice for creating a mesh of cooperating proxy servers.

Squid is in compliance with all the criteria stated above for a Web caching system.

But cooperating Squid servers need to have approximately the same idea of how long an object should be cached. If cache A only stores an object for 2 days, but cache B stores it for 2 months, cache A will probably get a stale copy from cache B the next time it asks for the object. This problem will probably be solved by a change in the ICP protocol soon.

Squid requires a lot of memory. It keeps an indexed list of everything on its disk cache in memory, so if you have lots of disk-space you'll need lots of memory. It is also possible to reserve a bit of memory to use as a very fast cache for popular objects.

UNINETT have made an easy to install package of Squid called SamSquid. This package is available for the SAMSON machines (hp-ux 9.05 with our own maintenance system for software) and Linux (for 2.0 kernel ELF systems). More information about this package will be available shortly.

SURFnet's service pages for more information on the SURFnet caching mesh.

2.4 Other proxy servers

The first widely deployed Web cache server was CERNs cache server, a first-generation Web cache server with several shortcomings [5]. This server is no longer developed.

The Web Consortium is implementing Jigsaw [9], a "proof of concept" Web cache server. It is implemented in Java and supports both HTTP/1.1 and ICP.

Cached-3.* has support for ICP (in a different flavor from Squid) and is a good alternative for building a Web cache mesh for those who prefer commercial software with support.

WCol [10] is a prefetching Web cache server with limited support for ICP.

There is a number of Web cache servers without support for ICP. These are not well suited for building Web cache meshes.

2.5 Configuring a Web cache server for heavy traffic

A Web cache server which expects heavy traffic should be configured with this in mind. The server software usually has features that are convenient, but not strictly neccesary. Many of them will slow down the server, and should be avoided.

It is possible (and sometimes the default) to log the domain-names of connecting clients in most Web cache server software. But this is not recomended on a busy server. Logging the domain-names will slow down the service, and it is very easy to do a DNS lookup on these names when the logs are processed. Not logging the connecting clients names will force you to configure access-control on the cache server by IP numbers, but this does not make configuration more difficult.

Please note that DNS is not eliminated as a single point of failure if you do not log hostnames. DNS is still needed to resolve hostnames in the URL's. Because of this, it is very important to use a stable and fast DNS service. If you have trouble with your DNS service, you should consider starting a caching-only DNS server which serves only the Web cache server. You can run the DNS server on the same machine as the Web cache server, but make sure you have enough memory to do this without much swapping.

A Web cache server gives you the possibillity to log more than just the accesses. Logging of user agent, mime-headers, and special debugging info are nice to have, but they are not crucial. Unless you really need this information you should turn off these logs. Usually you will only need this information for special projects, and then you can turn them on for the duration of the project.

It is also a good idea to buffer the nessecary logging. This will reduce the number of disk-writes. Some server software is also able to do identd lookups, but you should not use this unless you really need it.

The rest of this document is largely based upon our experiences with Squid, and Squid specific issues are set apart this way in the rest of the document.

Henny Bekker, Lars Slettjord, Ingrid Melve

Last modified: Mon Mar 24 17:48:49 1997 Web caching architecture, clients [DESIRE] [DESIRE Web cache] [Web cache Architecture]

3 Clients

Your clients
3.1 Disk cache in clients
3.2 Netscape client configuration
3.3 Configuration of other clients

Checklist

We recommend that you

configure all clients to use your first level Web cache
if you use Netscape or Internet Explorer, set Automatic Proxy Configuration to avoid single point of failure

Be aware that only Netscape Navigator and Microsoft Internet Explorer offers a fallback mechanism (more browsers in beta).

You need to configure all the clients of all users to use your Web cache.

Clients (apart from Netscape Navigator 2.0 and higher, Microsoft Internet Explorer 3.0a and higer, and NCSA Mosaic 2.7b5) will not fail gracefully when the cache goes down. They fail in spectacular ways, causing the end user to have to manually turn off the Web caching. This should not be blamed on the end user, but on browser implementors.

Users will not change back unless they are given an incentive to do so, stick or carrot. A stick may be that you turn off access to port 80 (but if the proxy goes down you are then unable to use the Web) or bill users for traffic, a carrot may be access to certain resources; e.g. the University of Tromsø provides access to Encyclopædia Britannica exclusively through their Web cache.

A survey conducted in the fall of 1996 by ICM Poland, for the Terena Taskforce CACHE shows that around 18% of the cache administrators block access to port 80.

3.1 Disk cache in clients

You will probably want to turn off disk caching in web clients if their environment is a LAN connected directly to the Internet, and they share a common network disk for storing their cache on.

Dial-up users (modem, ISDN) are encouraged to keep a disk cache in order to save download time. Their most severe bottleneck is normally their connection to the ISP. Therefore, they should have a Web cache on their side of the bottleneck.

3.2 Netscape client configuration

Users with Netscape browsers should use Netscape's Automatic Proxy Configuration [APC]. This provides a gentle fall-back mechanism if your Web cache ever goes down and may connect you directly to servers. The client will check back to see if one of the cache servers becomes available again. When this happens it will start using it. All of this happens fast, and is completely transparent for the user.

To start using Netscape's Automatic Proxy Configuration you'll have to do this:

Create a file named something.pac and put it on your Web server
Contents of the file something.pac:
```
function FindProxyForURL(url, host) {
  if (isPlainHostName(host) || dnsDomainIs(host, ".domain.tld"))
    return "DIRECT";
  else
    return "PROXY yourproxy.domain.tld:3128; 
            PROXY next.proxy:port; PROXY next.proxy:port; 
            DIRECT";
}
```
The first part will configure the client to connect directly to Web servers inside your domain. The second part will configure the client to connect to the first Web cache server for documents outside your domain. If that server fails it will automatically try the next, and the next, until it has exhausted the list of available cache servers. Then it will connect directly to the source.
The client will check back to see if one of the cache servers becomes available again. When this happens it will start using it. All of this happens fast, and completely transparent for the user.
Enable the Web server for MIME-type application/x-ns-proxy-autoconfig for the extention pac. In the Apache/NCSA servers this is done by adding the following line to the mime.types configuration file:
```
    application/x-ns-proxy-autoconfig       pac
```
In the Netscape Communications server (and probably also Enterprise and FastTrack) it is done by adding the following line to the mime.types configuration file:
```
    type=application/x-ns-proxy-autoconfig  exts=pac
```
Configure all your clients to use http://your.web.server/path/something.pac in Options/Network Preferences/Proxies

3.3 Configuration of other clients

Use standard proxy settings to your Web cache.

With Internet Explorer 3.0a (and higher) you may point the client at the same configuration script as Netscape.

Most of the others have no fallback mechanisms, if the first level Web cache goes down the user have to turn off Web caching manually to get access to the Web.

Some advice on how clients are configured to use a cache server:

Lars Slettjord, Ingrid Melve

Last modified: Mon Mar 24 17:58:15 1997 Web caching architecture, location [DESIRE] [DESIRE Web cache] [Web cache Architecture]

4 Location

4.1 First level Web caches
4.2 Upper level Web caches

Checklist

Your location
We recommend that you

always place Web caches close to where the traffic flows
place caches on your side of network bottlenecks

First level cache
We recommend that you place your first-level Web cache

as close as possible to your external Internet connection
close to users
if you have several first-level: where networks come together

Upper-level Web cache
We recommend that you place your upper-level Web caches

where networks join
on or close to a national Internet exchange point
on or close to an international Internet exchange point

Location of Web caches should be a function of network topology and traffic flow.

Criteria for location of Web cache servers

Place Web cache servers on the users side of a bottleneck
Place Web cache servers close to the flow of traffic
Place a local Web cache server close to your Internet connection
Place Web cache servers where more networks join together

Web Cache servers should be placed as closely to the actual flow of webtraffic as possible, at the edges of the network using the cache. This means that the natural place for a national top-level cache neighborhood is at the international connection (particularly since data locality in national networks is low), preferably at the same place as the national Internet exchange point. Webtraffic flows from server to client, so the first-level cache should be placed close to the client and upperlevel caches should be placed upstream. The top-level cache for a specific server should be placed as far upstream as possible.

First level caches are typically placed on campus, close to the institution's Internet connection. Next-level caches are placed at the regional level. Each level of caching adds latency in case of a cache miss, and more than 3 level of caches should be avoided.

More caches may cooperate to spread load and increase hit levels as well as redundancy. The most obvious problem currently annoying the users of the World-Wide Web is that of network congestion [Baen96]. Models of cooperation is largely expanded in HTTP/1.1 with support for multilevel caching, with a number of new caching headers coming into use. Parts of HTTP/1.1 are implemented in caches today. Origin Web servers are not expected to upgrade their HTTP version until summer 1997 at the earliest.

4.1 First level Web caches

The location rules of the first level caching server (the institution caching server) are the same as for the top-level caching servers. There should be no differences except for the fact that for the top level caching server there is a problem with multiple Internet service provides (ISP) that arises form the fact that a Web caching mesh spanning more than on autonomous system may violate routing policies.

Place a first level Web cache close to your users, on the same LAN if possible. If you are running one first level cache, place it as close to your Internet connection as possible. You know that Web traffic flows through your Internet connection on its way to you users.

Firewall solutions are not discussed here. If you have a firewall you need to take care of security aspects that are outside the scope of this document. Placing the Web cache server outside your firewall may be a solution if you serve documents from your domain to external costumers. If you want Web caching primarily for your employees and students you may want to place the cache server on or inside your firewall.

4.2 Upper-level Web caches

Web caches should be placed where several networks join together [Bekker96], on campus or at an access point or Internet exchange point.

If there is a national Internet exchange point, this is a good location for Web caching. Administrators should either agree on running a common Web cache, as has been successfully done in New Zealand [Neal96], or have agreements between Web caches placed close to the exchange point. A significant amount of traffic enters and leaves a network at Internet exchange points, and since you want to place your Web caches close to the traffic flow, this is a good place. Another factor that makes an Internet exchange point a good place for upper level Web caches, is that traffic exchange agreements are already in place and you may not need to draft another agreement on Web traffic exchange. These arguments also hold true for international Internet exchange points. Several Web caches around an Internet exchange point will normally be set up as neighbors, unless one wants to route traffic through another network, which will normally not be the case.

As long as a Web cache serves only local users at one location, network topology is not an issue. When you start to build meshes and hierarchies, topology comes into play. If you locate the servers of your meshes without consideration for the topology, you may end up generating more network traffic than without Web caching. Some links are critical. One prime example is inter-continental links, another is international links. One reason for this is the price; such links are normally very expensive, and international bandwith is not plentiful.

Scaling is one more issue where topology knowledge is needed in order to build services and meshes of servers. An ISP is likely to know both what links will be upgraded and where bottlenecks are going to persist - and ISPs might be able to influence link upgrades.

Within an area where one language is used, it is important to be aware of the end users' need to have easy access to Web documents in their own language and how this influences the traffic pattern (traffic in Norway between Norwegian ISPs are increasing, and 15% of the traffic in UNINETT is over the national Internet exchange point, only 30% is internal to UNINETT).

Ingrid Melve

Last modified: Tue Mar 25 10:13:25 1997 Web caching architecture, mesh [DESIRE] [DESIRE Web cache] [Web cache Architecture]

5 Mesh

5.1 ISP-wide neighbors/parents
5.2 International neighbors/parents
5.3 Institutional neighbors/parents
5.4 Configuring Squid in a mesh
5.5 Naming strategy for a ISP-wide Web cache

Checklist

We recommend that you

allow access to your Web cache server only for those servers you have set up an agreement with
always ask the administrator before you set a Web cache as you parent or neighbor
go to NLANR registration database, find potential neighbors and parents in your area
contact you Internet Service Provider

Caches and cache administrators need to cooperate in building and joining a national or network wide Web caching mesh.

Meshes are highly dependent on network topology. The main point is to build a mesh to avoid links with high costs, where costs may be download time for documents, economic cost, heavy load or long response times.

The hit rates will decrease on top-level servers. This is to be expected since servers on a lower level cache the documents they fetch from it, and it may be a while until they request the same document again. A toplevel server needs several different children and more traffic to be effective, as the popularity distribution of Web documents is a Zipf-like hyperbolic distribution [Cunha95].

Simple Mesh

The figure shows a simple Web cache mesh with two servers strung one after another. The width of the connections indicate both how fast a user can connect and how traffic flows in a well implemented mesh. Even in a more complex mesh with more servers and different relations between servers, the client relates to one proxy configuration. The mesh is a black box with hidden complexity for both client and origin Web server.

Several Norwegian colleges have distributed campuses, where sites may be 60-80 km apart. These different sites are normally linked with medium-capacity links (1-2Mbps), and heavy load may be a problem during work hours. Their Internet connection is typically 256 kbps. A parent located at the central Internet access point may reduce the load on the external connection. Neighbors located at each campus site may cooperate to share load. Misses (documents not found in cache) are resolved through parents, not through neighbors/siblings in Squid. This is a way to avoid loops. Configuration of neighbors and parents are handmade, hopefully there will be a better way to do this shortly (manual configuration does not scale).

5.1 ISP-wide neighbors/parents

Due to routing considerations and routing policies, one single national parent is unlikely. The top-level of the caching mesh is most likely on the ISP level. Some alternatives for an ISP-wide mesh are

One single top-level server.
A neighborhood of several servers that together make a top-level.
Separate specialized Web caching servers, one for each scope of Web space, e.g. today you may want one server for .com domain, and all requests for documents in the .com domain is directed to that server.
One caching server at each end of the overloaded link may be the best solution if the ISP has an overloaded connection to the world [Dias96].

If there is a single top-level server, this needs to be dimensioned for the considerable load it will take. Load is a function of the number of users. Decide whether or not to allow end users to access your toplevel caching server.

5.2 International neighbors/parents

Normally a Web cache mesh is terminated inside the ISPs network, as special care is needed not to violate routing policies for meshes spanning several autonomous systems (AS). At this stage of development in Web caching, establishing bi-lateral agreeements on Web caching between ISPs is reccomended if inter-ISP cooperation is needed. For academic networks, the need to consider the Acceptable Use Policy is of special concern.

An introduction to routing issues is found in [ripe-081] and [ripe-181]. An autonomous system (AS) is a group of IP networks having a single clearly defined routing policy which is run by one or more network operators. Inside an AS routing is done using one or more Interior Routing Protocols (today: OSPF). Routing decisions here are normally derived from technical parameters like topology, link speeds and load. When ASes exchange routing information with other ASes using Exterior Routing Policies (today: BGP4), routing decisions are often based on policy rather than purely technical parameters.

Exchange of routing information between ASes is subject to routing policies, where routes are announced only by those willing to route traffic to a network. This information may or may not be accepted by the recipient of the announced routing information. Only if the announced routes are accepted does traffic flow, and in order for traffic to flow in both directions routes between the two fringe networks must be announced and accepted in both directions.

Multilevel Web caches are two or more cooperating Web caches, often arranged in a hierarchy where a first-level Web cache is close to the user (local Web caching) and upper level Web caches have first-level Web caches as their clients (regional or national Web caches.)

Web cache meshes are several cooperating servers that together form a mesh doing multilevel Web caching. A mesh may span a university or a network, or several networks.

The lesson to learn from traditional routing is that purely technical routing decisions and policy decisions are different and may require two different sets of configuration and setups in order to work in an Internet. If Web caching meshes span several ASes special consideration must be given to routing policies.

We should stop treating cooperative Web caching as interior routing and start paying attention to exterior routing policies. Today's practice cannot continue if Web caching becomes widespread. Today, Web caching does not constitute a threat to "normal" traffic flow as few meshes span several ASes. As we aim to change this, we will have to find a way for Web caches to communicate policies.

When a mesh aggregates Web traffic, this aggregated traffic should be dealt with in close cooperation with the ISP. The two main reasons for this is topology knowledge and routing issues.

5.3 Institutional neighbors/parents

Several of the Norwegian colleges have distributed campuses, where sites may be 60-80 km apart. These different sites are normally linked with medium-capacity links (1-2Mbps), and heavy load may be a problem during workhours. Their Internet connection is typically 256 kbps. A parent located at the central Internet access point may reduce the load on the external connection. Neighbors located at each campus site may cooperate to share load.

5.4 Configuring Squid in a mesh

General configuration

When you cooperate with other Web cache servers, you need to agree about how long an object should be in the cache. If one of the siblings caches objects for 3 months, and your cache only caches these objects for 14 days, you will sooner or later get objects from your sibling which violate your own rules on how long these objects should be cached. This is a problem with both the HTTP/1.0 and the ICP protocol, which hopefully will be solved soon. But as of now an agreement on this issue between parents and siblings is the best solution. Implementing HTTP/1.1 in Web caches will solve this problem, and some progress has been made in the later versions of Squid. In Squid-1.1b11, an age value and checks for freshness replaced the previous Time To Live model. And the HTTP/1.1 "Cache-Control: Max-age=nnn" was added in version 1.1b17.
To protect the server machine against intruders the Web cache server should be run in a chroot environment. There are no known security problems in Squid as of now, but this may change. And Squid is very easy to install in a chroot environment. Just compile it using the -static flag in gcc, install some files in the chrooted dev and etc directories, install Squid there also, and just start it.

Security issues for the first level server

Mesh setup

A first-level Web cache server within an organization can be a security risk unless it is correctly configured. An organization (outlined in the figure) will almost always have some internal Web objects. The usual method used by Web server administrators is to protect these objects (by IP-number or hostname) so only hosts from the organization's domain are allowed to fetch them.

A first-level Web cache server is almost always placed within the organization's domain. And if this Web cache server cooperates with Web cache servers outside the organization's domain you have to do something to protect your internal objects. If not, a cooperating Web cache server will be able to connect to the organization's Web cache server and request a protected (solid red in the figure) object. The organization's Web cache server will then request it from the origin Web server, and it will get the document because the origin Web server will see this as a request from within the local domain. The object will then be passed on to the cooperating Web cache server outside the organization.

This can be solved in several different ways. The most practical solution, as of this writing, is to deny other Web cache servers access to internal services through the organization's Web cache server. To avoid access denied messages, your partners will have to configure their Web cache servers not to go through your Web cache server when fetching documents from your domain. This makes sense since contacting your source should be just as fast as contacting your Web cache server (unless you have some very slow internal connection's, and then you should move or mirror important servers in front of the slow connection).

Denying other Web cache servers access to internal services through your Web cache server can be done like this in Squid:
acl yourdomain src 129.242.0.0/255.255.0.0 acl yourhttp url_regex ^http://.*\.your\.domain ^http://129\.242 acl yourftp ftp_regex ^ftp:///.*\.your\.domain ^ftp://129\.242 acl partner1 src 128.39.2.22 129.177.12.12 ... acl partnerN src 193.213.103.91 http_access allow yourdomain http_access deny yourhttp http_access allow yourpartner1 ... http_access allow yourpartnerN http_access deny all

To avoid access denied messages your partners will have to configure their Web cache servers to not go through your Web cache server when fetching documents from your domain. As mentioned above this should normally be just as fast as contacting your Web cache server.

Configuration parameters for another Squid Web cache server to use your Web cache server as a sibling or a parent:
# Using you as a neighbor cache_host your.web.cache.server sibling 3128 3130 cache_host_domain your.web.cache.server !.your.domain # Using you as a parent cache_host your.web.cache.server parent 3128 3130 cache_host_domain your.web.cache.server !.your.domain

If you want to force someone to only use you as a sibling (only allow them to fetch HITS) you can use the miss_access parameter in your own configuration file.

Security issues for the ISP level server

A ISP (or national) top-level cache server open for general access could very soon be swamped with access requests. Many of these accesses will be in direct conflict with routing policies, and this could give you trouble with your own ISP.

Access should be granted only to neighbors/siblings and parents with which you have set up an agreement. In this agreement, you can ensure that your routing policy is followed.

Restricting access to your Web cache service can be done like this in Squid:
acl your-domain src 129.242.0.0/255.255.0.0 acl partners src 193.213.103.91 129.177.12.12 acl sibling-partners src 130.237.222.70 192.87.46.3 194.179.100.4 http_access allow your-domain http_access allow partners http_access allow sibling-partners http_access deny all icp_access allow your-domain icp_access allow partners icp_access allow sibling-partners icp_access deny all miss_access allow your-domain miss_access allow partners miss_access deny all
This will allow your regular partners and your own domain to fetch both hits and misses. Your sibling partners will only be allowed to fetch hits, and the rest will be denied access to your service.

5.5 Naming strategy for a ISP-wide Web cache

A suggestion today when setting up a ISP-wide Web caching mesh in UNINETT would be to make the domain cache.uninett.no and name

www.cache.uninett.no for endusers (to seed the cache when starting up, and to have a service for dial-up users. This may be superfluent if you have many first-level caches operational and only want to provide service to other Web caches)
world.cache.uninett.no for all documents
no.cache.uninett.no for caches that request documents from the no-domain
com.cache.uninett.no for documents in the com-domain
edu.cache.uninett.no for documents in the edu-domain

and possibly

uk.cache.uninett.no for UK documents since they are popular
nordunet.cache.uninett.no for Nordic documents (since they are close, netwise), with se.cache.uninett.no and fi.cache.uninett.no and dk.cache.uninett.no as CNAMEs
world2.cache.uninett.no as backup for world

In the beginning all names would point to the same server. This makes scaling a smaller problem, and makes it possible to have specialized servers for certain popular domains (which may rise hitrates and you get to chose those that are most important for your users). It is possible for a local Web cache administrator to do a lookup to determine what domains have specialized caches.

Growing from one (as a start) to several servers would be less painful if this was implemented from the start, and it would be possible to start out with one single server. Implicit in this proposal is that there is a backbone with sufficient bandwith, if not you will be better off with several servers as a start.

Info about this config should be accessible at a central database (today in the database at NLANR). All children of the upper level cache should have access to this information, and providing child caches with standard config files is encouraged.

Operational experience with this approach is not yet available.

Ingrid Melve, Lars Slettjord

Last modified: Tue Mar 25 10:07:29 1997 Statistics, Web cache architecture [DESIRE] [DESIRE Web Cache] [Web Cache Architecture]

6 Statistics

6.1 Log files
6.2 Log file analyzing
6.3 What to analyze for
6.4 Presentation of statistics

Checklist

We recommend that you

keep logs unavailable unless anonymized
rotate and process logfiles once a day (or more) due to their large size
keep the large size of the log files in mind if you write programs to analyze them
keep the privacy of your users in mind when you publish results of log file analyzing
find out what kind of statistics you want from your server, or you'll drown in numbers

Aggregated statistics is useful to both your users and other Web cache managers in your mesh, make it available to your users.

6.1 Log files

We recommend that you

keep logs unavailable unless anonymized
rotate and process logfiles once a day

Laws on privacy differs from one country to another, make sure that the caching system does not violate the privacy of users. Be very careful with your logs, avoid giving out information about your individual users and remember that usage patterns may be sensitive information (i.e. on what material users access).

Logfiles on a busy server can grow very large. A server with about 300 000 requests a day will produce an access-log file of about 30-40 MB. To save disk-space the log files should be rotated, compressed and stored once a day.

And you never know when somebody, by intent or accident, is going to make heavy use of your server. We've seen examples where a script has downloaded the same few files over and over again for a whole day. In this case the server was able to take the load, but the disk where the log file was stored was filled, and we lost a lot of log data. So if analyzing the log file is important, you should have some extra space on your log file disk.

6.2 Log file analyzing

A log file analyzing program must be written with the large size of the log file in mind. It should not try to read the whole log file into memory at once. If it does you may very well use both all the available memory and swap space.

Analyzing large log files may take both considerable time and CPU power. If the server is very busy, you should consider analyzing the log files on a separate machine to avoid disturbing the cache server.

6.3 What to analyze for

There are a lot of parameters you may want to analyze for. Here we will mention the ones we have found significant when analyzing log files from a Squid server. Most of them apply in general for all Web cache servers.

Request type

To measure how efficient the Web cache server is you will want to analyze for the following parameters:

Hit: When the requested object is served directly from the cache.
If Modified Since (IMS): When an IMS request to the origin server confirms the freshness of the object. This will be done on stale objects, and when a client forces the server to check a fresh object.
Refresh: When the client forces the server (Pragma: no-cache) to fetch the object from the origin server regardless of the state on the cached copy.
Miss: When the object has to be fetched from parents, siblings or the origin server because it doesn't exist in the cache, or is stale.
Error: When the server is unable to serve the request for some reason.
Denied: When the server denies to serve the client.

These parameters should be counted in both accesses and bytes served to the clients. They should also be counted for both the HTTP and the ICP protocol. Please note that refresh is not applicable on ICP requests, and you may also want to split the type of hit on these. If the object is small enough (it fits into a udp-packet), the ICP protocol allows the object to be included in the reply on a hit. If it is too large it will have to be fetched by a separate HTTP request.

Traffic and usage

These parameters will give you an idea of how many users you serve, how much time your server uses to process requests, and how busy your server is.

Hosts using the server: You should count how many different hosts (IP-numbers) that use your server. You can also count how much traffic (in accesses and bytes) they cause. But in this case you should consider the privacy of your clients. If you don't need these numbers per host you should count per domain only.
Elapsed time for requests: NOTE! This only applies to HTTP requests. It is a good idea to count how many requests that are served within a given time. I.e. you count how many requests are served within 2, 4, 16, ... seconds. This will give you an idea of how fast your users are served. It is a good idea to use a logarithmic scale, preferably a log(2) scale.
Connections per second: This will give you an idea of how busy your server is.

Parents and siblings

These parameters will tell you how effective your parents and siblings are.

Hierarchy: This will tell how the request was resolved. I.e. if it was sent to the origin server, or resolved through a parent or sibling. It will also give you hints of why the request was routed this way (source fastest, parent/sibling hit, ...).
Family: If the request was resolved through a parent or a sibling you should gather statistics on how well the specific parent or sibling performs. You can do that by ignoring all the hierarchy tags that indicate a connection directly to the origin server, and count the resulting hierarchy tags by host name. This will give you an idea of how much traffic that the different parents and siblings handle for you. But it will not give you things like the hit rate for a particular parent/sibling. By just analyzing the log file you are not able to find out how many requests that were sent to a particular parent/sibling. You can only count how many requests it answered. If you want to know how many requests it got you'll have to incorporate the the rules for contacting parents/siblings given in the configuration file for the Squid server. If you do this it will not be easy to analyze old log files since the configuration may change.

DePStat (Desire Project Statistics)

The Desire Project has made some experimental scripts for analyzing native format Squid logs available. They can be fetched from the DePStat home. These scripts implements the requirements stated above.

6.4 Presentation of statistics

Example: UNINETT hit rates

[UNINETT top level hit rate] UNINETT
[Tromsø first level hit rate] Tromsø
UNINETT
Tromsø
The figures shows the hit rates of the first-level cache at the Tromsø University and the UNINETT top-level cache for the period 21. Oct - 1. Dec 1996, the drop in hit rate on 25. Nov is due to a total cleanout of the UNINETT cache. More statistics are available for the UNINETT and Tromsø University Web caches.

Typical total savings for the two-level cache system are around 50% on the number of connections made to the origin server, and around 55% for bytes downloaded.

Lars Slettjord, Ingrid Melve

Last modified: Tue Mar 25 10:14:18 1997 Web caching architecture, policy [DESIRE] [DESIRE Web Cache] [Web Cache Architecture]

7 Policy

7.1 Restrictions
7.2 Web Cache Server
7.3 Internet Service Providers
7.4 Good relations

Checklist

We recommend that you

register your server
contact your Internet Service Provider before setting up parents outside your network
observe your Acceptable Use Policy (if you have one)
keep logs and statistics unavailable unless anonymized
If you are an Internet Service Provider, we recommend that you

establish peering agreements with other providers, this is very important within a country/language zone
investigate your routing agreements with connection providers

A top-level cache server should cache everything it can (unlike low-level cache servers which normally avoid caching local domains). This will give both national/ISP users and neighbors/siblings in other domains the best possible result.

7.1 Restrictions

Always ask the administrator before you set a Web cache as your parent or neighbor. Do not regard open access to a Web cache server as a carte blanche to do whatever you want. Open access may be a misconfiguration or intentional, ask the adminstrator.

Laws on privacy differ from one country to another. Make sure that the caching system does not violate the privacy of users. Be very careful with your logs. Avoid giving out information about your individual users and remember that usage patterns may be sensitive information.

Restricting access to Web pages through the cache must happen at the lowest possible level, close to users. An upper level cache will normally have a few restrictions on access, unless there is a need to impose access restricitons at a high level, as is done in Singapore where legislation places restrictions at the ISP level for the entire country. Censorship should be carefully aimed at the target group of users, if you put restrictions on upper level caches these may not be aimed as accuratly.

Do not offer access to your Web cache for the world outside your organization unless you know what you are doing. Sudden heavy load may take down your server (this has been experienced: the Norwegian branch of a major international company suddenly pointed all Web access through a small Web cache server set up to serve a user population of 14, which caused trouble until the company was shut out by reconfiguration of the server)

7.2 Web Cache Server

Register your server at NLANR or at a national registration service [NLANR]. This will help you get the best possible overview on how many are using Web caching systems in your part of the network.

Aggregated statistics are useful to both your users and other Web cache managers in your mesh. Make such statistics available.

7.3 Internet Service Providers

ISPs should run Web caching systems that their customers can join. Typically, ISPs should provide an end-user Web cache (for dial-up users) as well as upper-level caches to parent for first-level Web caches. Proxy configuration scripts and sensible configuration of Web clients should be provided to network administrators and end users.

ISPs normally have established peering with other ISPs at a national and international level. Make sure that you do not violate agreements on routing. Remember that Web caching heavily influences the flow of Web traffic. (That is why you want to do it!)

ISPs may benefit from having agreements with other ISPs on Web caching. These may follow established agreements on peering and traffic exchange.

7.4 Good relations

To have good relations with other caches

Agree on a standard configuration within a group of cooperating Web cache servers.
Publish your policy on relations, who you are willing to be a parent or neighbor/sibling for. (This is the Internet and you get to choose your neighbors, children and parents.)

A good child cache always asks permission of its parents before using them, and it chooses its neighbors and parents carefully to make sure that they are optimally placed in the network topology. A long chain of parents is not a good idea. Limit the hierarchy to a maximum of 3 levels (preferably two).

How to be a good child

Choose you neigbors and parents carefully, make sure that they are placed in the line of information streaming into your Web cache. If you look at yourself as sitting by a stream of information, choose your parents upstream. Make sure that you use parents that are actually upstream of you, AND downstream of the information you want. A first level Web cache for an UNINETT institution should use as its parent an upper level Web cache to get documents from outside Norway only if it knows that the upper level cache actually does provide access to documents outside Norway. It makes no sense for a Web cache located close to Oslo to use a Web cache in Tromsø as its parent for documents outside Norway when the International connection is in Oslo (and Tromsø is far away, even in networked terms).

Always ask for permission from your parents before you use them.

Beware that time to live for documents at your parent influences your cache. It might be a good idea to agree on a standard configuration within a group of cooperating Web cache servers.

How to be a good neighbor

Do not configure your server to be wide open unless you know you can take the load this may lead to.

Publish your policy on who you are willing to be neighbor to (remember that this is the Internet and you get to choose your neighbors).

Beware that time to live for documents at your neighbors influences your cache.

How to be a good parent

Publish your policy on who you are willing to be parent for (remember that this is the Internet and you get to choose your children).

Long chains of parents is not a good idea, try to limit the hierarchy to 3 levels. Publish your neighbor/sibling and parent configuration.

Do not configure your server to be wide open unless your general routing agreements allow you to route traffic from third parties. Web traffic should not break general routing agreements. One of the consequenses of this is that the UNINETT top-level cache may not act as a parent for non-Norwegian pages without UNINETT breaking the contract with NORDUnet (provider of international connections).

Web caching is application level routing, and as Web traffic forms a considerable part of the total traffic (mesurements in UNINETT indicate that 50% of traffic flowing into the network is Web traffic, mesurements in UK JANET indicates 75-90% Web traffic on international links) may influence the traffic pattern.

Ingrid Melve, Lars Slettjord

Last modified: Mon Mar 24 19:24:04 1997 Web cache architecture, hardware [DESIRE] [DESIRE Web cache] [Web cache Architecture]

8 Caching server hardware considerations

8.1 Introduction
8.2 Different types of caching servers
8.3 Web server sizing
8.4 Disk
8.5 Performance enhancements
8.6 Tuning system parameters
8.7 References

Checklist

Toplevel server

We recommend that you either have

more than one big strong UNIX server (dedicated), if you have one, be prepared to buy another
or a farm of smaller specialized servers (dedicated)

Local server
Depending on your user community we recommend

Linux, or NetBSD/FreeBSD if you have a reasonable amount of users (low cost equipment)
any UNIX-system with

good I/O
lots of disk if you have many users
lots of memory if you have lots of disk

This section looks at the hardware issues involved in caching. Note that a caching server acts like a normal Web server for the clients that connect to it. Furthermore, it acts as a client as seen from the origin Web server. Therefore, regular Web server sizing and tuning aspects apply to caching as well.

Note: Web caching is a rapidly evolving area and interest in and deployment of caching systems grows very fast. Therefore, the facts, figures, and recommendations in this document reflect the current state-of-the-art, i.e. September 1996. The information contained herein has most likely a limited TTL (time-to-live).

8.1 Introduction to caching server hardware

A universal recommendation for a caching system does not exist. So much depends on the local situation. Factors that obviously influence the choice of hardware are budget, amount of available manpower for management (two systems require more attention than a single one), preference of a specific vendor or brand, necessity for caching (saving bandwidth, blocking access to selected sites, better performance and availability of documents), etc. This section lists some issues to consider when setting up a caching server and gives references to current examples.

8.2 Different types of caching servers

One of the aspects that determines the dimensions of a caching server is its place in the mesh, the hierarchy of caching servers. A departemental server, for example, will usually be co-located with their Web server with some extra disk space. An organisation's caching server could be co-located with their Web server as well, or run on a separate system, especially when a firewall is being used. In the latter case the caching server will be inside the firewall, the (public part of the) Web server outside. A national or ISP-wide caching server should run on one or more dedicated systems.

In most situations one will start off with a small caching server that will expand as the use of the service grows, unless all users are forced to use the proxy/caching server, e.g. like in Singapore. The evolutionary scheme allows for adjustments in hardware and software (configuration!) as the load increases.

For a local server with a reasonable amount of users, there are good experiences with Linux as a lowcost solution. Any UNIX-system with good I/O and lots of disk, if you have lots of users (and lots of memory if you have lots of disk) will serve. Squid has been ported to OS/2. Solutions for WindowsNT is on the workbench, none of these support collaborative Web caching today.

The server should not be used much for interactive work. If the squid-process starts swapping you'll get a very bad performance, so you should have enough memory. How much memory squid will use depends much on how much disk you have, and how much memory you configure as memory-cache.

8.3 Web server sizing

As a caching server acts in some respects like a normal Web server, most considerations for sizing a Web server apply. There are some exceptions, however. A caching server won't execute any cgi-scripts for example. The same holds for encryption (e.g. SSL). Or, put differently, a caching server only serves static HTML files and no dynamic content and does no back-end processing. This influences especially the CPU requirements. One exception to this is if the caching server does redirection of URLs, this is done by a script and may place additional CPU requirements on the server.

When taking the previous exceptions into account, your vendor will be able to advise on your hardware requirements. Most vendors have papers available on-line or even forms where the expected number of requests, LAN and WAN bandwidth, amount of disk space, etc. can be filled out resulting in a suggestion for a configuration. Here are some references:

You may find suitable configurations in the Terena survey of caching systems in Europe in the survey results.

Another issue to consider is the way a caching server deals with its processes. In Ref. [Overview] the differences between the CERN, Netscape and Harvest (now Squid) caching servers are listed with respect their operation: pre-forking, multi-threading, non-blocking I/O, etc.

When the load on your caching server gets too high for a single system, a configuration with several servers can be created. Options are: several machines with a shared set of disks, sharing the same cachedisk is not possible with Squid, several machines acting as each other neighbours (Ref. [Singapore]), or parents (Ref. [NLANR]), or a (geographically) distributes set of machines (Ref. [HENSA]). In order to make the set of servers into a single virtual system, techniques like Round-Robin DNS (Ref. [DNSload]) or Local Director are being used. Always honor the TTL values of DNS when a client makes a lookup.

8.4 Disks

Disks are important as well. There are several types of disks (SCSI, SCSI-II, SCSI-WIDE, SSA), each with its own performance and thoughput characteristics. The configuration of the disks also impacts the performance. Striping, mirroring, or RAID are the buzzwords here. As data-integrity and data-availability are not essential in case of caching (the disks will fill itself again after a crash, only reducing the hit ratio), mirroring and most RAID solutions are not recommended. Consult your supplier for specific guidelines.

In most cases cache performance is I/O bound, and having multiple disks spread over several controllers may increase performance, especially for large caches. All major Web cache products have support for multiple disks. On some hardware systems striping disks is a good alternative when using multiple disk (not tested in UNINETT). A disk used for Web caching should not be partitioned into several partions, as the disk is most efficient when not partioned as compared to several partions on the same disk. It is recommended to use a full partition for your caching disk space. Since you do not need to backup your cache (it will happily fill itself up again) putting it on a partition dedicated only to cache is a good idea. Partioning disks should be done with regard to disk size, a huge disk may need to be partitioned, since the number of files in one catalogue may prove to much. With newer version of Squid this is not neccesary since it is possible to configure how many directories it should use on the first and second level in it's swap-directories. If you have a large disk simply add more directories.

If you have several disk or partitions per cache Squid will write to them in a round-robin schedule, thus filling them up pretty even. If you have partitions or disks with different sizes they should have sizes which are a dividend of each other. (I.e if disk a is 500 MB and disk b is 250 MB, you have two cache directoriea on disk a, an one on disk b.)

If your hardware supports striping this may speed up your disk, consider this if you expect many requests and have a fast network connection. RAID may increase stability and speed on you disk and also supports striping, you do not need RAID for mirroring as losing your cache data is not disastrous.

On machines with relative slow disks and low bandwidth (e.g. SCSI disks) its wise to have a limited number of disks per controller (say two) and devide the load on multiple controllers. On most Linux systems this may be of importance as disks on a "normal" PC are quite slow.

The available network bandwidth influences you need for disk I/O. It is no use to have a very high disk I/O (100-MB/sec) when the system is connected to ethernet (10-MB/sec). If storage is done in parallell with reading, you may need more than the available network bandwith for internal use. The disk I/O (single disk) and bandwidth of the I/O controller are connected, e.g. on a SCSI controller of 25-Mhz a maximum of 3 SCSI-disk should be connected as this will keep the SCSI controller from getting saturated.

Total size of disks needed for local and upper level is a function of th number of users (100 MB may be too big for 15 users) and the time the document is cached (which is dependent on network costs and willingness to serve stale documents).

8.5 Performance enhancements

When the right set of hardware has been installed some measures can be taken to enhance the performance of the caching server. For example:

Pay attention to the network connection: how is the caching server connected to the Wide Area Internet? Is a (dedicated) ethernet close to the router sufficient? Or is fast ethernet, FDDI, or ATM (with configurable SVC pipes) more appropriate?

Don't run the caching server from inetd, but as a standalone process.

Most caching/Web servers make DNS lookups of client IP numbers, usually for logging purposes. This may decrease the performance of your server significantly. It is recommended to disable DNS lookups and if needed process the IP-numbers in the log file seperately.

8.6 Tuning system parameters

On system level a number of parameters exist that may influence the performance of your caching server. These parameters and especially their values are very system dependent. Therefore, a general recommendation is impossible. Nevertheless, some of them are listed below so you may take them into consideration when trying to improve your system performance.

The parameters include:
somaxconn, sominconn, tcp_keep_timer_in_close, tcp_keepidle, tcp_keepintvl, tcp send, tcp_conn_req_max, tcp_rexmit_interval_min, tcbhashsize, maximum number of processes per "user".

Most vendors will be able to supply you with the patches you need for your specific choice of operating system. These patches are the same that are needed for a Web server (as mentioned above when discussing Web server sizing).

Finally, the caching server may have some parameters that can be fine-tuned, e.g. for Netscape: MaxProcs, MaxThreads, listen queue.

Ton Verschuren & Ingrid Melve 96-09-25
Lars Slettjord 96-11-09

Last modified: Mon Mar 3 15:54:21 1997 Web caching architecture, examples [DESIRE] [DESIRE Web cache] [Web cache Architecture]

9 Examples

9.1 Location
9.2 Relations
9.3 Statistics

9.1 Location

The UNINETT top-level Web cache server is located in Oslo, close to the international connection and national Internet exchange point.

For the universities in the star, connected via high-bandwith lines (Supernett), it makes sense to be neighbors, as the added latency for connections to the other points of the star is (almost) negligible. Whether these institutions use the top-level UNINETT server as parent or neighbor is depending on their caching server. As of this writing, some are children and some are neighbors. We will investigate the optimum configuration for a first-level server when bandwith is plentiful.

These examples are taken from UNINETT, an overview of the network is found at traffic statistics server.

[UNINETT topology] [UNINETT topology, color codes]
UNINETT backbone network

The network is built like a star with the center in Trondheim (21). The international connection and national Internet exchange point are in Oslo (15), one of the star points. From the star points and center, connections in a tree (two-level) go to user institutions. The first level is shown in the figure above. Another local Internet exchange point is being added in Bergen (4), at another of the star points.

The UNINETT top-level Web cache server is located in Oslo, close to the international connection and national Internet exchange point.

For the universities in the star, connected via high-bandwidth lines (Supernett), it makes sense to be neighbors, as the added latency for connections to the other points of the star is (almost) negligible. Whether these institutions use the top-level UNINETT server as parent or neighbor depends on their caching server. As this is written some are children and some are neighbors, and we will investigate what is the optimum configuration for a first-level server when bandwidth is plentiful.

For the colleges at the leafnode it makes sense to have the top-level server as their parent, as their connections lead to the star, and as soon as they reach the star bandwidth is plentiful. Using other leafnode institutions as neighbors is not recommended as their tree connections are slow. The bottleneck for the colleges is normally their connection to the star, and for other institutions the bottleneck is mostly their connection to the tree.

Having one single top-level server places high availability demands on that server, and the redundancy added by having two servers is recommended. This is a question of how much money can be spent (which is a function of how much money you save by adding Web caching).

9.2 Relations

SURFnet relations

These are some of the relations of the SURFnet top-level Web cache server. The lower part of the figure shows relations inside SURFnet, the upper left part shows relations with other Dutch ISPs and the upper right shows the dual parent relations with UNINETT, where UNINETT top-level cache is parent for all documents in the .no domain and SURFnet top-level cache is parent for all documents in the .nl domain.

UNINETT relations

These are some of the relations of the UNINETT top-level Web cache server. Note that Telenor (another ISP) is only allowed a sibling relationship. The Universities (UiO, UiB and UiTø) should be siblings, but at the moment they are not. KTH is used as a parent for the .se domain, and as a sibling for everything else. NLANR is actually several servers, and we use one of them as a sibling for the .com domain.

9.3 Statistics

There are a number of different log analyzing programs around, these figures are generated by some home made scripts.

Figure 5: UNINETT top-level

Figure 6: Tromsø first level

Figure 7: UNINETT top-level

Figure 8: Tromsø first level

The figures shows the hit rates for the first-level cache at the University of Tromsø and the UNINETT top-level cache for the period 21. Oct - 1. Dec 1996. The drop in hit rate on 25. Nov is due to a total clean out of the UNINETT cache. More statistics are available for the UNINETT and the University of Tromsø Web caches.

Typical total savings for the two-level cache system are around 50% on the number of connections made to the origin server, and around 55% for bytes downloaded.

Ingrid Melve

Last modified: Tue Feb 11 15:21:59 1997 Web caching architecture, references [DESIRE] [DESIRE Web cache] [Web cache Architecture]

11 Web caching architecture, references

[Baen96] Michael Baentsch, George Molter, Peter Sturm Introducing Application-level Replication and Naming into today's Web, Computer Networks and ISDN Systems, volume 28, p 921 - 930, 1996

[Bekker96] Henny Bekker, Ton Verschuren, Ingrid Melve Requirements and Recommendations of Web caching services DESIRE Technical Report D4.1, 1996

[Cormack96] Andrew Cormack, UK Advisory Committee on Networking report on Web Caching, University of Wales, Cardiff, September 1996

[Cunha95] Cunha, Bestravos and Crovella Characteristics of WWW Client-based Traces, BU-CS-95-010 (Boston University Technical Report), 1995

[Dias96] Gihan V. Dias, Graham Cope, Ravi Wijayaratne A Smart Internet Caching System, Proceedings INET'96, June 1996

[Gwer95] J.S. Gwertzman, M. Seltzer The Case for Geographical Push-Caching Proceedings 5th Workshop on Hot Topics in Operating Systems, May 1995, Orcas Island, USA

[ICP-draft] Duane Wessels, Kim Claffy Internet Cache Protocol (ICP), version 2 Work in progress

[Neal96] Donald Neal The Harvest Object Cache in New Zealand Computer Networks and ISDN Systems, volume 28, p 1415 - 1430, 1996

[Smith96] Neil G. Smith The UK national Web cache - The state of the art Computer Networks and ISDN Systems, volume 28, p 1407 -1414, 1996

[HTTP1.1] Roy Fielding, Jim Gettys, Jeff Mogul, Henrik Frystyk, Tim Berners-Lee, Hypertext Transfer Protocol -- HTTP/1.1, RFC 2068 (Proposed Standard), 1996

[W3C95] The Web Consortium Problem Statement on Propagation, Replication and Caching 1995

[Woodr96] Allison Woodruff et al An Investigation of Documents from the World Wide WebComputer Networks and ISDN Systems, volume 28, p 963 - 980

[DNSload] Example of load balancing using DNS

[HENSA] HENSA, the UK national caching service, http://www.hensa.ac.uk/cache/

[NLANR] NLANR Caching Project http://www.nlanr.net/Cache/

[Singapore] Singnet's caching service, http://www.singnet.com.sg/cache/

[APC] Netscapes Automatic Proxy Configuration, http://home.netscape.com/eng/mozilla/2.0/relnotes/demo/proxy-live.html

[Squid] Squid http://squid.nlanr.net/

[Cached2] Harvest Cached-2.*http://www.netcache.com/

[Netscape] Netscape Proxy Server 2.0 http://www.netscape.com/comprod/server_central/product/proxy/

[Wierenga96] Klaas Wierenga, Tim Dixon The Desire project - General Information, 1996

[ripe-081] Tony Bates, Jean-Michel Jouanigot, Daniel Karrenberg, Peter Lothberg, Marten Terpstra Representation of IP Routing Policies in the RIPE Database, February 1993

[ripe-181] Tony Bates, Elise Gerich, Laurent Joncheray, Jean-Michel Jouanigot, Daniel Karrenberg, Marten Terpstra, Jessica Yu Representation of IP Routing Policies in a Routing Registry (ripe-81++), October 1994

Hardware

[Digital Alpha]: Sizing a Responsive Web Server
[Hewlett-Packard]: HP
[IBM RS/6000]: IBM RS/6000 Web Server Performance Brief
[Linux]: List of problems with Apache & Linux (most of these also holds for other Web and Web cache servers).
[Silicon Graphics]: WebFORCE Web Server Tuning Guidelines
[Sun]: World Wide Web Server Performance

Ingrid Melve

Web caching architecture

Web caching System Design Checklist

First level cache

Upper-level Web cache

Toplevel server

Local server

10 Executive summary

1 Introduction

Contents

2 Web caching software

Contents

Checklist

3 Clients

Contents

Checklist

3.1 Disk cache in clients

3.2 Netscape client configuration

3.3 Configuration of other clients

4 Location

Contents

Checklist

Your location

First level cache

Upper-level Web cache

5 Mesh

Contents

Checklist

6 Statistics

Contents

Checklist

6.1 Log files

6.2 Log file analyzing

6.3 What to analyze for

Request type

Traffic and usage

Parents and siblings

DePStat (Desire Project Statistics)

6.4 Presentation of statistics

Example: UNINETT hit rates

7 Policy

Contents

Checklist

7.1 Restrictions

7.2 Web Cache Server

7.3 Internet Service Providers

7.4 Good relations

How to be a good child

How to be a good neighbor

How to be a good parent

8 Caching server hardware considerations

Contents

Checklist

Toplevel server

Local server

9 Examples

Contents

11 Web caching architecture, references

Hardware