The Norwegian research network
  
  Search:

5 Mesh

tp://www.nic.surfnet.nl/surfnet/projects/desire/">[DESIRE] [DESIRE Web cache] [Web cache Architecture]

Contents

Checklist

We recommend that you

  • allow access to your Web cache server only for those servers you have set up an agreement with
  • always ask the administrator before you set a Web cache as you parent or neighbor
  • go to NLANR registration database, find potential neighbors and parents in your area
  • contact you Internet Service Provider

Caches and cache administrators need to cooperate in building and joining a national or network wide Web caching mesh.

Meshes are highly dependent on network topology. The main point is to build a mesh to avoid links with high costs, where costs may be download time for documents, economic cost, heavy load or long response times.

The hit rates will decrease on top-level servers. This is to be expected since servers on a lower level cache the documents they fetch from it, and it may be a while until they request the same document again. A toplevel server needs several different children and more traffic to be effective, as the popularity distribution of Web documents is a Zipf-like hyperbolic distribution [Cunha95].

Simple Mesh
Simple Mesh

The figure shows a simple Web cache mesh with two servers strung one after another. The width of the connections indicate both how fast a user can connect and how traffic flows in a well implemented mesh. Even in a more complex mesh with more servers and different relations between servers, the client relates to one proxy configuration. The mesh is a black box with hidden complexity for both client and origin Web server.

Several Norwegian colleges have distributed campuses, where sites may be 60-80 km apart. These different sites are normally linked with medium-capacity links (1-2Mbps), and heavy load may be a problem during work hours. Their Internet connection is typically 256 kbps. A parent located at the central Internet access point may reduce the load on the external connection. Neighbors located at each campus site may cooperate to share load. Misses (documents not found in cache) are resolved through parents, not through neighbors/siblings in Squid. This is a way to avoid loops. Configuration of neighbors and parents are handmade, hopefully there will be a better way to do this shortly (manual configuration does not scale).

5.1 ISP-wide neighbors/parents

Due to routing considerations and routing policies, one single national parent is unlikely. The top-level of the caching mesh is most likely on the ISP level. Some alternatives for an ISP-wide mesh are

If there is a single top-level server, this needs to be dimensioned for the considerable load it will take. Load is a function of the number of users. Decide whether or not to allow end users to access your toplevel caching server.

5.2 International neighbors/parents

Normally a Web cache mesh is terminated inside the ISPs network, as special care is needed not to violate routing policies for meshes spanning several autonomous systems (AS). At this stage of development in Web caching, establishing bi-lateral agreeements on Web caching between ISPs is reccomended if inter-ISP cooperation is needed. For academic networks, the need to consider the Acceptable Use Policy is of special concern.

An introduction to routing issues is found in [ripe-081] and [ripe-181]. An autonomous system (AS) is a group of IP networks having a single clearly defined routing policy which is run by one or more network operators. Inside an AS routing is done using one or more Interior Routing Protocols (today: OSPF). Routing decisions here are normally derived from technical parameters like topology, link speeds and load. When ASes exchange routing information with other ASes using Exterior Routing Policies (today: BGP4), routing decisions are often based on policy rather than purely technical parameters.

Exchange of routing information between ASes is subject to routing policies, where routes are announced only by those willing to route traffic to a network. This information may or may not be accepted by the recipient of the announced routing information. Only if the announced routes are accepted does traffic flow, and in order for traffic to flow in both directions routes between the two fringe networks must be announced and accepted in both directions.

Multilevel Web caches are two or more cooperating Web caches, often arranged in a hierarchy where a first-level Web cache is close to the user (local Web caching) and upper level Web caches have first-level Web caches as their clients (regional or national Web caches.)

Web cache meshes are several cooperating servers that together form a mesh doing multilevel Web caching. A mesh may span a university or a network, or several networks.

The lesson to learn from traditional routing is that purely technical routing decisions and policy decisions are different and may require two different sets of configuration and setups in order to work in an Internet. If Web caching meshes span several ASes special consideration must be given to routing policies.

We should stop treating cooperative Web caching as interior routing and start paying attention to exterior routing policies. Today's practice cannot continue if Web caching becomes widespread. Today, Web caching does not constitute a threat to "normal" traffic flow as few meshes span several ASes. As we aim to change this, we will have to find a way for Web caches to communicate policies.

When a mesh aggregates Web traffic, this aggregated traffic should be dealt with in close cooperation with the ISP. The two main reasons for this is topology knowledge and routing issues.

5.3 Institutional neighbors/parents

Several of the Norwegian colleges have distributed campuses, where sites may be 60-80 km apart. These different sites are normally linked with medium-capacity links (1-2Mbps), and heavy load may be a problem during workhours. Their Internet connection is typically 256 kbps. A parent located at the central Internet access point may reduce the load on the external connection. Neighbors located at each campus site may cooperate to share load.

5.4 Configuring Squid in a mesh

General configuration

When you cooperate with other Web cache servers, you need to agree about how long an object should be in the cache. If one of the siblings caches objects for 3 months, and your cache only caches these objects for 14 days, you will sooner or later get objects from your sibling which violate your own rules on how long these objects should be cached. This is a problem with both the HTTP/1.0 and the ICP protocol, which hopefully will be solved soon. But as of now an agreement on this issue between parents and siblings is the best solution. Implementing HTTP/1.1 in Web caches will solve this problem, and some progress has been made in the later versions of Squid. In Squid-1.1b11, an age value and checks for freshness replaced the previous Time To Live model. And the HTTP/1.1 "Cache-Control: Max-age=nnn" was added in version 1.1b17.

To protect the server machine against intruders the Web cache server should be run in a chroot environment. There are no known security problems in Squid as of now, but this may change. And Squid is very easy to install in a chroot environment. Just compile it using the -static flag in gcc, install some files in the chrooted dev and etc directories, install Squid there also, and just start it.

Security issues for the first level server

Mesh setup
Mesh setup

A first-level Web cache server within an organization can be a security risk unless it is correctly configured. An organization (outlined in the figure) will almost always have some internal Web objects. The usual method used by Web server administrators is to protect these objects (by IP-number or hostname) so only hosts from the organization's domain are allowed to fetch them.

A first-level Web cache server is almost always placed within the organization's domain. And if this Web cache server cooperates with Web cache servers outside the organization's domain you have to do something to protect your internal objects. If not, a cooperating Web cache server will be able to connect to the organization's Web cache server and request a protected (solid red in the figure) object. The organization's Web cache server will then request it from the origin Web server, and it will get the document because the origin Web server will see this as a request from within the local domain. The object will then be passed on to the cooperating Web cache server outside the organization.

This can be solved in several different ways. The most practical solution, as of this writing, is to deny other Web cache servers access to internal services through the organization's Web cache server. To avoid access denied messages, your partners will have to configure their Web cache servers not to go through your Web cache server when fetching documents from your domain. This makes sense since contacting your source should be just as fast as contacting your Web cache server (unless you have some very slow internal connection's, and then you should move or mirror important servers in front of the slow connection).

Denying other Web cache servers access to internal services through your Web cache server can be done like this in Squid: 
    acl yourdomain src       129.242.0.0/255.255.0.0
    acl yourhttp   url_regex ^http://.*\.your\.domain ^http://129\.242 
    acl yourftp    ftp_regex ^ftp:///.*\.your\.domain ^ftp://129\.242
    acl partner1   src       128.39.2.22 129.177.12.12
    ...
    acl partnerN   src       193.213.103.91

    http_access allow yourdomain
    http_access deny  yourhttp
    http_access allow yourpartner1
    ...
    http_access allow yourpartnerN
    http_access deny  all

To avoid access denied messages your partners will have to configure their Web cache servers to not go through your Web cache server when fetching documents from your domain. As mentioned above this should normally be just as fast as contacting your Web cache server.

Configuration parameters for another Squid Web cache server to use your Web cache server as a sibling or a parent: 
    # Using you as a neighbor
    cache_host        your.web.cache.server sibling 3128 3130
    cache_host_domain your.web.cache.server !.your.domain
  
    # Using you as a parent
    cache_host        your.web.cache.server parent 3128 3130
    cache_host_domain your.web.cache.server !.your.domain

If you want to force someone to only use you as a sibling (only allow them to fetch HITS) you can use the miss_access parameter in your own configuration file.

Security issues for the ISP level server

A ISP (or national) top-level cache server open for general access could very soon be swamped with access requests. Many of these accesses will be in direct conflict with routing policies, and this could give you trouble with your own ISP.

Access should be granted only to neighbors/siblings and parents with which you have set up an agreement. In this agreement, you can ensure that your routing policy is followed.

Restricting access to your Web cache service can be done like this in Squid: 
    acl your-domain      src 129.242.0.0/255.255.0.0
    acl partners         src 193.213.103.91 129.177.12.12
    acl sibling-partners src 130.237.222.70 192.87.46.3 194.179.100.4

    http_access allow your-domain
    http_access allow partners
    http_access allow sibling-partners
    http_access deny  all

    icp_access allow  your-domain
    icp_access allow  partners
    icp_access allow  sibling-partners
    icp_access deny   all

    miss_access allow your-domain
    miss_access allow partners
    miss_access deny  all
This will allow your regular partners and your own domain to fetch both hits and misses. Your sibling partners will only be allowed to fetch hits, and the rest will be denied access to your service.

5.5 Naming strategy for a ISP-wide Web cache

A suggestion today when setting up a ISP-wide Web caching mesh in UNINETT would be to make the domain cache.uninett.no and name

and possibly In the beginning all names would point to the same server. This makes scaling a smaller problem, and makes it possible to have specialized servers for certain popular domains (which may rise hitrates and you get to chose those that are most important for your users). It is possible for a local Web cache administrator to do a lookup to determine what domains have specialized caches.

Growing from one (as a start) to several servers would be less painful if this was implemented from the start, and it would be possible to start out with one single server. Implicit in this proposal is that there is a backbone with sufficient bandwith, if not you will be better off with several servers as a start.

Info about this config should be accessible at a central database (today in the database at NLANR). All children of the upper level cache should have access to this information, and providing child caches with standard config files is encouraged.

Operational experience with this approach is not yet available.

Ingrid Melve, Lars Slettjord

cache-desire@uninett.no 2002-10-29