CONFIDENTIAL -- Do not redistribute or duplicate without permission.
Copyright 2002 Carnegie Mellon University. All rights reserved.


Quality of Service Policy Enforcement Review

 

Packeteer PacketShaper v. Allot NetEnforcer

September 5, 2002                                                     Peter Hill, Kevin Miller, Kunal Trivedi

Carnegie Mellon University


Synopsis

            Both the PacketShaper and NetEnforcer products give network administrators a higher degree of control over quality-of-service policy enforcement than traditional network switches and routers. They offer a myriad of methods for traffic classification (IP address, TCP/UDP ports, QoS tags, and others). After classification, policy-based bandwidth control mechanisms are applied to the traffic. Alternately, the boxes can mark ToS/CoS[1] bits and allow other network elements to perform the actual traffic policing.

            While both products are strong contenders, our analysis is that the NetEnforcer line is a stronger platform for implementing bandwidth control policies. The concept of equitable bandwidth allocation is fundamental to the design of the unit, and its strategy of unifying the inbound and outbound traffic for purposes of queuing seems to give it a fundamental advantage over the PacketShaper.

Need for Quality of Service

Before diving into a review of the products, it’s useful to understand the rationale behind the quality of service policy enforcement products. The products in the field are slowly gaining popularity as the need becomes apparent. Specifically, traditional Ethernet-and-IP network elements (routers and switches) perform well in lightly-loaded situations, but problems arise as traffic increases. For example, shared-access Ethernet rapidly degrades in throughput after about 30% utilization. In this situation, the number of collisions and retransmissions grows quickly, reducing the network efficiency.

While the issues of half-duplex shared-Ethernet are not a problem on full duplex core network links, oversubscription of links, especially outbound links, is a problem. When machines try pushing more data than administratively allowed, routers are forced to begin queuing packets or eating into burst-buffer space. As the link utilization increases dramatically beyond the limit, constant over-subscription and queuing become the norm. Queuing has limits, however, as does user patience. While a large router with nearly infinite queues might guarantee packet delivery, most users aren’t willing to wait several days. Indeed, infinite queuing is not a solution. Most routers implement a ‘tail-drop’ system, where packets are simply dropped as the queue limit is reached. In this case, high-rate connections can overpower low-rate connections (such as simple web browsing) by simply sending or receiving more data.

The solution, as the sales people keep reminding us, is to install a QoS policy shaper, such as the PacketShaper or NetEnforcer. The policy shapers provide controls to administrators that allow far greater control of traffic than routers. Different types of traffic can be categorized and grouped, and specific policies applied to the groups. Categorization of traffic happens by inspecting the packets and matching on certain elements of the packet. This can include the source or destination IP address, source or destination TCP/UDP port numbers, or pre-defined quality of service tags. Today, categorization engines are also looking ‘deeper’ into packets, inspecting application layer content such as HTTP URLs and peer-to-peer filenames.

After categorization, administrators can then apply specific policies to the different groups of traffic. Standard policies that can be implemented include giving certain traffic a higher priority (and thus access to the bandwidth) than other traffic. Alternatively, a policy might grant a specific amount of bandwidth to traffic in a certain group, or a certain amount of bandwidth to each unique connection (for example, giving each voice-over-IP connection a specific bandwidth allocation).

The expectation, in categorizing and policing the traffic, is to reduce the need for queuing on the router. The packet shapers use different methods to manipulate TCP’s built-in flow control mechanisms, slowing down lower priority traffic. This causes some connections to begin flowing at slower speeds than they would otherwise and reduce congestion at the router.

            The subtext, however, is that the packet shapers will be most effective when specific policy is encoded. They can certainly use the rate control mechanisms to reduce congestion at the router, but in general this will continue to cause high-rate connections to dominate low-rate connections. At minimum, users will continue to receive sub-par bandwidth for what they consider “important” uses (typically, interactive applications such as web browsing). Consequently, creating policies that prioritize important connections, while slowing those that do not require immediacy, can reduce the consumed bandwidth while making the network appear more responsive.

Existing Functionality

            With the need for quality of service mechanisms apparent, router and switch vendors offer some controls in these products for enforcing QoS policy. The Catalyst 6500 platform, which we currently have deployed as the campus backbone, has some basic QoS controls. The 6500s can classify traffic similar to the packet shaping devices: based upon IP addresses, TCP/UDP port numbers, and ToS/CoS tags. Once classified, specific rate limits can be applied either as ‘microflow’ rates or ‘aggregate’ rates, or both. Microflow rates limit the rate a single flow can consume, while aggregate rates limit the rate that all traffic of the particular type can consume.

            Unfortunately, the 6500 platform cannot perform complicated traffic shaping, such as manipulating TCP’s flow control mechanisms, to slow down out-of-profile connections. Instead, it uses a simple tail-drop system. Traffic exceeding the rate limit is simply dropped. When packets are dropped, TCP responds by scaling down the transfer rate and retransmitting the packet. Thus, while some amount of dropping is expected, excessive dropping causes a dramatic increase in the retransmission rate, further reducing the network efficiency.

            Currently, our QoS policy on our commodity Internet connection specifies explicit rates for dormitory and non-dormitory traffic (25Mbps and 55Mbps, respectively). Today, we are hitting these limits almost constantly (from about 5-7am, the total outbound traffic drops to around 65Mbps). It appears that during the day we are discarding around 10Mbps of traffic due to these limits. This is somewhat misleading, though, since traffic will automatically scale back when packets are dropped. If there were no limit, we’d expect the traffic usage to increase substantially (before limits were applied in March, we were consuming about 110Mbps of outbound commodity traffic). The expectation is that the demand for commodity Internet bandwidth will only increase. As the demand increases, the QoS policies currently available will be inadequate for effectively controlling the bandwidth (without causing noticeably slow connections).

Example Policy Configuration

            In a deployment of the NetEnforcer unit to packet shape egress traffic, a sample configuration might be to have an LDAP server with the policy. This would enable easy delegation of administrative control, as well as provide rapid updates as needed. Traffic would be limited to 80Mbps in both directions.

            The primary method of traffic balancing would be to use traffic priorities, enabling traffic from a low-priority group to consume more bandwidth if higher-priority classes are unused. Priority 10 is the highest priority and priority 1 the lowest. A sample provisioning of priority values might be:

 

Priority Level

Type of Traffic

10

Network critical traffic (routing protocols)

Interactive traffic (telnet, SSH) below a configured rate

8

Inbound web traffic

Priority outbound traffic (certain web servers)

7

Other inbound traffic (internal connections)

6

Other outbound traffic (external connections)

5

Specific non-interactive outbound traffic (large FTP, SMTP)

3

Specifically identified low-priority ports or identified applications (peer-to-peer) importing or exporting data

 

Products

PacketShaper

            The Packeteer PacketShaper is a popular product among Universities, including peer institutions (e.g. Stanford). They boast 30+ patents on the packet shaping technology in the unit. The range of Packeteer products includes those that shape from 2Mbps to 200Mbps (half-duplex[2]). Related products to the PacketShaper include: PolicyCenter, a central policy administration server, and ReportCenter, a central server for recording monitoring data.

NetEnforcer

          The Allot Communications NetEnforcer comes with recommendations from Merit Networks, but is a relatively unknown product. The range of Allot products includes those that shape from 128Kbps to 155Mbps (full-duplex[3]). Allot also sells NetPolicy, a central policy administration server. Available as add-ons for the NetEnforcer include: NetAccountant, which tracks usage information (so-called “Internet call records”), NetBalancer, which redirects client requests to available content servers, and CacheEnforcer,  which redirects local content requests through a central cache.


Feature Matrix

            The matrix below highlights many of the basic features of the units. Important differences are displayed boldfaced. The PacketShaper has the concept of “partitions” of bandwidth with “classes” beneath the partition. Classes identify specific types of traffic. The NetEnforcer similarly uses “pipes” to indicate bandwidth partitions, and “Virtual Channel” (VC) to identify specific types of traffic.

 

 

PacketShaper

NetEnforcer

6500

8500

601C

701C

701F

Shaping Rate – Total

100M

200M

200M

310M

310M

List Price (Thousands)

$24

$32

$32

$38

$38

Maximum Partitions/Pipes

512

512

4,000

4,000

4,000

Maximum Classes/VCs

1,000

1,000

28,000

28,000

28,000

Maximum Dynamic Partitions

5,000

10,000

28,000

28,000

28,000

Maximum IP Hosts

25,000

100,000

n/a

n/a

n/a

Maximum Flows

n/a

n/a

256,000

256,000

256,000

 

 

 

 

 

 

Configuration                         

 

 

 

 

 

Web Management Interface

CLI Management Interface

External LCD (+ Keypad)

LCD

LCD

LCD +

LCD +

LCD +

 

 

 

 

 

 

Policy Source

 

 

 

 

 

Local Policy Configuration

Single Policy Config Screen

 

 

COPS

 

 

LDAP

[4]

 

 

 

 

 

 

Classification

 

 

 

 

 

IP/MAC Address & Port

ISL/802.1q Classification

DiffServ/ToS

HTTP URL

Extensive Layer 7

 

 

 

 

 

 

 

 

 

Packet Shaping

 

 

 

 

 

Admission Control (ACL)

TCP & UDP Rate Control

DiffServ/ToS/Cos Marking

TCP Flow Control

 

 

 

Packet Queuing Control

Flow-Based Traffic Control

 

 

Fairness of Equal Flows

 

 

Per-Class Bandwidth Limits

Per-Class Priority Queuing

Relative Priority Weighting

 

 

Dynamic Host Limits

Connection Number Limits

[5]

Bandwidth Limit Tree

 

 

 

Bandwidth “Borrowing”

 

 

 

 

 

 

 

 

 

Monitoring

 

 

 

 

 

Top Clients/Servers

Top Applications

Total Bandwidth Utilization

Historical Data Store

Slowest Clients/Servers

 

 

 

Top Traffic Classes/Utilization

Dropped Packets/Efficiency

 

 

 

 

 

 

Accounting

 

 

 

 

 

RADIUS Accounting

 

 

ODBC/SQL Accounting

 

 

SNMP Export

XML/CSV Data Export

 

 

 

 

 

 

 

 

 

Reliability

 

 

 

 

 

Dual Power Supply

Hot-Swap Power Supply

 

Expandable – 2 PCI

 

 

 

Redundant Unit Support

 

 

Automatic Layer-1 Bypass

 

Bypass on Non-Fatal Error

 

 

 

 

 

 

 

 

Interfaces

 

 

 

 

 

Fast Ethernet – Copper

 

 

 

Gigabit Ethernet – Copper

 

 

 

Gigabit Ethernet – Fiber

 

 

 

 

Fast Ethernet Management

 

 

Console – DB-9

 

 

 

 

 

 

Physical

 

 

 

 

 

19” Rackmount Units

2 RU

2 RU

2 RU

2 RU

 2 RU

 


Configuration

          Configuration of both units is principally accomplished by way of the web interfaces. Both units additionally have a command line interface for configuration. The PacketShaper uses a proprietary, Cisco-like command line interface (CLI). The NetEnforcer, running embedded Linux, provides the traditional administration commands one would expect to find. Configuration of the NetEnforcer-specific parameters is accomplished using specific binaries.

            The PacketShaper web interface uses fairly standard HTML and JavaScript[6]. Multiple administrative users (with “touch” access) can use this interface simultaneously. There are five main tabs for configuration and reporting capabilities, however there are myriad links among the sections. Policy configuration generally requires several round-trips loading web pages[7]. Configuration changes take effect immediately. Because of the use of standard HTML and the web modality of click-reload, the box can provide consistent behavior for multiple simultaneous administrators. We would expect this interface to be accessible to machines with substandard or no support for Java. It was tested and worked with Internet Explorer on Windows and Mozilla under Linux. Surprisingly, however, Internet Explorer and Mozilla under MacOS X were unable to use the Javascript-based menus.

            The NetEnforcer web interface is a Java-based console[8]. Only a single user with read-write access may be connected to the web interface, though multiple read-only users may be connected. When attempting to connect while another user has administrative access, the new user is offered the ability to take read-write control. The unit has separate areas for box setup, policy configuration, and monitoring/reporting. Changes in policy or setup are made interactively with standard Java user interface elements and committed to the box when complete[9]. The interface worked using Internet Explorer on Windows and MacOS X, as well as Mozilla under Linux.           

Analysis

            Both units provide functional web interfaces for configuration. Due to the relatively complicated nature of policy definitions, the web interface is definitely the best way to configure and monitor the devices, though both have CLIs that can theoretically perform all of the functions (modulo the graphs).

            While the use of Java may restrict the platform/browsers that can reliably use the interface, we greatly preferred the NetEnforcer’s interface. Policy configuration on the PacketShaper was slow and arduous, since each operation involved a round-trip to the unit. Changes take effect immediately, so the implications of intermediate steps must be considered. While this is standard for Cisco configuration, for example, the policy configuration can be quite complex and much more dependent upon the whole picture. Additionally, the PacketShaper defines separate policies for inbound and outbound traffic, potentially requiring twice the work for similar policy configuration. Configuration is done using multiple screens that configure specific elements of the policy.

            The NetEnforcer’s policy configuration is done through a single main window. The interactive nature of the dialog makes the configuration process swift. The box makes extensive use of so-called “catalogs” to define elements of the policy. For example, host groups are formed using catalogs. The quality of service parameters (such as the queue priority and bandwidth limits) are defined in a separate catalog space. The catalog elements are then accessible on the main configuration screen. Once the new configuration is complete, a “save” operation is performed, committing the new parameters.

            The monitoring and reporting capabilities of the NetEnforcer are also superior to those of the PacketShaper[10]. The unit comes with several standard graphs (Bandwidth, Utilization, Top Servers, Top Clients) in a well-defined subsection of the Java interface. The graphs are designed for real-time analysis and updated every 30 seconds. Additional historical data is also stored on the unit, and can be exported using SQL/ODBC, RADIUS, or SNMP.

            On the other hand, the PacketShaper has a few basic graphs (utilization, network efficiency) but other graphs must be custom-defined[11]. The graphs are optimized for long term reporting (the minimum graph size is one-hour, while the NetEnforcer scales the graphs automatically, starting with the first data point). Additionally, custom-defined graphs require two separate browser windows to keep active, and are difficult to use for short-term analysis. Long term historical data can be exported using a XML or CSV formatted data over an HTTP connection, or SNMP.

System Testing

Setup

          The goal of testing the packet shaping devices is to assess the behavior of them in scenarios that we believe reflect those of the larger network. Both units are installed inline with single buildings. This provides us the ability to assess the performance of the units in monitoring and shaping the true network traffic. Since we have little control of this traffic, however, the tests are conducted using controlled connections.

The testing network consists of three laptops on these existing building networks and two server machines across the core. While certainly the number of machines involved in the test is relatively small, the small number of machines involved allows close analysis of the performance of the packet shaping units. The laptops were running MacOS X, Linux, and Windows XP, while the servers were running Linux. All of the machines were connected using full-duplex Fast Ethernet, to minimize any effects of Ethernet retransmissions.

Both units are configured with a 5Mbps limit (inbound and outbound) for the three laptop testing machines. Establishing specific limits is done to prevent skew in the results due to non-test network traffic. Additionally, it prevents the testing from interfering with the normal operation of the network. Recording of the test results is accomplished using client and server reports of the net effective rates, as well as rates reported by the packet shaping units. Measurements of the transfer rates were observed during all parts of the test, but the reported values are taken from the middle of the test. This was done to minimize the effects of ramp-up or ramp-down during the tests.

Three primary testing modules are used to examine the units: the “Web In” module, the “Web Out” module, and the “IPerf-Out” module.

·              Web In
The web-in module is designed to simulate the behavior of web browsing. All three laptops run the test, which simultaneously initiates 4-8 retrievals of files from a web server running on Server 1. The file sizes were two each among 500KB, 1MB, 2MB, and 4MB. While larger than typical web files, the larger file size allows more accurate computation of the transfer rates. The module automatically restarts the download of a file after the previous instance of the download completes.

·              Web Out
The web-out module simulates the effects of a large internal web server pushing content to external clients. A web server on Laptop 1 provides two large files (50MB and 90MB), and Server 1 initiates a download of these two files. The individual tests are constructed such that the file downloads are completed as part of the testing duration, so no restart of the file retrieval is required.

·              IPerf-Out
The iperf utility is used during the test to generate additional traffic on the wire. Most commonly it is used to simulate low-priority traffic and is queued separately from the other web traffic. The laptops initiate TCP iperf connections to Server 2, and the principal flow of data is from the client to the server (unlike web connections).

 

Three major classification and queuing strategies are implemented on both boxes, and the results from each test recorded. The strategies are:

 

·              Traffic Bandwidth Allocation
A common recommended strategy for implementing a packet shaping device is to create classes for various traffic types. Categorization by application layer protocol or transport layer port is employed to assign traffic into the classes. Specific bandwidth allocations (limits) are given to the classes. The limits apply to the aggregate of all traffic matching the policy definition.

For purposes of our testing, bandwidth allocations are placed for outbound traffic. Web-Out is assigned a specific allocation and IPerf-Out is assigned a different allocation. The assumption is that Web-In does not need a specific limit.

·              Traffic Priorities
Like the specific bandwidth allocation strategy, this strategy places different priorities on the various traffic classes. The PacketShaper products support only an absolute priority. Traffic in a higher priority class is completely serviced prior to traffic in a lower class receiving any bandwidth. The NetEnforcer products support a relative priority system: higher priority queues receive more bandwidth than lower priority queues by a specific multiplier. For example, traffic at priority 8 receives 2.7 times the bandwidth at priority 3.

In our testing, Web-Out and IPerf-Out traffic is assigned different priorities (using the different strategies).

·              Host-Based Fair Bandwidth Allocation
Unlike the other modules, an approach of having the packet shaper fairly allocate bandwidth amongst all the active users does not rely upon application identification.

During the tests, the three laptops are given equal bandwidth allocations for outbound bandwidth. Clients running both an IPerf-Out and Web-Out should expect to receive the same slice of the available bandwidth as clients running just the Web-Out module.

           

Test Network Map

 

 

Results

Base Case

To begin, we initiate the Web-In, Web-Out, and IPerf-Out modules. All three Web-In clients are configured to attempt simultaneous downloads of 8 files. This is conducted with no specific policies except the 5Mbps limit in both directions. The observed transfer rates are:

 

 

            The results of this base case test are rather surprising. The tests ensure that the 5Mbps pipe can be completely saturated in both directions. It appears that the NetEnforcer’s strategy of treating the inbound and outbound portions of each connection as a single entity, as well as its strategy of equally allocating bandwidth among connections, provides better overall results. While the NetEnforcer is moving the full 5Mbps in each direction, the PacketShaper moves a paltry 1.6Mbps outbound. We presume this is because the inbound web traffic is preventing even TCP ACKs from moving across the inbound link.

 

Priority Queuing

            The next test involves the creation of priority queue values, differentiating the three classes of traffic. The settings are chosen to maximize the bandwidth for Web-In traffic, assuming users will be most likely to perceive network response as a function of interactive network use, such as web browsing. Web-Out is given slightly less priority than Web-In, and IPerf-Out is given the least bandwidth. The IPerf-Out traffic is meant to simulate peer-to-peer or other low-priority outbound traffic

 

            The settings used in this test are below. The NetEnforcer uses a scale of 1-10 and uses a relative queuing system. That is, traffic at a certain priority is serviced more often, with a specific multiplier, than traffic at lower priorities. The PacketShaper priorities are based on a scale of 1-7, and indicate absolute priorities. Given the NetEnforcer settings, Web-In traffic should receive 5 times the bandwidth of Web-Out, and Web-Out should receive 1.6 times the bandwidth of IPerf-Out.

 

 

            The results of the Priority Queuing test are below. All of the tests could individually consume the entire 5Mbps if run independently.

 

 

            Again, the results of the PacketShaper are surprising: the IPerf-Out traffic is actually receiving more than 5 times the bandwidth of Web-Out, even though the strict queuing should have allocated no bandwidth to the IPerf. The NetEnforcer results are in line with expectations, as the IPerf-Out result (1.8) times the expected multiplier (1.6) is 2.88, slightly more than the bandwidth that Web-Out ultimately received. The Web-In module requires only a small amount of outbound bandwidth for the TCP ACKs, which it appears to be receiving using both boxes.

Bandwidth Allocations

            Continuing with the tests, we change the policies to grant specific bandwidth allocations to the traffic types. On both boxes, Web-Out is allocated 2Mbps of outbound bandwidth, while IPerf-Out is allocated 1.2Mbps.

 

 

            The results of this test are in line with previous tests. Namely, the Web-In traffic continues to cause havoc on the PacketShaper, while the NetEnforcer allocates the bandwidth nearly as expected. Because the outbound policies on the PacketShaper do not affect inbound traffic at all, the PacketShaper is unable to shape the traffic on the inbound side to provide the outbound bandwidth allocations.

Fair Bandwidth Allocation   

The final test strategy is to configure the units to perform fair bandwidth allocation based on IP source address. The rationale is to divide the bandwidth amongst the active network users, providing extra bandwidth to users only after all the active users receive an equal share.

            This allocation strategy is supported by both units with differing degrees. The PacketShaper provides the ability to create so-named “Dynamic Sub-partitions” at multiple  points along a tree-based policy definition. It is therefore more flexible than the NetEnforcer’s capability of creating “Template VCs” at just one level. The PacketShaper also supports the creation of dynamic sub-partitions by simply describing the sub-partition criteria (source address, for example). The NetEnforcer requires that the members of the sub-partition be enumerated.

            While attempting to configure the NetEnforcer with several thousand addresses, the Java interface slows to a crawl and is nearly unusable. Other methods of definition include flat-file and LDAP backend, but these methods were not examined. Using these template VCs would only be possible if the actual members was not enumerated in the Java interface.

            Additionally, whether deployment of fair bandwidth allocation per-source address across campus is a good idea is an open question. PacketShaper engineers claim that many dynamic sub-partitions will cause the unit to slow dramatically. The NetEnforcer would also need to be tested to see if it could perform well with high numbers of dynamically created VCs.

            To test the capability of the units to handle this strategy, however, we configure the policies using the web interfaces. Each is configured to fairly allocate the outbound bandwidth. Throughout the test the Web-Out module is running, consuming outbound bandwidth. The IPerf-Out test is not used. Laptop 1 is configured to retrieve all 8 files using the Web-In module. Laptop 2 is configured to only retrieve 6 files, while Laptop 3 is setup to retrieve 4 files.

            First, both boxes are tested without the fair allocation. This ‘Best Effort’ mode imposed only the 5Mbps bidirectional limit. The Web-In test will run indefinitely, but testing is stopped after approximately three minutes, when it is clear that the network is in a steady state. Then, the fair allocation policies are configured and the test is restarted. After determining the new allocation rates, the test is concluded.

 

 

           

The results clearly indicate that both boxes can perform fair bandwidth allocation. However, the wisdom in doing so is still an open question. Both units have substantial uncertainty about the performance with many active hosts, and the NetEnforcer interface is not suited to configuring this for a large number of hosts.

Conclusion

            While both the Packeteer PacketShaper and Allot NetEnforcer provide much more flexibility in the application of quality of service policies to enterprise networks, our testing and experience indicates that the NetEnforcer is more suited to our environment. The NetEnforcer configuration is simple and responsive. While it does not support an entire tree of policy settings, the NetEnforcer’s two-level classification and policy application configuration will be acceptable for all but the most complex needs. The top NetEnforcer model can be purchased with fiber interfaces, allowing simple integration into our network, and the separate Fast Ethernet management interface is a nice addition.

            Apart from the configuration and management capabilities, we are impressed by the NetEnforcer’s fundamental support of equitable division of bandwidth among the connections. During testing, similar connections among the laptops results in equitable bandwidth division, even when fair bandwidth allocation policies are not configured. Similar testing on the PacketShaper results in a widely skewed bandwidth allocation – some flows are treated much better than others. We believe the NetEnforcer also performs better given its method of unifying the inbound and outbound connection for purposes of queuing. This is evident in the test results, as the NetEnforcer achieves better total throughput under saturated-network conditions.

           


Appendix A: PacketShaper Setup View

 
Appendix B: PacketShaper Manage View


Appendix C: PacketShaper Monitoring View


Appendix D: NetEnforcer Setup View


Appendix E: NetEnforcer Policy Configuration


Appendix F: NetEnforcer Monitoring View



[1] “Type of Service” is a byte in the IP header; “Class of Service” is a byte in the Ethernet header

[2] Within the PacketShaper product, the shaping rate must be allocated between inbound and outbound bandwidth -- a 200Mbps unit would shape 100Mbps in both directions, for example.

[3] The NetEnforcer series shapes a set rate in each direction – generally matching popular line rates (100Mbps, 155Mbps).

[4] Using Packeteer PolicyCenter.

[5] The number of subpartitions can be bounded.

[6] See Appendix A

[7] See Appendix B

[8] See Appendix D

[9] See Appendix E

[10] See Appendix F

[11] See Appendix C