QoS with TCP/IP

cOS Core DiffServ Support

It is important to understand that cOS Core traffic shaping does not add new DiffServ information as packets traverse a Clavister firewall. The cOS Core traffic shaping priorities described later in this chapter are for traffic shaping within cOS Core only and are not translated into DiffServ information that is then added to packets.

DSCP with cOS Core is also discussed in a Clavister Knowledge Base article at the following link:

https://kb.clavister.com/324735663

Explicit Congestion Notification Handling

ECN with cOS Core is also discussed in a Clavister Knowledge Base article at the following link:

https://kb.clavister.com/317180249

The Traffic Shaping Solution

cOS Core provides QoS control by allowing the administrator to apply limits and guarantees to the network traffic passing through the Clavister firewall. This approach is often referred to as traffic shaping and is well suited to managing bandwidth for local area networks as well as to managing the bottlenecks that might be found in larger wide area networks. It can be applied to any traffic including that passing through VPN tunnels.

Traffic Shaping Objectives

Traffic shaping does not typically work by queuing up immense amounts of data and then sorting out the prioritized traffic to send before sending non-prioritized traffic. Instead, the amount of prioritized traffic is measured and the non-prioritized traffic is limited dynamically so that it will not interfere with the throughput of prioritized traffic.

	Note: Traffic shaping will not work with the SIP ALG
	Any traffic connection that triggers an IP policy that uses the SIP ALG cannot also be subject to traffic shaping.

cOS Core offers extensive traffic shaping capabilities for the packets passing through the Clavister firewall. Different rate limits and traffic guarantees can be created as policies based on a filter that specifies the traffic's source, destination and protocol. This filter is similar to the filter used for IP rule set entries.

The two key components for traffic shaping in cOS Core are:

Note that using pipes for traffic shaping is also described in an article in the Clavister Knowledge Base at the following link:

https://kb.clavister.com/324736152

Pipes

Pipes are simplistic in that they do not care about the types of traffic that pass through them nor the direction of that traffic. They simply measure the aggregate data that passes through them and then apply the administrator configured limits for the pipe as a whole or for Precedences and/or Groups (these concepts are explained later in Section 11.1.6, Precedences).

cOS Core is capable of handling hundreds of pipes simultaneously, but in reality most scenarios require only a handful of pipes. It is possible that dozens of pipes might be needed in scenarios where individual pipes are used for individual protocols. Large numbers of pipes might also be needed in an ISP scenario where individual pipes are allocated to each client.

Pipe Rules

	Caution: Avoid using "any" as the source interface
	Filtering criteria in a pipe rule should be as specific as possible, triggering on specific interfaces and networks. In particular, avoid using any as the source interface. In certain cases, this could result in traffic shaping being applied to the same traffic twice.

Once a new connection is permitted by the IP rule set, the pipe rule set is then checked for any matching pipe rules. Pipe rules are checked in the same way as other rule sets, by scanning entries from top to bottom (first to last). The first matching rule, if any, decides if the connection is subject to traffic shaping. Keep in mind that any connection that does not trigger a pipe rule will not be subject to traffic shaping and could potentially use as much bandwidth as it wants.

The rule set for pipe rules is initially empty with no rules predefined. At least one rule must be created for traffic shaping to begin to function.

Pipe Rule Chains

Figure 11.1. Pipe Rules Determine Pipe Usage

The pipes that are to be used are specified in a pipe list. If only one pipe is specified then that is the pipe whose characteristics will be applied to the traffic. If a series of pipes are specified then these will form a Chain of pipes through which traffic will pass. A chain can be made up of a maximum of 8 pipes.

Explicitly Excluding Traffic from Shaping

This provides a means to explicitly exclude particular traffic from traffic shaping. Such rules are not absolutely necessary but if placed at the beginning of the pipe rule set, they can guard against accidental traffic shaping by later rules.

Pipes Will Not Work With Stateless Policy Rules

The reason for this is that traffic shaping is implemented by using the cOS Core state engine which is the subsystem that deals with the tracking of connections. A Stateless Policy does not set up a connection in the state engine. Instead, packets are considered not to be part of a connection and are forwarded individually to their destination, bypassing the state engine.

Figure 11.2. A Stateless Policy Bypasses Traffic Shaping

Using Pipes with Application Control

This feature is discussed further in Section 3.7, Application Control.

The simplest use of pipes is for bandwidth limiting. This is also a scenario that does not require much planning. The example that follows applies a bandwidth limit to inbound traffic only. This is the direction most likely to cause problems for Internet connections.

Device:/> add Pipe std-in LimitKbpsTotal=2000

Device:/> add PipeRule SourceInterface=lan
			SourceNetwork=lan_net
			DestinationInterface=wan
			DestinationNetwork=all-nets
			Service=all_services
			ReturnChain=std-in
			Name=Outbound

Using a Single Pipe for Both Directions

In the previous example only bandwidth in the inbound direction is limited. In most situations, this is the direction that becomes full first. But what if the outbound traffic must be limited in the same way?

Just inserting std-in in the forward chain will not work since we probably want the 2 Mbps limit for outbound traffic to be separate from the 2 Mbps limit for inbound traffic. If 2 Mbps of outbound traffic attempts to flow through the pipe in addition to 2 Mbps of inbound traffic, the total attempting to flow is 4 Mbps. Since the pipe limit is 2 Mbps, the actual flow will be close to 1 Mbps in each direction.

Raising the total pipe limit to 4 Mbps will not solve the problem since the single pipe will not know that 2 Mbps of inbound and 2 Mbps of outbound are the intended limits. The result might be 3 Mbps outbound and 1 Mbps inbound since this also adds up to 4 Mbps.

Using Two Separate Pipes Instead

Device:/> add Pipe std-out LimitKbpsTotal=2000

Device:/> set PipeRule Outbound ForwardChain=std-out

In the previous examples a static traffic limit for all outbound connections was applied. What if the aim is to limit web surfing more than other traffic? Assume that the total bandwidth limit is 250 Kbps and 125 Kbps of that is to be allocated to web surfing inbound traffic.

The Incorrect Solution

A surf-in pipe is therefore first created for inbound traffic with a 125 Kbps limit. Next, a new Pipe Rule is set up for surfing that uses the surf-in pipe and it is placed before the rule that directs everything else through the std-in pipe. That way web surfing traffic goes through the surf-in pipe and everything else is handled by the rule and pipe created earlier.

Unfortunately this will not achieve the desired effect, which is allocating a maximum of 125 Kbps to inbound surfing traffic as part of the 250 Kbps total. Inbound traffic will pass through one of two pipes: one that allows 250 Kbps, and one that allows 125 Kbps, giving a possible total of 375 Kbps of inbound traffic but this exceeds the real limit of 250 Kbps.

The Correct Solution

Figure 11.3. Differentiated Limits Using Chains

If surfing uses the full limit of 125 Kbps, those 125 Kbps will occupy half of the std-in pipe leaving 125 Kbps for the rest of the traffic. If no surfing is taking place then all of the 250 Kbps allowed through std-in will be available for other traffic.

This does not provide a bandwidth guarantee for web browsing but instead limits it to 125 Kbps and provides a 125 Kbps guarantee for everything else. For web browsing the normal rules of first-come, first-forwarded will apply when competing for the 125 Kbps bandwidth. This may mean 125 Kbps, but it may also mean much slower speed if the connection is flooded.

Setting up pipes in this way only puts limits on the maximum values for certain traffic types. It does not give priorities to different types of competing traffic.

The Default Precedence is Zero

There are 8 Possible Precedence Levels

Figure 11.4. The Eight Pipe Precedences

Precedence Priority is Relative

Allocating Precedence to Traffic

Specifying Precedences Within Pipes

As described above, the Default Precedence is the precedence taken by a packet if it is not explicitly assigned by a pipe rule.

The minimum and maximum precedences define the precedence range that the pipe will handle. If a packet arrives with an already allocated precedence below the minimum then its precedence is changed to the minimum. Similarly, if a packet arrives with an already allocated precedence above the maximum, its precedence is changed to the maximum.

For each pipe, separate bandwidth limits may be optionally specified for each precedence level. These limits can be specified in kilobits per second and/or packets per second (if both are specified then the first limit reached will be the limit used).

	Tip: Specifying bandwidth
	Remember that when specifying network traffic bandwidths, the prefix Kilo means 1000 and NOT 1024. For example, 3 Kbps means 3000 bits per second. Similarly, the prefix Mega means one million in a traffic bandwidth context.

Precedence Limits are also Guarantees

The Lowest (Best Effort) Precedence

Packets with a higher precedence than best effort and that exceed the limit of their precedence will automatically be transferred down into the lowest (best effort) precedence and they are treated the same as other packets at the lowest precedence.

In the illustration below the minimum precedence is 2 and the maximum precedence is 6. Precedence 2 is taken as the best effort precedence.

Figure 11.5. Minimum and Maximum Pipe Precedence

Lowest Precedence Limits

Precedences Only Apply When a Pipe is Full

When the pipe is full, traffic is prioritized by cOS Core according to precedence with higher precedence packets that do not exceed the precedence limit being sent before lower precedence packets. Lower precedence packets are buffered until they can be sent. If buffer space becomes exhausted then they are dropped.

If a total limit for a pipe is not specified, it is the same as saying that the pipe has unlimited bandwidth and consequently it can never become full so precedences have no meaning.

Applying Precedences

The effect of doing this is that the SSH and Telnet rule sets the higher priority on packets related to these services and these packets are sent through the same pipe as other traffic. The pipe then makes sure that these higher priority packets are sent first when the total bandwidth limit specified in the pipe's configuration is exceeded. Lower priority packets will be buffered and sent when higher priority traffic uses less than the maximum specified for the pipe. The buffering process is sometimes referred to as "throttling back" since it reduces the flow rate.

The Need for Guarantees

Using Precedences as Guarantees

To change the prioritized SSH and Telnet traffic from the previous example to a 96 Kbps guarantee, the precedence 2 limit for the std-in pipe is set to be 96 Kbps.

This does not mean that inbound SSH and Telnet traffic is limited to 96 Kbps. Limits in precedences above the best effort precedence will only limit how much of the traffic gets to pass in that specific precedence.

If more than 96 Kbps of precedence 2 traffic arrives, any excess traffic will be moved down to the best effort precedence. All traffic at the best effort precedence is then forwarded on a first-come, first-forwarded basis.

	Note: A limit on the lowest precedence has no meaning
	Setting a maximum limit for the lowest (best effort) precedence or any lower precedences has no meaning and will be ignored by cOS Core.

Differentiated Guarantees

The solution is to create two new pipes: one for telnet traffic, and one for SSH traffic, much like the "surf" pipe that was created earlier.

First, remove the 96 Kbps limit from the std-in pipe, then create two new pipes: ssh-in and telnet-in. Set the default precedence for both pipes to 2, and the precedence 2 limits to 32 and 64 Kbps, respectively.

Then, split the previously defined rule covering ports 22 through 23 into two rules, covering 22 and 23, respectively:

Keep the forward chain of both rules as std-out only. Again, to simplify this example, we concentrate only on inbound traffic, which is the direction that is the most likely to be the first one to fill up in client-oriented setups.

Set the return chain of the port 22 rule to ssh-in followed by std-in.
Set the return chain of the port 23 rule to telnet-in followed by std-in.
Set the priority assignment for both rules to Use defaults from first pipe; the default precedence of both the ssh-in and telnet-in pipes is 2.

Using this approach rather than hard-coding precedence 2 in the rule set, it is easy to change the precedence of all SSH and Telnet traffic by changing the default precedence of the ssh-in and telnet-in pipes.

Notice that we did not set a total limit for the ssh-in and telnet-in pipes. We do not need to since the total limit will be enforced by the std-in pipe at the end of the respective chains.

The ssh-in and telnet-in pipes act as a "priority filter": they make sure that no more than the reserved amount, 64 and 32 Kbps, respectively, of precedence 2 traffic will reach std-in. SSH and Telnet traffic exceeding their guarantees will reach std-in as precedence 0, the best-effort precedence of the std-in and ssh-in pipes.

	Note: The return chain ordering is important
	Here, the ordering of the pipes in the return chain is important. Should std-in appear before ssh-in and telnet-in, then traffic will reach std-in at the lowest precedence only and hence compete for the 250 Kbps of available bandwidth with other traffic.

cOS Core provides a further level of control within pipes through the ability to split pipe bandwidth into individual resource users within a group and to apply a limit and guarantee to each user.

Individual users can be grouped by cOS Core using one of the following:

This feature is enabled by enabling the Grouping option in a pipe. The individual users of a group can then have a limit and/or guarantee specified for them in the pipe. For example, if grouping is done by source IP then each user corresponds to each unique source IP address.

A Port Grouping Includes the IP Address

Grouping by Networks Requires the Size

Specifying Group Limits

Combining the Group Total and Precedences

The illustration below shows this flow where the grouping has been selected to be according to source IP.

Figure 11.6. Traffic Grouped By IP Address

Another Simple Groups Example

Bandwidth is now allocated on a "first come, first forwarded" basis but no single destination IP address can ever take more than 100 Kbps. No matter how many connections are involved the combined total bandwidth can still not exceed the pipe limit of 400 Kbps.

Combining Pipe and Group Limit Precedence Values

In this case, the Group Limits precedence value is a guarantee and the Pipe Limits value for the same precedence is a limit. For example, if traffic is being grouped by source IP and the Group Limits precedence 5 value is 5 Kbps and the Pipe Limits precedence 5 value is 20 Kbps, then after the fourth unique source IP (4 x 5 = 20 Kbps) the precedence limit is reached and the guarantees may no longer be met.

Dynamic Balancing

If a total group limit of 100 Kbps is also specified with dynamic balancing, then this still means that no single user may take more than that amount of bandwidth.

Precedences and Dynamic Balancing

Continuing with the previous example, we could limit how much guaranteed bandwidth each inside user gets for inbound SSH traffic. This prevents a single user from using up all available high-priority bandwidth.

First we group the users of the ssh-in pipe so limits will apply to each user on the internal network. Since the packets are inbound, we select the grouping for the ssh-in pipe to be Destination IP.

Now specify per-user limits by setting the precedence 2 limit to 16 Kbps per user. This means that each user will get no more than a 16 Kbps guarantee for their SSH traffic. If desired, we could also limit the group total bandwidth for each user to some value, such as 40 Kbps.

There will be a problem if there are more than 5 users utilizing SSH simultaneously: 16 Kbps times 5 is more than 64 Kbps. The total limit for the pipe will still be in effect, and each user will have to compete for the available precedence 2 bandwidth the same way they have to compete for the lowest precedence bandwidth. Some users will still get their 16 Kbps, some will not.

Dynamic balancing can be enabled to improve this situation by making sure all of the 5 users get the same amount of limited bandwidth. When the 5th user begins to generate SSH traffic, balancing lowers the limit per user to about 13 Kbps (64 Kbps divided by 5 users).

Dynamic Balancing takes place within each precedence of a pipe individually. This means that if users are allotted a certain small amount of high priority traffic, and a larger chunk of best-effort traffic, all users will get their share of the high-precedence traffic as well as their fair share of the best-effort traffic.

The pipes CLI Command

Consider the following example usage:

Device:/> pipes -users my_pipe1

This will report on only the active users over the one second period after the command is entered in the Pipe called my_pipe1. The number of users active over that one second period may only be a fraction of those active over a longer time interval.

For a complete description of pipes command options, see the separate cOS Core CLI Reference Guide.

If using traffic shaping with IPsec or any tunneling protocol, the following should be noted:

The Importance of a Pipe Limit

Relying on the Group Limit

Limits should not be more than the Available Bandwidth

Limits should be less than Available Bandwidth

The reason for the lower pipe limit is how cOS Core processes traffic. For outbound connections where packets leave the firewall, there is always the possibility that cOS Core might slightly overload the connection because of the software delays involved in deciding to send packets and the packets actually being dispatched from buffers.

For inbound connections, there is less control over what is arriving and what has to be processed by the traffic shaping subsystem and it is therefore more important to set pipe limits slightly below the real connection limit to account for the time needed for cOS Core to adapt to changing conditions.

Attacks on Bandwidth

Watching for Leaks

If there is traffic going through the Internet connection that the pipes do not know about, cOS Core cannot know when the Internet connection becomes full.

The problems resulting from leaks are exactly the same as in the cases described above. Traffic "leaking" through without being measured by pipes will have the same effect as bandwidth consumed by parties outside of administrator control but sharing the same connection.

Troubleshooting

Device:/> pipe -u <pipename>

cOS Core traffic shaping provides a sophisticated set of mechanisms for controlling and prioritizing network packets. The following points summarize the important points when using it:

This section looks at some more scenarios and how traffic shaping can be used to solve particular problems.

A Basic Scenario

Figure 11.7. A Basic Traffic Shaping Scenario

The reason for using 2 different pipes in this case, is that these are easier to match to the physical link capacity. This is especially true with asynchronous links such as ADSL.

First, two pipes called in-pipe and out-pipe need to be created with the following parameters:

Dynamic Balancing should be enabled for both pipes. Instead of PerDestIP and PerSrcIP we could have used PerDestNet and PerSrcNet if there were several networks on the inside.

The next step is to create the following Pipe Rule which will force traffic to flow through the pipes.

The rule will force all traffic to the default precedence level and the pipes will limit total traffic to their 1 Mbps limit. Having Dynamic Balancing enabled on the pipes means that all users will be allocated a fair share of this capacity.

Using Several Precedences

Assume there is a symmetric 2/2 Mbps link to the Internet. Descending priorities and traffic requirements will be allocated to the following users:

To implement this scheme, we can use the in-pipe and out-pipe. We first enter the Pipe Limits for each pipe. These limits correspond to the list above and are:

Now create the Pipe Rules:

These rules are processed from top to bottom and force different kinds of traffic into precedences based on the Service. Customized service objects may need to be first created in order to identify particular types of traffic. The all_services rule at the end, catches anything that falls through from earlier rules since it is important that no traffic bypasses the pipe rule set otherwise using pipes will not work.

Pipe Chaining

Note that in-other and out-other are first in the pipe chain in both directions. This is because we want to limit the traffic immediately, before it enters the in-pipe and out-pipe and competes with VoIP, Citrix and Web-surfing traffic.

A VPN Scenario

An important consideration which has been discussed previously, is allowance in the Pipe Total values for the overhead used by VPN protocols. As a rule of thumb, a pipe total of 1700 bps is reasonable for a VPN tunnel where the underlying physical connection capacity is 2 Mbps.

It is also important to remember to insert into the pipe all non-VPN traffic using the same physical link.

The pipe chaining can be used as a solution to the problem of VPN overhead. A limit which allows for this overhead is placed on the VPN tunnel traffic and non-VPN traffic is inserted into a pipe that matches the speed of the physical link.

To do this we first create separate pipes for the outgoing traffic and the incoming traffic. VoIP traffic will be sent over a VPN tunnel that will have a high priority. All other traffic will be sent at the best effort priority (see above for an explanation of this term). Again, a 2/2 Mbps symmetric link is assumed.

The pipes required will be:

The following pipe rules are then needed to force traffic into the correct pipes and precedence levels:

With this setup, all VPN traffic is limited to 1700 Kbps, the total traffic is limited to 2000 Kbps and VoIP to the remote site is guaranteed 500 Kbps of capacity before it is forced to best effort.

SAT with Pipes

A simple solution is to put a "catch-all-inbound" rule at the bottom of the pipe rule. However, the external interface (wan) should be the source interface to avoid putting into pipes traffic that is coming from the inside and going to the external IP address. This last rule will therefore be:

	Note: SAT and ARPed IP Addresses
	If the SAT is from an ARPed IP address, the wan interface needs to be the destination.