6.5. Route Load Balancing

Home Prev	cOS Stream 4.10.00 Administration Guide	Next

Overview

In cOS Stream, Route Load Balancing (RLB) allows the distribution of traffic across multiple network paths. This feature is commonly used to utilize multiple network links, such as multiple ISP connections, to increase total bandwidth or reduce the load on specific routes.

Figure 6.4. A Common Route Load Balancing Scenario

RLB can be enabled per routing table or on individual routes and uses RLB Profile objects that define the algorithm and stickiness behavior used for routing decisions. Only destination-based route lookups are typically affected by RLB.

Routes or routing tables that do not have an assigned RLB profile will not participate in load balancing.

Enabling RLB

RLB is enabled by configuring an RLB Profile, which defines the algorithm, stickiness, and keepalive behavior used for routing decisions.

An RLB profile can be applied in two ways:

Routing table level - The profile is set as the default for a routing table. Routes without their own profile assignment will use this default.
Route level - The profile is explicitly assigned to an individual route. This overrides any default profile set at the routing table level.

Each route must belong to exactly one profile or inherit one from the table to participate in RLB. If no profile applies to a route, it will be excluded from RLB selection.

	Note: The Main Routing Table Cannot Use a Profile
	It is not possible to assign a RLB profile to the main routing table as it is read-only. The main routing table will be limited to using profiles on individual routes.

	Note: Priority Order, Using Profiles on Routing Table vs. Individual Routes
	Profiles can be used both on a routing table and on individual routes within the table. Profiles set on individual routes take priority over those set on the routing table itself.

RLB profiles define how cOS Stream selects among matching routes. Profiles act as isolated balancing domains — only routes using the same profile are considered together, and stickiness state is not shared across profiles.

The following example sets up an RLB profile on a custom routing table:

Example 6.5. RLB Profile Configuration Example

This configuration creates a profile named MyProfile using the Flow algorithm. Stickiness is not needed here, as the full name of the Flow algorithm is Deterministic Flow Selection. This means that the Stickiness option has no effect, since the algorithm already ensures the same route is selected for each unique flow. The profile is set as the default for the routing table MyTable.

Command-Line Interface

System:/> add RLBProfile MyProfile
			 Algorithm=Flow
			 Stickiness=None
			 Keepalive=Usage

System:/> set RoutingTable MyTable DefaultRLB=MyProfile

Disabling RLB

RLB is disabled by removing any profile assignment from both the routing table and from individual routes. A route that does not have an assigned profile and is not affected by a default profile on the routing table will not participate in load balancing.

When no RLB profile is applied, a route is treated as a standard static route and will be excluded from all RLB-related selection logic. Such routes are resolved independently and do not contribute to or interact with routes in any RLB group.

RLB Operation

When RLB is enabled on a routing table, cOS Stream uses the configuration defined in the assigned RLB Profile to determine how to select the target route during a destination-based route lookup.

RLB is only applied to routes that meet the following criteria:

Shortest network range - Only routes with the most specific matching range are considered. E.g. the network 192.168.1.0/25 would be more specific than 192.168.1.0/24.
Same RLB profile - All participating routes must share the same profile.
Identical destination range - All routes must match the same IP range exactly.
Lowest metric - Among overlapping routes with the same network range, only those with the lowest metric value are considered for route load balancing.

Once the applicable routes are selected, cOS Stream applies the algorithm defined in the profile to choose the final route. Each algorithm determines route selection based on different criteria and input values.

Supported Algorithms

Each RLB profile defines which algorithm is used to select between matching routes. The algorithm controls how cOS Stream distributes traffic across eligible routes. The following algorithms are supported:

Flow - Uses a hash of the flow's 5-tuple (Source/destination IP address, source/destination port and protocol) to consistently select a route for each unique flow.
SourceIP - Selects a route based on source IP and receiving interface. Useful when client-based path consistency is needed.
DestinationIP - Chooses a route based on destination IP. Ensures the same target gets the same route.
RoundRobin - Selects routes in a fixed repeating sequence, weighted by the configured route weights.

Figure 6.5. An Example of the RoundRobin Algorithm
Random - Picks a route randomly, using weights to bias the selection.
Throughput - Prefers the route whose interface is currently handling less traffic, based on throughput.
Usage - Chooses the route currently being used by the fewest active flows, adjusted by route weight.

The algorithm works only across routes within the same RLB profile. This ensures isolation between different groups of routes, preventing route selection behavior in one profile from affecting another.

The following example shows the use of the Flow algorithm:

Example 6.6. RLB Operation Using the Flow Algorithm

This example demonstrates how the Flow algorithm can be used to distribute traffic across multiple routes without requiring stickiness. The algorithm uses a hash of the flow's source and destination IPs, ports, and protocol to consistently select a route for each new flow.

Since this algorithm is deterministic, the same flow will always result in the same route choice, as long as the available routes and their weights remain unchanged. Due to the deterministic nature of the flow algorithm, stickiness is not required.

Command-Line Interface

Create an RLB profile using the Flow algorithm without stickiness:

System:/> add RLBProfile MyProfile
			 Algorithm=Flow
			 Stickiness=None

Set the profile as the default for the routing table:

System:/> set RoutingTable MyTable
			   DefaultRLB=MyProfile

Stickiness

Stickiness controls whether cOS Stream should reuse a previously selected route for traffic matching a specific source or destination. This behavior is defined in the RLB profile using the Stickiness parameter and is maintained for as long as the configured Keepalive condition is met. Reusing the same route can improve consistency, performance, and compatibility with stateful applications.

The following stickiness options are available:

None - No stickiness is applied. A new route selection is made for each new flow, based purely on the algorithm.
SourceIP - The same route is used for all traffic from a given source IP and receive interface.
SourceNetwork - The same route is used for traffic from the same source network range and receive interface.
DestinationIP - The same route is used for traffic to the same destination IP.
DestinationNetwork - The same route is used for traffic to the same destination network range.

The scope of stickiness is limited to the RLB profile. Previously selected routes based on algorithm and stickiness are not shared between different profiles.

The following example shows how to configure stickiness for source networks:

Example 6.7. Stickiness Configuration Example

This configuration sets up stickiness by source network. The same route will be reused for traffic from a given network, as long as there are flows open from that group. The Usage keepalive option ensures the stickiness remains in effect as long as flows are active.

Command-Line Interface

System:/> add RLBProfile MyProfile
			 Algorithm=RoundRobin
			 Stickiness=SourceNetwork
			 Keepalive=Usage

System:/> set RoutingTable MyTable DefaultRLB=MyProfile

RLB Weighting

Route weighting is used to influence how often a route is selected during load balancing. Each route can be assigned a numeric weight that adjusts its likelihood of being selected, depending on the algorithm used in the RLB profile.

Higher weight values increase the probability or preference for a route, while lower values reduce it. Weights are especially useful when different links have varying capacity or priority. The default weight value is 100.

The meaning of the weight varies depending on the selected algorithm:

Flow, SourceIP, DestinationIP - Weight biases how often a route is selected for matching traffic.
RoundRobin - The weight determines how many times a route is selected before moving to the next one in the sequence.
Random - The weight adjusts the probability of a route being selected from the pool of matching routes.
Throughput - The interface throughput is divided by the route's weight. Routes with lower resulting scores are preferred.
Usage - The number of active flows is divided by the route's weight. Lower scores are preferred during selection.

In all algorithms only the relative difference between weights matters. For example, assigning weights of 100 and 200 has the same effect as using 1 and 2. Use consistent scaling across routes for predictable results.

The following example applies different weights to two routes based on their relative capacity:

Example 6.8. Weighted Route Configuration Example

This example shows how to configure route weights in an RLB setup. The profile will use the Flow algorithm, and different weights will be applied to two routes based on the relative capacity of the WAN interfaces. The effect of this is that traffic will be distributed proportionally — for example, if one route has twice the weight of the other, it will be selected roughly twice as often during route lookups, assuming all other conditions are equal.

Command-Line Interface

First, create a new RLB profile named MyProfile that uses the Flow algorithm. Stickiness and keepalive settings are also defined.

System:/> add RLBProfile MyProfile
			 Algorithm=Flow
			 Stickiness=None
			 Keepalive=Timeout

Next, change context to the routing table so that new routes can be added:

System:/> cc RoutingTable main

Now, add the first route using interface WAN1 with gateway WAN1_GW. This route is assigned a weight of 200, reflecting a higher-capacity link.

System:/RoutingTable/main> add Route Interface=WAN1
			   Network=all-nets
			   Gateway=WAN1_GW
			   Weight=200
			   RLBProfile=MyProfile

Add the second route using interface WAN2 with gateway WAN2_GW. This route has a lower weight of 100.

System:/RoutingTable/main> add Route Interface=WAN2
			   Network=all-nets
			   Gateway=WAN2_GW
			   Weight=100
			   RLBProfile=MyProfile

Finally, return to the root context:

System:/RoutingTable/main> cc
System:/>

These configuration changes must now be saved by entering an activate followed by a commit command.

Keepalive

The Keepalive setting in an RLB profile determines how long cOS Stream should continue to use a previously selected route when stickiness is enabled. It defines the conditions for when stickiness should expire.

Keepalive applies only when stickiness is set to something other than None. It allows administrators to fine-tune how long a route is remembered for a specific source or destination before cOS Stream evaluates a new route using the algorithm.

The following keepalive options are available:

Deadline - Retain the routing decision for a fixed time window, then discard it regardless of usage.
Forever - Retain the routing decision indefinitely, or until the profile's limit is reached. This limit is controlled by the MaxGroups parameter in the RLB profile, which defines the maximum number of concurrent entries. The default value is 2048.
Timeout - Retain the routing decision as long as new flows are being opened. A timeout can be set to allow it to persist for a period after the last matching flow. The default value is 10 seconds.
Usage - Retain the routing decision while there are any active flows using it. A timeout can be set for additional persistence after all flows have closed.

Keepalive behavior, like stickiness itself, is isolated per RLB profile. Decisions are not shared across profiles.

RLB Algorithm Reset and Reset Behavior

Some RLB algorithms and stickiness options maintain internal state while cOS Stream is running. This includes values such as which route was last used in a round robin sequence or which route was previously selected for a particular source or destination.

In most cases, this internal state is reset or cleared when any of the following events occur:

System restart - All algorithm state is cleared when cOS Stream restarts.
Reconfiguration - Applying a new configuration may reset routing-related state depending on the scope of the configuration changes.
HA failover - RLB state is not fully synchronized between HA nodes. However, if stickiness is associated with active, HA-synchronized flows, that stickiness will be preserved after failover. Other internal state, such as round robin positions or inactive sticky entries, may be reset.

In all of these cases, cOS Stream will resume normal operation using the current configuration, but routing decisions may differ from before the reset since the prior state is lost.

For example, a client that was previously routed over the WAN1 interface using SourceIP stickiness may be assigned a different route (e.g. WAN2) after a failover or restart. This is expected behavior.

	Note: Reset behavior may affect path consistency
	When RLB state is reset due to system restart, reconfiguration, or HA failover, cOS Stream may select a different route for the same traffic compared to before the event (depending on algorithm). This is expected and reflects the stateless or partially synchronized nature of routing algorithm and stickiness memory.

Route Load Balancing Scenario A

This section describes a typical route load balancing scenario where clients behind cOS Stream access external servers via two separate Internet links.

The topology consists of a firewall with a LAN interface connected to three clients: Client A, Client B, and Client C.

On the external side, the firewall has two uplink interfaces: WAN1 and WAN2, each connected to separate Internet providers via gateway routers GW ISP1 and GW ISP2, respectively. Beyond the ISPs lies the Internet cloud.

In this example, RLB is used to distribute outbound traffic across both Internet links. Stickiness is based on the source IP, meaning each client will consistently use the same outbound path for as long as flows remain open. This provides a degree of session affinity and helps avoid issues with protocols sensitive to IP changes.

Figure 6.6. Route Load Balancing Scenario A, Dual ISPs With Different Connection Speeds

The two routes to the Internet are added to the main routing table with equal metrics. We will use different route weights to bias the route selection towards WAN1. The RLB profile defines the selection algorithm and stickiness settings.

Route No.	Interface	Destination	Gateway	Metric
1	WAN1	all-nets	GW1	100
2	WAN2	all-nets	GW2	100

Weight vs Metric on a Route

Both Weight and Metric are used to influence routing decisions, but they serve very different purposes:

The Metric value is used first, during route lookup, to determine which routes are considered. Only routes with the lowest metric among matching routes are included in route load balancing.
The Weight value is then used by the selected algorithm to decide how often a particular route is chosen. It only affects the final selection among routes with equal metric and matching network range.

In summary: Metric filters which routes are eligible, while Weight controls how often each eligible route is used.

IP Rules

Any traffic requires both a route and an IP rule allowing the desired traffic. The following rules will allow traffic to be forwarded to either ISP and will NAT the traffic using the external IP addresses of interfaces WAN1 and WAN2.

Index	Action	Src Iface	Src Net	Dest Iface	Dest Net	Service	Translation
1	Allow	LAN	LAN_net	WAN1	all-nets	all_services	Src NAT
2	Allow	LAN	LAN_net	WAN2	all-nets	all_services	Src NAT

The service all_services is used in the above IP rule set entries as an example. In a real deployment, this should be replaced with a specific service or service group that reflects the intended traffic — the narrower, the better.

Example 6.9. Setting Up RLB in a Dual ISP Scenario

This configuration sets up the RLB profile, configures the routing table, and defines the routes through both ISPs. Route selection will use the Flow algorithm with None as configured stickiness. Stickiness is not needed here, as the full name of the Flow algorithm is Deterministic Flow Selection. This means that the Stickiness option has no effect, since the algorithm already ensures the same route is selected for each unique flow. Outbound traffic will favor WAN1 due to a higher weight.

The weight value in this scenario means that WAN1 would receive approximately twice as many flows as WAN2, assuming an equal amount of traffic matches both routes. Over time, this distribution may vary depending on flow timeouts, client traffic patterns, and other factors.

Command-Line Interface

Create the RLB profile with the desired algorithm and stickiness:

System:/> add RLBProfile MyProfile
			 Algorithm=Flow
			 Stickiness=None
			 Keepalive=Usage

Change context to the main routing table:

System:/> cc RoutingTable main

Add a route for WAN1 with a higher weight:

System:/RoutingTable/main> add Route Interface=WAN1
			   Network=all-nets
			   Gateway=GW1
			   Weight=200
			   RLBProfile=MyProfile

Add a second route for WAN2 with a lower weight:

System:/RoutingTable/main> add Route Interface=WAN2
			   Network=all-nets
			   Gateway=GW2
			   Weight=100
			   RLBProfile=MyProfile

Return to the root context:

System:/RoutingTable/main> cc
System:/>

To apply these changes, issue the activate and commit commands.

Route Load Balancing Scenario B - Dynamic Routes

This section describes a second route load balancing scenario where clients behind cOS Stream access external servers via two separate Internet links that are configured to use Border Gateway protocol (BGP).

The topology consists of a firewall with a LAN interface connected to three clients: Client A, Client B, and Client C.

On the external side, the firewall has two uplink interfaces: WAN1 and WAN2, each connected to a router. Beyond the routers lies the Internet cloud. The entire setup is configured to use BGP, including the cOS Stream firewall. It is assumed that BGP is already configured in both the firewall and connected routers, for more information about BGP see Section 6.9, BGP

Since BGP is a dynamic routing protocol that sends and receives routes automatically, it is not possible to configure an RLB profile directly on a route, as was done in the previous example. To enable route load balancing on dynamically learned routes, the RLB profile must be set on the routing table itself.

Figure 6.7. Route Load Balancing Scenario B, Dynamic Routes Through BGP

	Note: Dynamic Routes Not Limited to BGP
	Even though this example uses BGP, dynamic routes can also be added to the routing table from other functions and interfaces such as IPsec and more.

	Important: ECMP Is Required for RLB With BGP
	When using route load balancing with BGP, cOS Stream requires multiple routes to have equal cost in order to perform load balancing. This is commonly known as ECMP (Equal-Cost Multi-Path). If BGP does not provide routes with identical cost, only the lowest-cost route will be used and route load balancing will not be applied.

IP Rules

The following IP rules will allow traffic to be forwarded to either WAN1 or WAN2 interface and will NAT the traffic using the external IP addresses of interfaces WAN1 or WAN2.

Index	Action	Src Iface	Src Net	Dest Iface	Dest Net	Service	Translation
1	Allow	LAN	LAN_net	WAN1	all-nets	all_services	Src NAT
2	Allow	LAN	LAN_net	WAN2	all-nets	all_services	Src NAT

Example 6.10. Setting Up RLB in a Dynamic Route Scenario

This configuration sets up an RLB profile using the Throughput algorithm with SourceIP as configured stickiness and keepalive set to Usage. This has the effect that the current interface throughput is used to select the appropriate route, biased towards routes with higher weight. The interface with the lowest throughput will be used to smoothly distribute the throughput over multiple matching routes.

Stickiness is enabled to ensure that traffic from the same source continues to use the same route, which helps prevent reordering and supports connection persistence. Keepalive is used to retain the stickiness state as long as flows are active.

Since BGP automatically installs routes, RLB cannot be configured on individual routes. Instead, the RLB profile must be applied to the routing table. This is not supported on the <main> routing table, so a separate routing table (e.g., MyTable) should be used.

Command-Line Interface

Create the RLB profile with the desired algorithm, Stickiness and Keepalive:

System:/> add RLBProfile My_RLB_Profile
			 Algorithm=Throughput
			 Stickiness=SourceIP
			 Keepalive=Usage

Use the newly created RLB profile on the routing table:

System:/> set RoutingTable MyTable DefaultRLB=My_RLB_Profile

To apply these changes, issue the activate and commit commands.

	Note: The Security Equivalent Interface Option
	It may be needed to set the SecurityEquivalentInterfaces option on involved interfaces if asymmetric return paths are possible — which is often the case in multi-uplink BGP scenarios with RLB. It's not directly required by RLB or BGP — but it can be critical for session continuity and avoiding traffic drops if return traffic is not predictable.

	Caution: Throughput Algorithm With Only One Uplink Interface
	Using the Throughput algorithm in a situation where there is only one uplink interface is not recommended, since the algorithm depends on the interface throughput. In such cases, an algorithm like Round-Robin would be more suitable.