cOS Stream 4.00.01 CNF Series Getting Started Guide

Several additional components, that might not be available by default in all Kubernetes distributions, are needed to support running pods with multiple network interfaces. For starters, a CNI plugin that supports adding multiple interfaces is needed. cOS Stream supports using "Multus" to provide this functionality:

https://github.com/k8snetworkplumbingwg/multus-cni

To achieve high performance, and to be able to use DPDK, dedicated network devices need to be added to the pod. cOS Stream supports using "SR-IOV Network Device Plugin for Kubernetes" to manage these pools of network device resources that the pods can allocate devices from, which can be found here:

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin

An appropriate CNI plugin to assign and configure the devices in the pod are needed, a CNI plugin such as the "SR-IOV CNI plugin" which can be found here:

https://github.com/k8snetworkplumbingwg/sriov-cni

An alternative to the previous plugin ("SR-IOV CNI plugin") would be the "host-device" CNI plugin located here:

https://www.cni.dev/plugins/current/main/host-device

There are additional components, that are not required, but can help with some automation, such as the "Network Resources Injector" that can automate adding the resource allocations needed by the networks attached to the pod, to the pod specification.

https://github.com/k8snetworkplumbingwg/network-resources-injector

There may also be other components that are needed to manage the traffic flow to/from the firewall, in Kubernetes or in the surrounding network infrastructure.

This section outlines the configuration steps for running the cOS Stream container. The configuration process involves setting up various Kubernetes features and components to ensure that the containerized firewall operates efficiently and securely within the Kubernetes environment.

This section contains a full example of a pod specification that can be used to deploy the firewall. This is based on "netshield-pod.yaml" that is distributed with the software, but with some of the more exotic settings removed for better overview. It can be good enough for an initial test but since it uses "host-path" for persistent storage it is not suitable for more production-like deployments. For those scenarios something like "netshield-ls-statefulset.yaml" is more suitable. The differences are basically in the storage and that a stateful set is used instead of a single pod.

The details that are specific to the firewall product are mostly the same so the simpler single pod specification is a good starting point to not get distracted by other aspects that likely vary anyway depending on the environment where the firewall is being deployed. In later sections, parts of this example is presented under various themes, such as CPU, Memory or Networking, focus on those parts that affects that theme.

For samples of .yaml configuration files see examples found in: clavister-cos-stream-4.00.01.34-cnf-x64-generic-deploy.tar.gz

# Example pod definition to deploy a netshield pod.

apiVersion: v1
kind: Pod
metadata:
 name: netshield
 annotations:
      
# Optional request to multus, to use a specific network configuration
# as the default pod network
# v1.multus-cni.io/default-network:
# <name of the NetworkAttachmentDefinition to use>
       
# Optional request to multus, to request additional network interfaces

k8s.v1.cni.cncf.io/networks: lan-network@lan, wan-network@wan
 spec:
       
containers:
 - name: netshield
 image: '<some-registry-url>/cos-stream:4.00.00.00'
      
 env:

# Optional CPU list specification for control plane and data plane.
# Default is to detect CPUs via the cgroup cpuset controller.
# These needs to be used if dedicated CPU resources (pod in guaranteed
# QoS class and static CPU manager policy) are not used.
#- name: CP_CPU_LIST
#  value: '0' # rangelist format supported, for instance: '4,6,9-12'
#- name: DP_CPU_LIST
#  value: '1' # rangelist format supported, for instance: '4,6,9-12'
       
# NETS will expose the 'k8s.v1.cni.cncf.io/networks' annotation in the
# container's environment. No user configuration required.

- name: NETS
 valueFrom:
  fieldRef:
   fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/networks']
       
# CPU_REQ and CPU_LIMIT will expose the amount of CPU resources
# requested to the container. No user configuration required
# other than to make sure that the "containerName" matches the
# container's name.

- name: CPU_REQ
 valueFrom:
  resourceFieldRef:
   containerName: netshield
   resource: requests.cpu
   divisor: 1m
  - name: CPU_LIMIT
   valueFrom:
    resourceFieldRef:
     containerName: netshield
     resource: limits.cpu
     divisor: 1m
       
# The HUGEPAGES* variables below require that the DownwardAPIHugePages
# feature gate is enabled. When using downward API for hugepages and
# 2MB hugepages this one should be included:

- name: HUGEPAGES_2M
 valueFrom:
  resourceFieldRef:
   containerName: netshield
   resource: limits.hugepages-2Mi
   divisor: 1Mi

# When using downward API for hugepages and 1GB hugepages this one
# should be included:

- name: HUGEPAGES_1G
 valueFrom:
  resourceFieldRef:
   containerName: netshield
   resource: limits.hugepages-1Gi
   divisor: 1Mi
       
# When not using downward API for hugepages this one needs to be set
# and kept in sync with the amount of hugepages requested for the
# container.
#- name: DP_MEMORY_MB
#  value: '300'
       
resources:

# Requests and limits should usually be identical to qualify the
# pod for the "Guaranteed" quality of service class.

 requests:
  cpu: '10'
  hugepages-2Mi: 300Mi

#hugepages-1Gi: 1Gi

  memory: 900Mi

# Allocate resources/devices needed by the extra networks.
# (These can be injected automatically by the "Network Resources
# Injector"
# https://github.com/k8snetworkplumbingwg/network-resources-injector.)
# The "resource prefix" and the "resource name" must match the
# configuration of the device plugin used, for instance, the "SR-IOV
# Network Device Plugin for Kubernetes
# (https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin)

  example.com/lan_device: '1'
  example.com/wan_device: '1'
 limits:
  cpu: '10'
  hugepages-2Mi: 300Mi

#hugepages-1Gi: 1Gi

  memory: 900Mi
  example.com/lan_device: '1'
  example.com/wan_device: '1'
       
  securityContext:
   privileged: true
       
volumeMounts:

# If using just one size of hugepages then this generic
# one can be used, otherwise use one of each type..

- mountPath: /hugepages
 name: hugepages

 #- mountPath: /hugepages_2MB
#  name: hugepages-2mb
#- mountPath: /hugepages_1GB
#  name: hugepages-1gb

- mountPath: /etc/podinfo
 name: podinfo
 
 # The system expects persistent storage to be
# mounted/available at /mnt/storage.
 
- mountPath: /mnt/storage
 name: storage
volumes:

# If using just one size of hugepages then this generic one can
# be used, otherwise use one of each type..

- name: hugepages
 emptyDir:
  medium: HugePages

#- name: hugepages-2mb

#  emptyDir:
#    medium: HugePages-2Mi
#- name: hugepages-1gb
#  emptyDir:
#    medium: HugePages-1Gi

- name: podinfo
 downwardAPI:
  items:
   - path: "network-status"
    fieldRef:
     fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/network-status']
- name: storage
 hostPath:
  path: /opt/netshield/storage/pod1
  type: Directory

The pod specifications distributed in the software package needs to be updated so that the image specification matches where the container image is pulled from,as there is no public registry hosting this image. This should be set to the local image registry to which the container image in the software package was pushed.

apiVersion: v1
kind: Pod
metadata:
  name: netshield
spec:
  containers:
  - name: netshield
    image: '<some-registry-url>/cos-stream:4.00.00.00'

This section lists some addition environment variables that can be used to tweak the system in addition to the variables listed in other sections, such as the CPU section.

apiVersion: v1
	kind: Pod
	spec:
	  containers:
	  - name: netshield
		env:

# Optional switch to prevent the system from creating new cgroups and
# instead just set affinity on control plane processes. The system will
# still try to detect what CPU resources to use from the cgroup unless
# the *_CPU_LISTs are set.
#- name: NO_CREATE_CGROUP
#  value: '1' # Not set is default.

# Number of cores (not CPUs) to assign to control plane.
# Only used when CPU resources are auto-detected (that is, not used when
# the *_CPU_LISTs are specified).
#- name: NUM_CP_CORES
#  value: '1'

# Optional packet buffer size
#- name: PACKET_BUFFER_SIZE
#  value: '2500'

# Optional number of I/O threads
#- name: IO_THREADS
#  value: '2'

# Optional list of interface types that should use af_xdp instead of the
# default af_packet (when a native driver can't be used).
#- name: AF_XDP_TYPES
#  value: '' # Could be set to something like 'veth i40e'. Using an empty
# list for AF_XDP_TYPES corresponds to the default behavior of always
# using af_packet.

# Optional list of interface types that should use af_packet instead of the
# default af_xdp (when a native driver can't be used).
#- name: AF_PACKET_TYPES
#  value: ixgbevf ixgbe

# Don't set both of AF_XDP_TYPES and AF_PACKET_TYPES, use just the one that
# implies the desired default or none of them (resulting in af_packet)
# (AF_XDP_TYPES takes precedence and hence af_packet is the
# default default).

# Optional list of PCI device drivers that should use DPDK's built-in type
# specific driver instead of af_packet of af_xdp.
# It is not recommended to remove any drivers from the default value
# (especially the first three that don't have a corresponding
# netdev/link in Linux). The main purpose would be to be able to add
# rebranded Mellanox drivers (if they are rebranded on that level),
# but can also be used to force using af_packet or af_xdp on mellanox
# devices.
#- name: NATIVE_DRIVERS
#  value: vfio-pci igb_uio uio_pci_generic mlx4_core mlx5_core

# Optional AF-XDP busy budget (uses 0/disabled per default).
#- name: AF_XDP_BUSY_BUDGET
#  value: '0'

This section details the configuration of memory resources for the Firewall within a Kubernetes container, focusing on the distinct requirements of the control plane and data plane.

apiVersion: v1
	kind: Pod
	spec:
	  containers:
	  -name: netshield
		resources:
		  requests:
			memory: 900Mi
		  limits:
			memory: 900Mi

apiVersion: v1
kind: Pod
 containers:
 - name: netshield
  env:
  - name: HUGEPAGES_2M
   valueFrom:
    resourceFieldRef:
     containerName: netshield
     resource: limits.hugepages-2Mi
     divisor: 1Mi
  - name: HUGEPAGES_1G
   valueFrom:
    resourceFieldRef:
     containerName: netshield
     resource: limits.hugepages-1Gi
     divisor: 1Mi

#- name: DP_MEMORY_MB
#  value: '300'
	 
  resources:
   requests:
    hugepages-2Mi: 800Mi
    hugepages-1Gi: 1Gi
   limits:
    hugepages-2Mi: 800Mi
    hugepages-1Gi: 1Gi
	 
  volumeMounts:

#- mountPath: /hugepages
#  name: hugepages

  - mountPath: /hugepages_2MB
   name: hugepages-2mb
  - mountPath: /hugepages_1GB
    name: hugepages-1gb
volumes:

#- name: hugepages
#  emptyDir:
#    medium: HugePages

 - name: hugepages-2mb
  emptyDir:
   medium: HugePages-2Mi
 - name: hugepages-1gb
  emptyDir:
   medium: HugePages-1Gi

.valueFrom.resourceFieldRef.resource:
Unsupported value: "limits.hugepages-2Mi" (or "..-1Gi")

This is an example demonstrating how to allocate CPU resources to the container.

apiVersion: v1
kind: Pod
spec:
 containers:
 - name: netshield
  env:

# Optional CPU list specification for control plane and data
# plane. Default is to detect CPUs via the cgroup cpuset
# controller. These needs to be used if dedicated CPU resources
# (pod in guaranteed QoS class and static CPU manager policy)
# are not used.

#- name: CP_CPU_LIST
#  value: '0' # rangelist format supported, for instance: '4,6,9-12'
#- name: DP_CPU_LIST
#  value: '1' # rangelist format supported, for instance: '4,6,9-12'
	 
# CPU_REQ and CPU_LIMIT will expose the amount of CPU resources
# requested to the container. No user configuration required other than
# to make sure that the "containerName" matches the container's name.

   - name: CPU_REQ
    valueFrom:
     resourceFieldRef:
      containerName: netshield
      resource: requests.cpu
      divisor: 1m
   - name: CPU_LIMIT
    valueFrom:
     resourceFieldRef:
      containerName: netshield
      resource: limits.cpu
      divisor: 1m
   resources:
    requests:
     cpu: '10'
    limits:
     cpu: '10'

If the cluster is properly configured to support pod in the Guaranteed QoS class. Then what is needed is to add the CPU_REQ and CPU_LIMIT environment variables, making sure that the "containerName" matches the containers name and request the required amount of CPU in the resources section.

Both request and limit needs to be set to the same value for the pod to qualify for "Guaranteed" QoS. If for some reason those requirements can not be met then the environment variables CP_CPU_LIST and DP_CPU_LIST can be used as a workaround. The system will expect those CPUs to be dedicated to this pod.

There are some further tweaks possible, see the full specification examples for documentation of those.

This section focuses on the parts of the example that are related to attaching additional network interfaces to the pod, besides the default pod network (eth0). In this example Multus is used to attach two additional network interfaces:

apiVersion: v1
kind: Pod
metadata:
 name: netshield
 annotations:
  k8s.v1.cni.cncf.io/networks: lan-network@lan, wan-network@wan
spec:
 containers:
 - name: netshield
 env:
   - name: NETS
     valueFrom:
      fieldRef:
       fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/networks']
	 
 resources:
  requests:
   example.com/lan_device: '1'
   example.com/wan_device: '1'
  limits:
   example.com/lan_device: '1'
   example.com/wan_device: '1'
  
 volumeMounts:
  - mountPath: /etc/podinfo
    name: podinfo
volumes:
 - name: podinfo
  downwardAPI:
   items:
    - path: "network-status"
     fieldRef:
      fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/network-status']

The devices allocated under resources are the ones that are needed by the extra networks. The "resource prefix" (here "example.com") and the "resource name" (here: "lan_device" and "wan_device") must match the configuration of the device plugin used, for instance, the "SR-IOV Network Device Plugin for Kubernetes" which can be found here:

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin

The examples archive contains examples of both a device configuration map for the device plugin (sriovdp-configmap.yaml) and some example network attachment definitions (networks.yaml). These examples show where these names appear in the various objects. The resource allocations in the pod specification can be injected automatically by for instance the "Network Resources Injector", which can be found here:

https://github.com/k8snetworkplumbingwg/network-resources-injector

The injection would be based on the "networks" annotation in the pod specification and the "resourceName" annotation in the network attachment definition.

There are some additional tweaks that can be applied, also affecting the default pod network. See the full pod specification examples for documentation of those.

The firewall pod needs persistent storage to function properly. This storage will be used for configuration, diagnostic console logs, to store crashdumps and other persistent states. The example here is the simplest possible where storage provided by a hostPath volume. This only works reliably with a single worker node cluster or if the pod is pinned to a specific worker node by some other means, so, basically only for initial tests.

For production use a more advanced storage solution is needed. There are more examples distributed with the software using local storage or NFS to provide storage for the firewall, see netshield-ls-statefulset.yaml and netshield-nfs-statefulset.yaml, and associated files. From the firewall's perspective the important part is that storage exists and is mounted at this mountPath:

/mnt/storage.

	Note: The .yaml files
	The .yaml files referred above are provided in a separate file included in every release. The file name is typically named : clavister-cos-stream-4.00.01.34-cnf-x64-generic-deploy.tar.gz

apiVersion: v1
kind: Pod
spec:
 containers:
  - name: netshield
    volumeMounts:

# The system expects persistent storage to be
# mounted/available at /mnt/storage.

  - mountPath: /mnt/storage
    name: storage
 volumes:
  - name: storage
    hostPath:
     path: /opt/netshield/storage/pod1
     type: Directory

Liveness probes allow Kubernetes to continuously monitor the health of the firewall to detect if the container cannot forward traffic reliably and should be restarted. Both TCP and HTTP probes are supported.

It is recommended to configure startup probes in addition to liveness probes to make sure that the system has sufficient time to start up. Below is an example of this using a named port, in this case 8080. The system waits for 10 seconds before checking if the container has started each second. Once the container is up and running, Kubernetes will move on to monitor the health of the system every 10 seconds. If the container has not responded within a total of 10 + 30 seconds on startup or to three consecutive liveness probes, the firewall pod considered to be malfunctioning and will be restarted. Kubernetes will then wait for an additional 60 seconds to allow the system to shutdown gracefully.

	Note: Grace period
	If the grace period is set too low, the system may not have enough time to save states to permanent storage, some of which can be used for troubleshooting.

ports:
- name: health-port
	containerPort: 8080

startupProbe:
  httpGet:
	port: health-port
	path: /health
  initialDelaySeconds: 10
  periodSeconds: 1
  timeoutSeconds: 1
  failureThreshold: 30
	 
livenessProbe:
  httpGet:
	port: health-port
	path: /health
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
  terminationGracePeriodSeconds: 60

TCP probes are setup similarly with tcpSocket instead of httpGet and `path` omitted. The HEALTH_PORT_TCP_HTTP environment variable is used to specify which port the firewall should listen on for probes.

- name: HEALTH_PORT_TCP_HTTP
	value: '8880'
		

For each Ethernet interface a number of IP address objects will be created in the default configuration. These IP address objects will be assigned values run-time. The names of the address objects will be based on the interface name (in the pod specification/environment). This name will be used as a prefix and the following address objects will be created:

<prefix>_cni_ip

<prefix>_cni_priv0

<prefix>_cni_priv1

<prefix>_cni_br

<prefix>_cni_ip4_<#>

<prefix>_cni_ip6_<#>

There is no point in assigning IPs to these address objects, since they will always use the dynamically assigned value, so for instance if a different address than what eth0_cni_ip contains should be used as IP address on eth0 then change the IPAddress property on the EthernetInterface object instead of trying to change the value of eth0_cni_ip.

For each DNS server found during the first startup, an IP address object is provided named according to the following pattern: system_dnsserver<#>_ip, (system_dnsserver1_ip, etc.). Additionally a DNSServer object is added for each DNS server under the DNS object, using the corresponding IP address object as IPAddress.

The IP address objects are updated at every restart of the pod, but not created/removed if the number of servers are changed, that has to be done manually. DNSServer objects are not updated either, other than implicitly through the IP address objects being updated.

System:/> show Address IPAddress system_dnsserver1_ip

          Property  Value                 Remarks
------------------  --------------------  --------------------
             Name:  system_dnsserver1_ip
          Address:  0.0.0.0
    ActiveAddress:  10.152.183.10         Dynamically assigned
FQDNValidAfterTTL:  <empty>
         Comments:  <empty>

System:/>  cc DNS
System:/DNS>  show

DNSServer

  # IP address           Routing table
  - -------------------- -------------
  1 system_dnsserver1_ip main
    			

The system will per default send some of the system events to kubernetes as kubernetes events. The events that are reported this way is a subset of the events that can be seen in the dconsole log inside the system. This includes for instance events about start-up, reconfigurations, and shutdown. There is a setting that controls if these events are exported as kuberenetes events or not called ReportKubeEvents located under MiscSettings.

It is a boolean yes/no setting.

As mentioned in Chapter 4, Firewall Configuration there is an SSH remote management object created per default that allows SSH access to the firewall's CLI from the pod network. It is recommended to add a Kubernetes service in order to keep track of the management IP addresses of the pod.

Information about direct access to the Command Line Interface (CLI) for firewall management.

The CLI can also be accessed by directly executing the CLI command with, for instance, kubectl:

kubectl exec -it netshield -- cli

The firewall can also be managed from Clavister's centralized management product InCenter by adding it like any other firewall. A kubernetes service would be a good way to keep track of the firewall's management IP, unless it is assigned a static IP.

Below is a list of software versions that has been verified by Clavister to work with this guide. Newer versions released after this guide was written will most likely work as well but they are used at your own risk as full functionality have not been verified.

The following Network Interface Cards (NICs) have been verified to work during quality assurance of this release.

For best performance and in some cases functionality, it is best to expose base Ethernet interfaces to the firewall. If you want to use bonding/link aggregation, it is better to expose the member interfaces to the firewall, as-is, and configure link aggregation interface in the firewall configuration than to expose a bond-device to the pod. Same with VLAN, it is better to expose the Ethernet interface than some virtual VLAN interface, however that might be harder since the Ethernet interface might be needed elsewhere as well.

For VLAN and SR-IOV it is possible to set a single VLAN directly on the SR-IOV VF, that is, set the vlan parameter in the configuration for the SR-IOV CNI plugin, and then just treat that as any Ethernet interface as far as the firewall is concerned having the NIC automatically adding and stripping the VLAN tag. The firewall will then be restricted to that VLAN only.

VLAN over virtual Ethernet interfaces, like the virtual Ethernet pair likely used to provide the interface for the default pod network "/eth0", requires VLAN offload to be disabled on the "other end" of the Ethernet connection. This to ensure that the firewall receives the packets with the VLAN header in place in the raw packet data. Otherwise the VLAN information might not be picked up by the firewall. Disabling VLAN offload might require some custom scripts on the host.

When it comes to interface naming, vEth0 should be considered a reserved name.

For full support/identical naming inside the firewall, interface names should match the below regular expression. The current limit is 26 characters.

[a-zA-Z][a-zA-Z0-9_-]{,14}

A few other characters might work, but the interface will have a name without those characters inside the firewall (in the firewall's configuration), possibly with other modifications as well to make them unique in this context.

Whitespace or any of the below characters in the interface names are known to cause problems and are therefore unsupported. Other characters, except those in the first paragraph, are used at your own risk.

/ : " ' , = [ ; %