cOS Stream 4.00.00 CNF Series Getting Started Guide


Table of Contents

1. Introduction
2. Configuring Kubernetes
2.1. Additional Components
2.2. Configuration
2.2.1. CPU Manager
2.2.2. Topology Manager
2.2.3. Memory Manager
2.2.4. CNI/Device Plugins/Network Attached Definitions
2.2.5. SR-IOV CNI config details
3. Configuring/Deploying the Firewall Pod
3.1. Example
3.2. Image
3.3. Environment
3.4. Memory
3.4.1. Control Plane
3.4.2. Data Plane
3.5. CPU
3.6. Additional Networks (Multus)
3.7. Persistent Storage
3.8. Monitoring Firewall Health
4. Firewall Configuration
4.1. Interface IP Addresses
4.2. DNS Servers
4.3. Kubernetes Events
5. Firewall Management
5.1. SSH
5.2. Direct access to the CLI
5.3. InCenter
6. Considerations
6.1. Compatible Versions
6.2. Verified NICs
6.3. Expose Ethernet NICs
6.4. VLAN over virtual Ethernet
6.5. Interface naming

Chapter 1: Introduction

[Note] Note: This document is also available in other formats

A PDF version of this document along with all current and older documentation in PDF format can be found at https://my.clavister.com.

It is also available in a framed HTML version.

What the container version of cOS Stream provides on a conceptual level is a high-performance containerized firewall that can be managed by the Kubernetes framework. In a typical scenario it leverages SR-IOV interfaces for increased throughput and to be able to handle large volumes of user data, while using the default cluster network for system management. cOS Stream can replace traditional firewalls, with the advantage of fitting in the Kubernetes echo system, and by using resource pools, the number of firewalls can be scaled up or down depending on cluster demands.

Though the firewall runs in a container it is different from a regular micro service in several ways. Some important aspects to note are:

  • The firewall is not automatically integrated into the traffic flow of the Kubernetes cluster. It is simply a firewall running in a container, so the customer/administrator is responsible for setting up additional network interfaces and routing the traffic that should be firewalled to or through the container, as well as configuring rules, routing etc. in the firewall. The following is assumed:

    1. The default pod network is used to manage the firewall only, not to forward traffic.

    2. The actual workload traffic is forwarded over the additional network interfaces attached to the pod.

Container Deployment Example

Figure 1.1. Container Deployment Example

  • The firewall container is running in polled mode, so it will roughly use the same amount of CPU resources regardless of traffic workload. That is, processes inside the container are running at full speed all the time. Expect to observe high CPU load from the outside. To see the actual load, the firewall needs to be queried.

    1. Due to this, it is expected that the firewall container is assigned dedicated CPU resources (for instance by using a static CPU manager policy in the cluster) and is running in the guaranteed QoS class.

  • Using a Deployment, StatefulSet etc. and scaling to multiple pods might not work as expected (or automatically).

  • The firewall is based on DPDK, so for high performance, NIC devices should be bound to a driver DPDK have native support for (such as, mlx5 or vfio-pci).

  • High Availability (HA) is achieved by deploying a firewall node pair, with the two firewall HA nodes on separate hardware nodes within the same Kubernetes cluster.

  • The firewall container does not support running across NUMA boundaries, the container should have all resources assigned from the same NUMA node, example:

    --topology-manager-policy=single-numa-node

The software package contains both the container image needed to run the system, an example of the image name:

clavister-cos-stream-4.00.00.34-cnf-x64-generic.tar.gz

The package also contains an archive with some examples/templates of the files needed to deploy it into kubernetes, an example of the image name:

clavister-cos-stream-4.00.00.34-cnf-x64-generic-deploy.tar.gz

Chapter 2: Configuring Kubernetes

This section describes some of the steps that are needed to fulfill the requirements for running cOS Stream in a container in Kubernetes as highlighted in the introduction section.

2.1. Additional Components

Several additional components, that might not be available by default in all Kubernetes distributions, are needed to support running pods with multiple network interfaces. For starters, a CNI plugin that supports adding multiple interfaces is needed. cOS Stream supports using "Multus" to provide this functionality:

https://github.com/k8snetworkplumbingwg/multus-cni

To achieve high performance, and to be able to use DPDK, dedicated network devices need to be added to the pod. cOS Stream supports using "SR-IOV Network Device Plugin for Kubernetes" to manage these pools of network device resources that the pods can allocate devices from, which can be found here:

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin

An appropriate CNI plugin to assign and configure the devices in the pod are needed, a CNI plugin such as the "SR-IOV CNI plugin" which can be found here:

https://github.com/k8snetworkplumbingwg/sriov-cni

An alternative to the previous plugin ("SR-IOV CNI plugin") would be the "host-device" CNI plugin located here:

https://www.cni.dev/plugins/current/main/host-device

There are additional components, that are not required, but can help with some automation, such as the "Network Resources Injector" that can automate adding the resource allocations needed by the networks attached to the pod, to the pod specification.

https://github.com/k8snetworkplumbingwg/network-resources-injector

There may also be other components that are needed to manage the traffic flow to/from the firewall, in Kubernetes or in the surrounding network infrastructure.

2.2. Configuration

This section outlines the configuration steps for running the cOS Stream container. The configuration process involves setting up various Kubernetes features and components to ensure that the containerized firewall operates efficiently and securely within the Kubernetes environment.

2.2.1. CPU Manager

Since cOS Stream is running in polled mode and requires dedicated CPU resources, a change to static policy is needed. See the following documentation for more details see:

https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies

Once the static policy is configured you need to set aside dedicated CPUs for pods in the Guaranteed Quality Of Service (QoS) class, for more details see:

https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources

2.2.2. Topology Manager

If Kubernetes is hosted on a server with multiple NUMA nodes, change to single NUMA node policy, see:

https://kubernetes.io/docs/tasks/administer-cluster/topology-manager

2.2.3. Memory Manager

If Kubernetes is hosted on a server with multiple NUMA nodes, change the memory manager policy to "static", see:

https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/

2.2.4. CNI/Device Plugins/Network Attached Definitions

To attach additional Ethernet interfaces, the following steps are needed:

  1. Configure the network interface devices to create the required number of virtual function devices.

  2. Bind the network interface devices (usually virtual function devices) to the correct driver. Mellanox devices can usually stay bound to their default kernel driver, but most other types of network interface devices need to be bound to the vfio-pci driver to work efficiently with the firewall.

  3. Install the required CNIs; Multus, SR-IOV CNI, host-device CNI etc.

  4. Set up the config map for the "SR-IOV Network Device Plugin", which organizes the devices into resource pools that the pods can request resources from. There is an example for this config map in the examples archive in sriovdp-configmap.yaml, but it is just an example and will need to be adjusted to the environment where the plugin will be deployed.

  5. Deploy the "SR-IOV Network Device Plugin".

  6. Set up the Network Attachment Definitions for the extra networks. There are some example network attachment definitions in the examples archive in networks.yaml, but they are just examples and will need to be adjusted to the environment where the firewall will be deployed.

  7. Extend the pod specification with networks annotation and resource requests etc. See Section 3.6, Additional Networks (Multus) for details regarding this step.

2.2.5. SR-IOV CNI config details

Set "spoofchk" to "off", "trust" to "on" and optionally set "vlan" to limit the virtual function to a single VLAN, in the CNI configuration in the network attachment definition objects, see:

https://github.com/k8snetworkplumbingwg/sriov-cni/blob/master/docs/configuration-reference.md

Chapter 3: Configuring/Deploying the Firewall Pod

Firewall pods can be deployed using various object types. For initial tests or simple scenarios, directly deploying a pod might be sufficient. In other cases perhaps a StatefulSet is better. This section focuses on the pod details that are needed either way.

3.1. Example

This section contains a full example of a pod specification that can be used to deploy the firewall. This is based on "netshield-pod.yaml" that is distributed with the software, but with some of the more exotic settings removed for better overview. It can be good enough for an initial test but since it uses "host-path" for persistent storage it is not suitable for more production-like deployments. For those scenarios something like "netshield-ls-statefulset.yaml" is more suitable. The differences are basically in the storage and that a stateful set is used instead of a single pod.

The details that are specific to the firewall product are mostly the same so the simpler single pod specification is a good starting point to not get distracted by other aspects that likely vary anyway depending on the environment where the firewall is being deployed. In later sections, parts of this example is presented under various themes, such as CPU, Memory or Networking, focus on those parts that affects that theme.

For samples of .yaml configuration files see examples found in: clavister-cos-stream-4.00.00.34-cnf-x64-generic-deploy.tar.gz

# Example pod definition to deploy a netshield pod.

apiVersion: v1
kind: Pod
metadata:
 name: netshield
 annotations:
      
# Optional request to multus, to use a specific network configuration
# as the default pod network
# v1.multus-cni.io/default-network:
# <name of the NetworkAttachmentDefinition to use>
       
# Optional request to multus, to request additional network interfaces

k8s.v1.cni.cncf.io/networks: lan-network@lan, wan-network@wan
 spec:
       
containers:
 - name: netshield
 image: '<some-registry-url>/cos-stream:4.00.00.00'
      
 env:

# Optional CPU list specification for control plane and data plane.
# Default is to detect CPUs via the cgroup cpuset controller.
# These needs to be used if dedicated CPU resources (pod in guaranteed
# QoS class and static CPU manager policy) are not used.
#- name: CP_CPU_LIST
#  value: '0' # rangelist format supported, for instance: '4,6,9-12'
#- name: DP_CPU_LIST
#  value: '1' # rangelist format supported, for instance: '4,6,9-12'
       
# NETS will expose the 'k8s.v1.cni.cncf.io/networks' annotation in the
# container's environment. No user configuration required.

- name: NETS
 valueFrom:
  fieldRef:
   fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/networks']
       
# CPU_REQ and CPU_LIMIT will expose the amount of CPU resources
# requested to the container. No user configuration required
# other than to make sure that the "containerName" matches the
# container's name.

- name: CPU_REQ
 valueFrom:
  resourceFieldRef:
   containerName: netshield
   resource: requests.cpu
   divisor: 1m
  - name: CPU_LIMIT
   valueFrom:
    resourceFieldRef:
     containerName: netshield
     resource: limits.cpu
     divisor: 1m
       
# The HUGEPAGES* variables below require that the DownwardAPIHugePages
# feature gate is enabled. When using downward API for hugepages and
# 2MB hugepages this one should be included:

- name: HUGEPAGES_2M
 valueFrom:
  resourceFieldRef:
   containerName: netshield
   resource: limits.hugepages-2Mi
   divisor: 1Mi

# When using downward API for hugepages and 1GB hugepages this one
# should be included:

- name: HUGEPAGES_1G
 valueFrom:
  resourceFieldRef:
   containerName: netshield
   resource: limits.hugepages-1Gi
   divisor: 1Mi
       
# When not using downward API for hugepages this one needs to be set
# and kept in sync with the amount of hugepages requested for the
# container.
#- name: DP_MEMORY_MB
#  value: '300'
       
resources:

# Requests and limits should usually be identical to qualify the
# pod for the "Guaranteed" quality of service class.

 requests:
  cpu: '10'
  hugepages-2Mi: 300Mi

#hugepages-1Gi: 1Gi

  memory: 900Mi

# Allocate resources/devices needed by the extra networks.
# (These can be injected automatically by the "Network Resources
# Injector"
# https://github.com/k8snetworkplumbingwg/network-resources-injector.)
# The "resource prefix" and the "resource name" must match the
# configuration of the device plugin used, for instance, the "SR-IOV
# Network Device Plugin for Kubernetes
# (https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin)

  example.com/lan_device: '1'
  example.com/wan_device: '1'
 limits:
  cpu: '10'
  hugepages-2Mi: 300Mi

#hugepages-1Gi: 1Gi

  memory: 900Mi
  example.com/lan_device: '1'
  example.com/wan_device: '1'
       
  securityContext:
   privileged: true
       
volumeMounts:

# If using just one size of hugepages then this generic
# one can be used, otherwise use one of each type..

- mountPath: /hugepages
 name: hugepages

 #- mountPath: /hugepages_2MB
#  name: hugepages-2mb
#- mountPath: /hugepages_1GB
#  name: hugepages-1gb

- mountPath: /etc/podinfo
 name: podinfo
 
 # The system expects persistent storage to be
# mounted/available at /mnt/storage.
 
- mountPath: /mnt/storage
 name: storage
volumes:

# If using just one size of hugepages then this generic one can
# be used, otherwise use one of each type..

- name: hugepages
 emptyDir:
  medium: HugePages

#- name: hugepages-2mb

#  emptyDir:
#    medium: HugePages-2Mi
#- name: hugepages-1gb
#  emptyDir:
#    medium: HugePages-1Gi

- name: podinfo
 downwardAPI:
  items:
   - path: "network-status"
    fieldRef:
     fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/network-status']
- name: storage
 hostPath:
  path: /opt/netshield/storage/pod1
  type: Directory

3.2. Image

The pod specifications distributed in the software package needs to be updated so that the image specification matches where the container image is pulled from,as there is no public registry hosting this image. This should be set to the local image registry to which the container image in the software package was pushed.

apiVersion: v1
kind: Pod
metadata:
  name: netshield
spec:
  containers:
  - name: netshield
    image: '<some-registry-url>/cos-stream:4.00.00.00'

3.3. Environment

This section lists some addition environment variables that can be used to tweak the system in addition to the variables listed in other sections, such as the CPU section.

apiVersion: v1
	kind: Pod
	spec:
	  containers:
	  - name: netshield
		env:

# Optional switch to prevent the system from creating new cgroups and
# instead just set affinity on control plane processes. The system will
# still try to detect what CPU resources to use from the cgroup unless
# the *_CPU_LISTs are set.
#- name: NO_CREATE_CGROUP
#  value: '1' # Not set is default.

# Number of cores (not CPUs) to assign to control plane.
# Only used when CPU resources are auto-detected (that is, not used when
# the *_CPU_LISTs are specified).
#- name: NUM_CP_CORES
#  value: '1'

# Optional packet buffer size
#- name: PACKET_BUFFER_SIZE
#  value: '2500'

# Optional number of I/O threads
#- name: IO_THREADS
#  value: '2'

# Optional list of interface types that should use af_xdp instead of the
# default af_packet (when a native driver can't be used).
#- name: AF_XDP_TYPES
#  value: '' # Could be set to something like 'veth i40e'. Using an empty
# list for AF_XDP_TYPES corresponds to the default behavior of always
# using af_packet.

# Optional list of interface types that should use af_packet instead of the
# default af_xdp (when a native driver can't be used).
#- name: AF_PACKET_TYPES
#  value: ixgbevf ixgbe

# Don't set both of AF_XDP_TYPES and AF_PACKET_TYPES, use just the one that
# implies the desired default or none of them (resulting in af_packet)
# (AF_XDP_TYPES takes precedence and hence af_packet is the
# default default).

# Optional list of PCI device drivers that should use DPDK's built-in type
# specific driver instead of af_packet of af_xdp.
# It is not recommended to remove any drivers from the default value
# (especially the first three that don't have a corresponding
# netdev/link in Linux). The main purpose would be to be able to add
# rebranded Mellanox drivers (if they are rebranded on that level),
# but can also be used to force using af_packet or af_xdp on mellanox
# devices.
#- name: NATIVE_DRIVERS
#  value: vfio-pci igb_uio uio_pci_generic mlx4_core mlx5_core

# Optional AF-XDP busy budget (uses 0/disabled per default).
#- name: AF_XDP_BUSY_BUDGET
#  value: '0'

3.4. Memory

This section details the configuration of memory resources for the Firewall within a Kubernetes container, focusing on the distinct requirements of the control plane and data plane.

3.4.1. Control Plane

The memory allocation for the control plane is defined using Kubernetes pod specifications. It involves setting both the memory request and limit to the same value, a requirement for the pod to achieve Guaranteed Quality of Service (QoS).

apiVersion: v1
	kind: Pod
	spec:
	  containers:
	  -name: netshield
		resources:
		  requests:
			memory: 900Mi
		  limits:
			memory: 900Mi

For the control plane's memory allocation, it is important to specify both the memory request and limit similar in the Kubernetes pod configuration. This alignment is necessary for the pod to meet the criteria for Guaranteed Quality of Service (QoS), ensuring reliable resource availability for the control plane operations.

3.4.2. Data Plane

The data plane configuration utilizes hugepages, which are larger memory pages used to improve handling of network traffic. The setup involves specifying hugepages in the resources section, based on the expected memory requirements of the data plane.

apiVersion: v1
kind: Pod
 containers:
 - name: netshield
  env:
  - name: HUGEPAGES_2M
   valueFrom:
    resourceFieldRef:
     containerName: netshield
     resource: limits.hugepages-2Mi
     divisor: 1Mi
  - name: HUGEPAGES_1G
   valueFrom:
    resourceFieldRef:
     containerName: netshield
     resource: limits.hugepages-1Gi
     divisor: 1Mi

#- name: DP_MEMORY_MB
#  value: '300'
	 
  resources:
   requests:
    hugepages-2Mi: 800Mi
    hugepages-1Gi: 1Gi
   limits:
    hugepages-2Mi: 800Mi
    hugepages-1Gi: 1Gi
	 
  volumeMounts:

#- mountPath: /hugepages
#  name: hugepages

  - mountPath: /hugepages_2MB
   name: hugepages-2mb
  - mountPath: /hugepages_1GB
    name: hugepages-1gb
volumes:

#- name: hugepages
#  emptyDir:
#    medium: HugePages

 - name: hugepages-2mb
  emptyDir:
   medium: HugePages-2Mi
 - name: hugepages-1gb
  emptyDir:
   medium: HugePages-1Gi

Request hugepages totaling to the amount of memory that should be available to dataplane. Both request and limit needs to be set to the same value for the pod to qualify for guaranteed QoS. If using pages of both sizes only less than 1GiB of memory from 2MiB hugepages are supported. If using hugepages of just one size then the single "hugepages" mount/volume can be used. If using hugepages of both sizes the both the size-specific mounts/voluments (hugepages-2mb and hugepages-1gb) must to be used.

If downward API for hugepages is enabled in the cluster, then add both HUGEPAGES_2M and HUGEPAGES_1G to the environment, just make sure that the "containerName" matches the container's name. If they are added without proper downward API support then an error like the below example will be triggered:

.valueFrom.resourceFieldRef.resource:
Unsupported value: "limits.hugepages-2Mi" (or "..-1Gi")

The preferred solution to this is to enable the downward API. Workaround is to remove the HUGEPAGES_* environment variables and manually set environment variable DP_MEMORY_MB and keep it up to date/in-sync with the requests/limits.

3.5. CPU

This is an example demonstrating how to allocate CPU resources to the container.

apiVersion: v1
kind: Pod
spec:
 containers:
 - name: netshield
  env:

# Optional CPU list specification for control plane and data
# plane. Default is to detect CPUs via the cgroup cpuset
# controller. These needs to be used if dedicated CPU resources
# (pod in guaranteed QoS class and static CPU manager policy)
# are not used.

#- name: CP_CPU_LIST
#  value: '0' # rangelist format supported, for instance: '4,6,9-12'
#- name: DP_CPU_LIST
#  value: '1' # rangelist format supported, for instance: '4,6,9-12'
	 
# CPU_REQ and CPU_LIMIT will expose the amount of CPU resources
# requested to the container. No user configuration required other than
# to make sure that the "containerName" matches the container's name.

   - name: CPU_REQ
    valueFrom:
     resourceFieldRef:
      containerName: netshield
      resource: requests.cpu
      divisor: 1m
   - name: CPU_LIMIT
    valueFrom:
     resourceFieldRef:
      containerName: netshield
      resource: limits.cpu
      divisor: 1m
   resources:
    requests:
     cpu: '10'
    limits:
     cpu: '10'

If the cluster is properly configured to support pod in the Guaranteed QoS class. Then what is needed is to add the CPU_REQ and CPU_LIMIT environment variables, making sure that the "containerName" matches the containers name and request the required amount of CPU in the resources section.

Both request and limit needs to be set to the same value for the pod to qualify for "Guaranteed" QoS. If for some reason those requirements can not be met then the environment variables CP_CPU_LIST and DP_CPU_LIST can be used as a workaround. The system will expect those CPUs to be dedicated to this pod.

There are some further tweaks possible, see the full specification examples for documentation of those.

3.6. Additional Networks (Multus)

This section focuses on the parts of the example that are related to attaching additional network interfaces to the pod, besides the default pod network (eth0). In this example Multus is used to attach two additional network interfaces:

  • First interface using the "lan-network" network resource attachment definition, that will be called "lan", using one instance of the "example.com/lan_device" resource (the actual network device, for instance, an SR-IOV virtual function device).

  • Second interface using the "wan-network" network resource attachment definition, that will be called "wan", using one instance of the "example.com/wan_device" resource.

apiVersion: v1
kind: Pod
metadata:
 name: netshield
 annotations:
  k8s.v1.cni.cncf.io/networks: lan-network@lan, wan-network@wan
spec:
 containers:
 - name: netshield
 env:
   - name: NETS
     valueFrom:
      fieldRef:
       fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/networks']
	 
 resources:
  requests:
   example.com/lan_device: '1'
   example.com/wan_device: '1'
  limits:
   example.com/lan_device: '1'
   example.com/wan_device: '1'
  
 volumeMounts:
  - mountPath: /etc/podinfo
    name: podinfo
volumes:
 - name: podinfo
  downwardAPI:
   items:
    - path: "network-status"
     fieldRef:
      fieldPath: metadata.annotations['k8s.v1.cni.cncf.io/network-status']

The devices allocated under resources are the ones that are needed by the extra networks. The "resource prefix" (here "example.com") and the "resource name" (here: "lan_device" and "wan_device") must match the configuration of the device plugin used, for instance, the "SR-IOV Network Device Plugin for Kubernetes" which can be found here:

https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin

The examples archive contains examples of both a device configuration map for the device plugin (sriovdp-configmap.yaml) and some example network attachment definitions (networks.yaml). These examples show where these names appear in the various objects. The resource allocations in the pod specification can be injected automatically by for instance the "Network Resources Injector", which can be found here:

https://github.com/k8snetworkplumbingwg/network-resources-injector

The injection would be based on the "networks" annotation in the pod specification and the "resourceName" annotation in the network attachment definition.

There are some additional tweaks that can be applied, also affecting the default pod network. See the full pod specification examples for documentation of those.

3.7. Persistent Storage

The firewall pod needs persistent storage to function properly. This storage will be used for configuration, diagnostic console logs, to store crashdumps and other persistent states. The example here is the simplest possible where storage provided by a hostPath volume. This only works reliably with a single worker node cluster or if the pod is pinned to a specific worker node by some other means, so, basically only for initial tests.

For production use a more advanced storage solution is needed. There are more examples distributed with the software using local storage or NFS to provide storage for the firewall, see netshield-ls-statefulset.yaml and netshield-nfs-statefulset.yaml, and associated files. From the firewall's perspective the important part is that storage exists and is mounted at this mountPath:

/mnt/storage.

[Note] Note: The .yaml files

The .yaml files referred above are provided in a separate file included in every release. The file name is typically named :

clavister-cos-stream-4.00.00.34-cnf-x64-generic-deploy.tar.gz

apiVersion: v1
kind: Pod
spec:
 containers:
  - name: netshield
    volumeMounts:

# The system expects persistent storage to be
# mounted/available at /mnt/storage.

  - mountPath: /mnt/storage
    name: storage
 volumes:
  - name: storage
    hostPath:
     path: /opt/netshield/storage/pod1
     type: Directory

3.8. Monitoring Firewall Health

Liveness probes allow Kubernetes to continuously monitor the health of the firewall to detect if the container cannot forward traffic reliably and should be restarted. Both TCP and HTTP probes are supported.

It is recommended to configure startup probes in addition to liveness probes to make sure that the system has sufficient time to start up. Below is an example of this using a named port, in this case 8080. The system waits for 10 seconds before checking if the container has started each second. Once the container is up and running, Kubernetes will move on to monitor the health of the system every 10 seconds. If the container has not responded within a total of 10 + 30 seconds on startup or to three consecutive liveness probes, the firewall pod considered to be malfunctioning and will be restarted. Kubernetes will then wait for an additional 60 seconds to allow the system to shutdown gracefully.

[Note] Note: Grace period

If the grace period is set too low, the system may not have enough time to save states to permanent storage, some of which can be used for troubleshooting.

ports:
- name: health-port
	containerPort: 8080

startupProbe:
  httpGet:
	port: health-port
	path: /health
  initialDelaySeconds: 10
  periodSeconds: 1
  timeoutSeconds: 1
  failureThreshold: 30
	 
livenessProbe:
  httpGet:
	port: health-port
	path: /health
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
  terminationGracePeriodSeconds: 60

TCP probes are setup similarly with tcpSocket instead of httpGet and `path` omitted. The HEALTH_PORT_TCP_HTTP environment variable is used to specify which port the firewall should listen on for probes.

- name: HEALTH_PORT_TCP_HTTP
	value: '8880'
		

Chapter 4: Firewall Configuration

When the pod is started for the first time/without any firewall configuration, a default configuration will be generated. This configuration will contain the basic network parameters that could be identified from the environment during start-up. This include IPs and networks/routes on the Ethernet interfaces exposed to the pod. It also includes DNS server(s) and a default SSH remote management object to allow access to the CLI from the pod network (eth0) with default access credentials (admin:admin). Networks/routes are added as static objects and won't automatically change if the pod is restarted. IP addresses on the interfaces are created as dynamic address objects that will change at every start-up.

4.1. Interface IP Addresses

For each Ethernet interface a number of IP address objects will be created in the default configuration. These IP address objects will be assigned values run-time. The names of the address objects will be based on the interface name (in the pod specification/environment). This name will be used as a prefix and the following address objects will be created:

<prefix>_cni_ip
  • Contains all IP addresses assigned to the interface if the firewall is not part of an HA cluster, otherwise it contains no IP addresses. Will be set as the IPAddress property on the EthernetInterface.

<prefix>_cni_priv0
  • Contains the first IPv4 address and the first IPv6 address assigned to the interface if the firewall is the master node of an HA cluster, otherwise no IP addresses. Will be set as the PrivateIP.0 property on the EthernetInterface.

<prefix>_cni_priv1
  • Contains the first IPv4 address and the first IPv6 address assigned to the interface if the firewall is the slave node of an HA cluster, otherwise no IP addresses. Will be set as the PrivateIP.1 property on the EthernetInterface.

<prefix>_cni_br
  • The broadcast address, will be set as the IP4Broadcast property on the EthernetInterface.

<prefix>_cni_ip4_<#>
  • eth0_cni_ip4_1, eth0_cni_ip4_2 etc. for as many IPv4 addresses that are assigned to the interface. These are not used anywhere in the default configuration but provides flexibility to the admin if multiple addresses are assigned with different purposes.

<prefix>_cni_ip6_<#>
  • Same as above but for IPv6 addresses.

There is no point in assigning IPs to these address objects, since they will always use the dynamically assigned value, so for instance if a different address than what eth0_cni_ip contains should be used as IP address on eth0 then change the IPAddress property on the EthernetInterface object instead of trying to change the value of eth0_cni_ip.

4.2. DNS Servers

For each DNS server found during the first startup, an IP address object is provided named according to the following pattern: system_dnsserver<#>_ip, (system_dnsserver1_ip, etc.). Additionally a DNSServer object is added for each DNS server under the DNS object, using the corresponding IP address object as IPAddress.

The IP address objects are updated at every restart of the pod, but not created/removed if the number of servers are changed, that has to be done manually. DNSServer objects are not updated either, other than implicitly through the IP address objects being updated.

System:/> show Address IPAddress system_dnsserver1_ip

          Property  Value                 Remarks
------------------  --------------------  --------------------
             Name:  system_dnsserver1_ip
          Address:  0.0.0.0
    ActiveAddress:  10.152.183.10         Dynamically assigned
FQDNValidAfterTTL:  <empty>
         Comments:  <empty>

System:/>  cc DNS
System:/DNS>  show

DNSServer

  # IP address           Routing table
  - -------------------- -------------
  1 system_dnsserver1_ip main
    			

4.3. Kubernetes Events

The system will per default send some of the system events to kubernetes as kubernetes events. The events that are reported this way is a subset of the events that can be seen in the dconsole log inside the system. This includes for instance events about start-up, reconfigurations, and shutdown. There is a setting that controls if these events are exported as kuberenetes events or not called ReportKubeEvents located under MiscSettings.

It is a boolean yes/no setting.

Chapter 5: Firewall Management

This chapter contain brief descriptions on how to manage the firewall within a Kubernetes environment. For a more detailed description on how to configure management access in cOS Stream please see the Setting Up Management Accesssection in the Clavister cOS Stream Virtual Series Getting Started Guide.

5.1. SSH

As mentioned in Chapter 4, Firewall Configuration there is an SSH remote management object created per default that allows SSH access to the firewall's CLI from the pod network. It is recommended to add a Kubernetes service in order to keep track of the management IP addresses of the pod.

5.2. Direct access to the CLI

Information about direct access to the Command Line Interface (CLI) for firewall management.

The CLI can also be accessed by directly executing the CLI command with, for instance, kubectl:

kubectl exec -it netshield -- cli

5.3. InCenter

The firewall can also be managed from Clavister's centralized management product InCenter by adding it like any other firewall. A kubernetes service would be a good way to keep track of the firewall's management IP, unless it is assigned a static IP.

Chapter 6: Considerations

This chapter describes information that may need to be considered before deploying Clavister cOS Stream as a container in Kubernetes.

6.1. Compatible Versions

Below is a list of software versions that has been verified by Clavister to work with this guide. Newer versions released after this guide was written will most likely work as well but they are used at your own risk as full functionality have not been verified.

  • Linux Kernel: 4.15

  • Kubernetes: 1.26.1

  • Multus: Version 3.7

  • SR-IOV Network Device Plugin: Version 3.3.2

  • Drivers

    • i40e

    • ice

    • mlx5

6.2. Verified NICs

The following Network Interface Cards (NICs) have been verified to work during quality assurance of this release.

  • Intel® Ethernet Network Adapter X710

  • Intel® Ethernet Network Adapter E810

  • NVIDIA® ConnectX®-5

  • NVIDIA® ConnectX®-6

6.3. Expose Ethernet NICs

For best performance and in some cases functionality, it is best to expose base Ethernet interfaces to the firewall. If you want to use bonding/link aggregation, it is better to expose the member interfaces to the firewall, as-is, and configure link aggregation interface in the firewall configuration than to expose a bond-device to the pod. Same with VLAN, it is better to expose the Ethernet interface than some virtual VLAN interface, however that might be harder since the Ethernet interface might be needed elsewhere as well.

For VLAN and SR-IOV it is possible to set a single VLAN directly on the SR-IOV VF, that is, set the vlan parameter in the configuration for the SR-IOV CNI plugin, and then just treat that as any Ethernet interface as far as the firewall is concerned having the NIC automatically adding and stripping the VLAN tag. The firewall will then be restricted to that VLAN only.

6.4. VLAN over virtual Ethernet

VLAN over virtual Ethernet interfaces, like the virtual Ethernet pair likely used to provide the interface for the default pod network "/eth0", requires VLAN offload to be disabled on the "other end" of the Ethernet connection. This to ensure that the firewall receives the packets with the VLAN header in place in the raw packet data. Otherwise the VLAN information might not be picked up by the firewall. Disabling VLAN offload might require some custom scripts on the host.

6.5. Interface naming

When it comes to interface naming, vEth0 should be considered a reserved name.

For full support/identical naming inside the firewall, interface names should match the below regular expression. The current limit is 26 characters.

[a-zA-Z][a-zA-Z0-9_-]{,14}

A few other characters might work, but the interface will have a name without those characters inside the firewall (in the firewall's configuration), possibly with other modifications as well to make them unique in this context.

Whitespace or any of the below characters in the interface names are known to cause problems and are therefore unsupported. Other characters, except those in the first paragraph, are used at your own risk.

/ : " ' , = [ ; %