2.6. Diagnostic Tools

Home Prev	cOS Core 14.00.17 Administration Guide	Next

2.6.1. Overview

When troubleshooting network problems, cOS Core provides an assortment of tools to help with problem resolution.

The section describes some of the most important troubleshooting tools available to the administrator. Most of these are used as CLI commands.

2.6.2. The ping Command

The combination of the ICMP echo request and echo reply messages are known as ping. They provide a simple diagnostic tool to find out if a host is reachable. In the cOS Core CLI, the ping command provides this feature.

In its simplest form, the CLI command to ping a remote IP address takes the form:

Device:/> ping <ipaddress>

For example, to ping the IPv4 address 10.6.58.10:

Device:/> ping 10.6.58.10
		
Sending 1 4-byte ICMP ping to 10.6.58.10 from 192.168.3.20
using PBR table "main"
ICMP Reply from 192.168.1.1  seq=0  time=<10 ms  TTL=128

Ping Results:  Sent: 1, Received:1, Avg RTT: 10.0 ms

Here, the RTT is the round trip time for the ICMP echo request and reply messages. The TTL value is the Time To Live which is a hop counter. The initial TTL value is set by the sender and decremented by each router passed. When it reaches zero, the packet is discarded preventing packets from circulating forever.

This basic form of the ping command can also be used via the cOS Core Web Interface by going to: Status > Tools > Ping.

Choosing the Routing Table

By default, the outgoing source interface for ICMP ping is chosen by performing a lookup of the destination IP address in the main routing table. This can be overridden with the -pbr option in order to specify which routing table to use for the lookup. For example, if the routing table my_routing_table is to be used, the command would be:

Device:/> ping 10.6.58.10 -pbr=my_routing_table -verbose

IP Rule Set Entries for Outgoing Ping Messages

When the ICMP ping message is outgoing from cOS Core, it does not require that there is an IP rule set entry that allows the traffic since cOS Core is always trusted. In the cOS Core event message logs, an outgoing ping will generate a conn_open and conn_close log event using the Stock_Allow_All_Rule. The source interface will always be the core interface (meaning cOS Core itself).

IP Rule Set Entries for Incoming Ping Messages

Any ping messages that are incoming require an allowing IP rule set entry in order for cOS Core to respond and these entries should have their associated Service property set to be the predefined service ping-inbound. An example IP rule set entry for ping messages arriving on the wan interface would be the following:

Action	Source Interface	Source Network	Destination Interface	Destination Network	Service
Allow	wan	all-nets	core	wan_ip	ping-inbound

Using the -verbose Option

The -verbose option is recommended to get the maximum amount of information from ping usage. For example:

Device:/> ping 10.6.58.10 -verbose
		
Sending 1 4-byte ICMP ping to 10.6.58.10 from 192.168.3.20
 ... using route "192.168.3.20 via lan, gw (Iface IP)" in PBR table "main"
ICMP Reply from 192.168.3.20  seq=0  time=<10 ms  TTL=255

Ping Results:  Sent: 1, Received:1, Avg RTT: 10.0 ms

Here, the IPv4 address 192.168.3.20 is the IP address of the Ethernet interface on the firewall from which the ping is sent. The output shows the route lookup that was performed to find the correct interface.

Testing TCP and UDP Connectivity

ICMP messages are neither UDP or TCP but are considered to be their own third category of IP traffic. However, the cOS Core ping command has the ability to send a message to test either TCP or UDP connectivity.

To send as TCP, the -port option is used along with the -tcp option. Successful connectivity then results in a 3-way TCP handshake taking place with the destination host. For example:

Device:/> ping 10.6.58.10 -port=80 -tcp -verbose
		
Sending 0-byte TCP ping to 10.6.58.10:80 from 192.168.3.20:41207
using PBR table "main"
... using route "10.6.10.0/24 via aux, no gw" in PBR table "main"
TCP Reply from 10.6.58.10:80 to 192.168.3.20:41207 seq=0  SYN+ACK
time=>10 ms  TTL=128
TCP Reply from 10.6.58.10:80 to 192.168.3.20:41207 seq=0  ACK
time=>10 ms  TTL=128

TCP Ping Results:  Sent: 1, RST/ACKs Received:1, Loss: 0%, Avg RTT: 10.0 ms

This allows the remote hosts responsiveness to an incoming TCP connection to be established.

For testing UDP connectivity, use the -udp option with the -port option. The UDP message size can also be specified by using the -count option to specify the number of packets and the -length option to specify the packet length. For example:

Device:/> ping 10.6.58.10 -udp -port=2222 -verbose -count=1 -length=30

Sending 30-byte UDP ping to 10.6.58.10:2222 from 192.168.3.20:22307
using PBR table "main"
... using route "0.0.0.0/0 via ext, gw 192.168.3.1" in PBR table "main"
UDP Reply from 10.6.58.10:2222 to :192.168.3.20:22307 seq=0 time=50 TTL=58

Ping Results:  Sent: 1, Received:1, Loss: 0%, Avg RTT: 50.0 ms

If the size is not specified then cOS Core sends a single 4 byte UDP packet.

The "Could not open outbound connection" Message

This response may be encountered after the ping command is entered. There are a number of reasons for this message and they are listed in a Clavister Knowledge Base article at the following link:

https://kb.clavister.com/324735730

Incoming Packet Simulation Using the -srcif and -srcip Options

Instead of testing the responsiveness of a remote host, the cOS Core ping command can be used to simulate incoming traffic and test the configured IP rule set and routing table. This is done by using the -srcif option to specify the interface that receives the message and the -srcip option to specify the sending IP address. For example:

Device:/> ping 10.6.58.10 -srcif=wan -srcip=192.168.14.12 -verbose

This command will construct an ICMP packet with destination IP 10.6.58.10 and cOS Core will behave as though the packet has arrived on the wan interface and had come from the IP address 192.168.14.12 . Note when the -verbose option is included, the output from the command will show the configuration rules that are triggered by the simulation.

The destination IP address specified in the ping command. could be an actual external host in which case the packet will be forwarded to it through the firewall, providing the configuration allows this.

In the example output below, it can be seen how a ping simulation can show pipe rules that are triggered when an ICMP message is received on the lan interface.

Device:/> ping 10.6.58.10 -srcif=lan -srcip=192.168.3.1 -verbose
		
Rule and routing information for ping:
PBR selected by rule "iface_member_main" - PBR table "main"
     allowed by rule "nat_all_wan"
		   piped by rule "out_pipe" - Fwd Chain: out
			 piped by rule "out_pipe" - Ret Chain: in

Sending 1 4-byte ICMP ping to 10.6.58.10 from 192.168.3.20
sent via route "0.0.0.0/0 via lan, gw 192.168.3.1" in PBR table "main"
ICMP Reply from 10.6.58.10  seq=0  time= 10 ms  TTL=247

Ping Results:  Sent: 1, Received:1, Avg RTT: 10.0 ms

The above output shows how a Pipe Rule object called out_pipe is triggered.

If there is no route that matches the combination of source IP and receiving interface (the -srcif parameter), the packet will be dropped by the default access rule. For example:

Device:/> ping 10.6.58.10 -srcif=wan -srcip=192.168.14.12 -verbose
		
Rule and routing information for ping:
PBR selected by rule "iface_member_main" - PBR table "main"
     DROPPED by rule "Default_Access_Rule"

For the simulated traffic not to be dropped, there must not only be a route that matches the combination of source IP address and receiving interface but also an IP rule set entry that allows the traffic arriving on that interface. If the administrator simulates ICMP traffic coming from the Internet that arrives on the wan interface and destined for some host on the protected network lan_net, the allowing IP rule set entry might be the following:

Action	Source Interface	Source Network	Destination Interface	Destination Network	Service
Allow/NAT	lan	lan_net	wan_net	all-nets	ping-inbound

If there is no IP rule set entry that permits the packet, it will also be dropped. For example:

Device:/> ping 10.6.58.10 -srcif=wan -srcip=192.168.14.12 -verbose
		
Rule and routing information for ping:
PBR selected by rule "iface_member_main" - PBR table "main"
     DROPPED by rule "Default_Rule"

	Note: The -pbr option cannot be used with the -srcif option
	The -pbr option cannot be used with the -srcif option. The routing table used is decided by the cOS Core configuration.

A further description of ping packet simulation, specifically with examples for troubleshooting problems with IP rule sets and routing tables, can be found in a Clavister Knowledge Base article at the following link:

https://kb.clavister.com/324735909

Ping with IPv6

So far, the use of the ping command has been discussed only for IPv4 addresses. IPv6 addresses can also be used with the ping command. For example:

Device:/> ping 2001:DB8::2

The incoming packet simulation feature in the ping command can also be used with IPv6 source addresses.

Using IPv6 with cOS Core is discussed further in Section 3.2, IPv6 Support.

FQDN Resolution

When issuing a ping request from cOS Core, it is possible to specify the destination as a fully qualified domain name (FQDN). This is then resolved by cOS Core to a numerical IP address by using an external DNS server. For example:

Device:/> ping server.example.com

	Note: At least one DNS server must be configured
	For FQDN resolution to function, at least one DNS server must be configured in cOS Core. Configuring DNS servers is described in Section 3.10, DNS.

DNS servers might return either an IPv4 address or an IPv6 address or both. These three possibilities are treated as follows:

If only an IPv4 address is returned then that will be used by the ping command for the ICMP message.
If only an IPv6 address is returned then that will be used by the ping command for the ICMP message.
If both an IPv4 and an IPv6 address is available, cOS Core will use the IPv4 address by default. However, cOS Core can be forced to use the IPv6 address with the -6 command option. For example:
```
Device:/> ping www.example.com -6
```

2.6.3. The stats Command

If a serious cOS Core problem is suspected then the first step should be to use the CLI command:

Device:/> stats

The stats command will provide a snapshot of the system and indicate the date and time of the last system shutdown and can also indicate if there has been a serious error in cOS Core operation. It should be remembered however that the buffer which stats uses is cleared by certain operations such a reconfigure and the output will not therefore show what occurred prior to buffer clearance.

Below is a typical example of output from the command:

Device:/> stats
Uptime             : 7 days, 02:12:38
Last shutdown      : 2014-06-17 16:05:00: Activating configuration changes
CPU Load           : 4%
Connections        : 23 out of 512000
Fragments          : 0 out of 16384 (0 lingering)
Buffers allocated  : 43280
Buffers memory     : 43280 x 2636 = 111412 KB
Fragbufs allocated : 32
Fragbufs memory    : 32 x 10040 = 313 KB
Out-of-buffers     : 0

At the end of the Last shutdown line is the reason for the shutdown.

stats Command Output with Poll Offloading

When cOS Core runs on certain hardware platforms or in the KVM virtual environment, it can make use of a technique known as poll offloading to increase performance. However, the cOS Core license must explicitly allow the feature for it to be used.

With poll offloading, two CPU cores can be used simultaneously by cOS Core. One CPU core will be running most of cOS Core's functions and the second will be running that part of cOS Core that handles Ethernet interface polling.

When the stats command is used with poll offloading active, the CPU Load line in the command output will show two percentages instead of one. The first percentage is the load for the CPU core that is running most of cOS Core's functions. The second percentage shows the load for the CPU core that is running the interface polling subsystem. An example of this output is shown below:

CPU Load           : 12%, 1%

Poll offloading is usually turned on automatically by cOS Core if the hardware platform supports it and the administrator does not normally need to enable it.

The settings for controlling the poll offloading feature are described in Section 13.10, Miscellaneous Settings.

2.6.4. The connections Command

By using the connections command, the administrator can get a snapshot of all the connections currently set up in the cOS Core state engine. The command can be abbreviated to conn and some example output is shown below:

Device:/> conn

State    Proto   Source                      Destination            Tmout
-------- ------- --------------------------- ----------------------- ------
TCP_OPEN TCP     If1:10.4.4.24:54047         If2:192.168.9.3:338    261772
UDP      UDP     If2:192.168.109.11:4500     If3:10.152.0.22:450    130
PING     ICMP    vlan1:192.168.1.1:512       If3:90.152.1.1:512     8
FIN_RCVD TCP     If1:10.4.4.121:55679        core:10.4.0.31:444     69
TCP_OPEN TCP     If2:192.168.96.77:35217     If3:10.93.2.49:463     70855
UDP      UDP     vlan1:192.168.100.163:560   vpn-A:10.45.1.2:161    9
UDP      UDP     vlan1:192.168.100.163:582   vpn-B:10.25.1.2:161    76

Each line in the command's output corresponds to a single connection. The fields shown are:

State

This indicates the state of the connection and is only really relevant to TCP connections where different states apply. Some of the possible values are:
1. UDP - A UDP pseudo-connection.
2. PING - AN ICMP ping connection.
3. TCP_OPEN - A TCP connection is opening.
4. SYN_RCVD - A TCP connection has received a SYN packet and is open.
5. FIN_RCVD - A TCP connection has been closed. Connections wait, by default, for 80 seconds before all data is cleaned up by cOS Core so that the connection could be reopened. The 80 second value is controlled by the cOS Core setting TCP FIN Idle Lifetime. The ability to reopen a connection is controlled by the cOS Core setting Allow TCP Reopen which is disabled by default.
6. RAW IP - Another protocol which is identified in the Protocol column.
7. ZOMBIE - This is a transient state that indicates being queued for deletion from the connection table.
  
  This state can appear because there are a large number of connections that are queued for deletion from the table but cOS Core has not yet had time to remove them all. Note that while in the zombie state, a connection will not forward any traffic.
  
  Zombie connections are also discussed in an article in the Clavister Knowledge Base at the following link:
  
  https://kb.clavister.com/324735766
Proto

The protocol used for the connection. This can be the same as the State column is some cases. Some of the possible values are:
1. UDP - A UDP pseudo-connection.
2. ICMP - An ICMP ping connection.
3. TCP - A TCP connection.
4. ESP - Used for IPsec VPN tunnels.
5. A number or name indicating the protocol being used. The State column will have the value RAW IP.
Source

This consists of:
1. The source interface. This could be the name of any type of cOS Core interface object such as a VLAN or IPsec tunnel. It can also be Core which indicates cOS Core itself is the connection's source.
2. The source IP address for the connection.
3. The source port number for the connection.
Destination

This consists of:
1. The destination interface. This could be the name of any type of cOS Core interface object such as a VLAN or IPsec tunnel. It can also be Core which indicates cOS Core itself is the connection's destination.
2. The destination IP address for the connection.
3. The destination port number for the connection.
Tmout

The number of seconds until the connection times out because no traffic is detected. As soon as any traffic is detected being sent from either end of the connection, this value is reset to the default timeout. The defaults are controlled by the followed cOS Core settings:
1. TCP Idle Lifetime - For TCP connections. The default value is 262144 seconds.
2. UDP Idle Lifetime - For UDP connections. The default value is 130 seconds.
3. Ping Idle Lifetime - For ICMP Ping connections. The default value is 8 seconds.
4. Other Protocols Idle Lifetime - All other protocols. The default value is 130 seconds.

Filtering Connections

The connections command provides the ability to specify filters for which connections are display. To display only connections with the protocol TCP the command would be:

Device:/> conn -protocol=TCP

To see only connections with the source interface If3:

Device:/> conn -srciface=If3

Closing Connections

The connections command gives the administrator the ability to close all or selected connections. To close all, the command would be:

Device:/> conn -close -all

To close all connections with the source interface If3:

Device:/> conn -close -srciface=If3

The -verbose Option

When the -verbose option is used, the connections command adds another line of output for each connection that is prefixed with ...term:. This line shows the changes, if any, made by cOS Core in the interface or IP or port number as the connection traverses the firewall. For example, consider this output showing a single connection:

Device:/> conn -verbose

State    Proto   Source                      Destination            Tmout
-------- ------- --------------------------- ----------------------- ------
TCP_OPEN TCP     If1:10.4.0.16:60848         If3:192.168.96.82:80   262119
        ...term: If1:192.168.96.1:39097      If3:192.168.96.82:80   262119

Here, the original connection is subject to a NATing IP rule set entry which transforms If1:10.4.0.16:60848 into If1:192.168.96.1:39097. The destination remains the same but the source IP and port has been changed by cOS Core. For this example, a private IP address has been used for illustration but 192.168.96.1 would typically be a public IP address.

A complete list of all options for the connections command can be found in the separate CLI Reference Guide.

2.6.5. The dconsole Command

The next step is to use the CLI command:

Device:/> dconsole

This can be abbreviated to:

Device:/> dcon

The dconsole command provides a list of important events that have occurred during cOS Core operation and can help to establish the date, time and nature of events leading up to a serious problem occurring. The output might look similar like the following:

Showing diagnose entries since 2008-05-22:
2008-06-21 11:54:58                  Start (14.00.17-0:131)
2008-06-21 11:56:16                  Stop (RECONFIGURE)
2008-06-21 11:56:21                  Start (14.00.17-0:131)
2008-06-21 11:57:29                  Stop (RECONFIGURE)
2008-06-21 11:59:31                  Start (14.00.17-0:131)
2008-06-21 11:59:49                  Stop (NORMAL)
			"
			"

Output from dconsole can include a dump of the system memory in the case of serious runtime errors. Such a dump will look similar to the following:

'
'
Reason: Exception 'DataAbort' occurred at address 0x7aaea34
Generation date/time: 2008-07-04 14:23:56 List of loaded PE-modules:
fwloader(1.07.04): BA:0x00100000, EP:0x00101028, SS:0x0, IS:0xe7000
fwcore(814.00.17-2336): BA:0x07761038, EP:0x0007c630 Register dump:
----------------------------------------------------
r0 : 0xe1a0003c, r1 : 0x07c685dc, r2 : 0x00000004, r3 : 0x50013700,
r4 : 0x06cb2d04, r5 : 0x0753a740, r6 : 0x050ce1f8, r7 : 0x00000000,
r8 : 0x0753a79c, r9 : 0x050ce1f8, r10: 0x00000000, r11: 0x0775ff34,
r12: 0x00000004, sp : 0x0775fcec, lr : 0x079de7e4 Stack dump:
5da89306 c33613f4 c330cfc5 04411507 45515a49 86619f8b c0db0a81
4e395861 cb25b796 e3108934 932766c5 4dcff9e9 711c3463 b9cd5d1e
52149961 9324dea3 d340dc25 15458610 63582ded 689a0c54 dfb43131
02c7d971 a0ebb72c bfaae832 db216923 08ba693b 95e4de97 98d121a2
'
'

Although dconsole output may be difficult to interpret by the administrator, it can be emailed to Clavister support representatives for further investigation.

The dconsole command supersedes the crashdump command found in earlier versions of cOS Core.

Dconsole can also be run in the Web Interface by going to:
Status > Maintenance > Diagnostic Console
Selecting this option runs dconsole automatically and the output is immediately displayed in a text box. The contents of the text box can be copied and pasted into an email for review by support personnel. Pressing the Refresh button will run dconsole again and display the new output in the text box.

2.6.6. The pcapdump Command

A valuable diagnostic tool is the ability to examine the packets that enter and leave the interfaces of a Clavister firewall. For this purpose, cOS Core provides the CLI command pcapdump which not only allows the examination of packet streams entering and leaving interfaces but also allows the filtering of these streams according to specified criteria. Only the pcapdump CLI command usage is described in this section but its functions are also duplicated in the Web Interface.

The packets that are filtered out by pcapdump can then be saved in a file of type .cap which is the defacto libpcap library file format standard for packet capture.

The complete syntax of the pcapdump CLI command is described in the CLI Reference Guide. There is also an additional description of usage in the Clavister Knowledge Base at the following link:

https://kb.clavister.com/324736324

Another article in the Knowledge Base discusses stopping pcapdump output:

https://kb.clavister.com/324736057

A Simple Example

An example of pcapdump usage is the following sequence:

Device:/> pcapdump -start lan -size=1024
Device:/> pcapdump -stop lan
Device:/> pcapdump -show
Device:/> pcapdump -write lan -filename=cap_lan.cap
Device:/> pcapdump -cleanup

Going through this line by line we have:

1. Recording is started for the lan interface using a buffer size of 1024 Kbytes.

Device:/> pcapdump -start lan -size=1024

2. The recording is stopped for the lan interface.

Device:/> pcapdump -stop lan

3. The dump output is displayed on the console in a summarized form.

Device:/> pcapdump -show

4. The same information is written in its complete form to a file called cap_lan.cap.

Device:/> pcapdump -write lan -filename=cap_lan.cap

At this point, the file cap_lan.cap should be downloaded to the management computer for analysis.

5. A final cleanup is performed and all memory taken is released.

Device:/> pcapdump -cleanup

Re-using Capture Files

Since the only way to delete files from the firewall is through the local console, the recommendation is to always use the same filename when using the pcapdump -write option. Each new write operation will then overwrite the old file.

Running on Multiple Interfaces

It is possible to have multiple pcapdump executions being performed at the same time. The following points describe this feature:

All capture from all executions goes to the same memory buffer.

The command can be launched multiple times with different interfaces specified. In this case the packet flow for the different executions will be grouped together in different sections of the report.

If a clearer picture of packets flowing between interfaces is required in the output then it is best to issue one pcapdump command with the interfaces of interest specified.
If no interface is specified then the capture is done on all interfaces.
The -stop option without an interface specified will halt capture on all interfaces.
pcapdump prevents capture running more than once on the same interface by detecting command duplication.

Filter Expressions

Seeing all packets passing through a particular interface often provides an excess of information to be useful. To focus on particular types of traffic the pcapdump command has the option to add a filter expression which has one of the following forms:

-eth=<macaddr> - Filter on source or destination MAC address.
-ethsrc=<macaddr> - Filter on source MAC address.
-ethdest=<macaddr> - Filter on destination MAC address.
-ip=<ipaddr> - Filter source or destination IP address.
-ipsrc=<ipaddr> - Filter on source IP address.
-ipdest=<ipaddr> - Filter on destination IP address.
-port=<portnum> - Filter on source or destination port number.
-srcport=<portnum> - Filter on source port number.
-destport=<portnum> - Filter on destination port number.
-proto=<id> - Filter on protocol where id is the decimal protocol id.
-<protocolname> - Instead of using -proto=, the protocol name alone can be specified and can be one of -tcp, -udp or -icmp.

Examining Multiple Ports

It is possible to specify the port as a list of port numbers:

Device:/> pcapdump -start lan -size 1024 -port=67,68

It is also possible to specify a port range:

Device:/> pcapdump -start lan -size 1024 -port=8080-8088

Downloading the Output File

As shown in one of the examples above, the -write option of pcapdump can save buffered packet information to a file on the firewall.

These output files are placed into the cOS Core root directory and the filename is specified in the pcapdump command line, usually with a filetype of .cap. The name of output files must follow certain rules which are described below. Files can be downloaded to the local management computer using Secure Copy (SCP) (see Section 2.1.8, Using SCP). A list of all files in the cOS Core root directory can be viewed by issuing the ls CLI command.

The -cleanup option will erase the files so cleanup should only be done after file download is complete.

Output File Naming Restrictions

The name of the file used for pcapdump output must comply with the following rules:

Excluding the filename extension, the name may not exceed 8 characters in length.
The filename extension cannot exceed 3 characters in length.
The filename and extension can only contain the characters A-Z, 0-9, "-" and "_".

Combining Filters

It is possible to use several of these filter expressions together in order to further refine the packets that are of interest. For example we might want to examine the packets going to a particular destination port at a particular destination IP address.

Compatibility with Wireshark

The open source tool Wireshark (formerly called Ethereal) is an extremely useful analysis tool for examining logs of captured packets. The industry standard .pcap file format used by pcapdump with its -write option means that it is compatible with Wireshark.

For more complete information about this topic, see http://www.wireshark.org.

Using pcap through the Web Interface

All the functions described above are available in the Web Interface by going to Status > Tools > PCAP. It is also possible to directly download the .cap file to the management computer. If Wireshark has been installed and the file association with it set up, the file can be opened directly in the software.

Extensive Packet Capture Could Affect Throughput

Over a short period, packet capture will not have a noticeable effect on firewall throughput. However, the administrator should be aware that extensive packet capture over a longer period may impact overall traffic throughput because of the processing resources that are required. The potential impact is hard to quantify because of the large number of variables.

2.6.7. The traceroute Command

Overview

The cOS Core traceroute CLI command provides information about the routes that packets take as they traverse the routers in the external network and the round-trip transit time to and from these routers. A similar traceroute command is found on many other systems such as Microsoft Windows™.

Traceroute functions by sending packets with a time-to-live (TTL) value that starts at 1 and is progressively incremented for subsequent packets. A router will decrement the time-to-live as a packet traverses it. If the value becomes zero, the packet is dropped by the router and an ICMP time-exceeded message is sent back to the source which sends another packet with a time-to-live of 2 in order to reach the next router. The incrementing of the time-to-live continues until the intended destination is reached. The ICMP time-exceeded messages sent back by the routers between the source and destination provide the basis for the traceroute output.

By default, cOS Core sends its traceroute packets as ICMP ping messages. However, it is also possible to send messages using UDP or TCP. The basic form of the command for sending ICMP messages is the following:

Device:/> traceroute <destination>

The <destination> can be an IPv4 or IPv6 address, for example:

Device:/> traceroute 192.168.4.1

Or it can be a DNS resolvable Fully Qualified Domain Name (FQDN), for example:

Device:/> traceroute server.example.com

When using traceroute with an FQDN then at least one DNS server must have been configured in cOS Core to perform the resolution. Doing this is described in Section 3.10, DNS. To send a TCP message instead of ICMP, the command would be the following:

Device:/> traceroute -tcp server.example.com

To send a UDP message:

Device:/> traceroute -udp server.example.com

Note that the -tcp and -udp options must be placed before the destination in the command.

Traceroute Output

Below is some typical output from traceroute using the default settings with the destination specified as an FQDN:

Device:/> traceroute server.example.com
		
Tracing server.example.com [10.194.40.247], 30 hops max, 32 bytes of data
Hop#    RTT      RTT      RTT   Host
  1    10 ms    10 ms    10 ms  10.4.16.1
  2    10 ms    10 ms    10 ms  10.4.0.2
  3    10 ms     0 ms    10 ms  10.194.40.247
Trace complete.

Here, each line of output corresponds to an attempt by traceroute to reach the next router.

By default, traceroute tries 3 times for each router hop and the Round Trip Time for each attempt expressed in milliseconds is shown under a corresponding RTT heading.

There were two routers in the path to the target destination in the above output (hop number 1 and hop number 2). The final hop (number 3) is the destination which was DNS resolved as the IPv4 address 10.194.40.247.

The Route Can Change

Given the dynamic nature of a packet switch network, it is possible for consecutive packets sent by the traceroute command to pass through different sets of routers. It is difficult to know that this is occurring and it is not indicated in the traceroute command output.

It is possible that different routers could respond for a given hop value. The address displayed at the end of lines of traceroute output under the Host column is always the router that dealt with the last ICMP message sent for that hop.

Traceroute Options

The following are some of the important options for the traceroute command:

-6

When the destination is specified as an FQDN, cOS Core will only request an IPv4 address from the resolving DNS server and will use that as the destination address. This option must be used if an IPv6 address is to be requested from the DNS server and used as the destination address.
```
Device:/> traceroute server.example.com -6
```
Alternatively, the IPv6 address could be entered directly after the traceroute command.
-maxhops

This specifies the maximum value for the time-to-live parameter of the packets sent.
```
Device:/> traceroute server.example.com -maxhops=20
```
The default value is 30.

Maxhops is important because if cOS Core gets no response from a router within the set timeout (the default is 1 second) then it will continue to send ICMP ping messages with an increasing time-to-live value until the maxhops limit is reached.
-count

This specifies how many attempts are made for each hop.
```
Device:/> traceroute server.example.com -count=1
```
The default value is 3.
-size

This specifies how large the payload is that is sent. The payload is made up of random data.
```
Device:/> traceroute server.example.com -size=128
```
The default value is 32.
-nodelay

This specifies that each query is to be sent as fast as possible.
```
Device:/> traceroute server.example.com -nodelay
```
The default is that this is disabled.

The number of traceroute ICMP messages specified by the -count parameter are first sent continuously with no delay. Then there is a fixed delay of one second before the next set of messages are sent with an incremented time-to-live value. By specifying -nodelay, the one second delay is removed and the sets of messages are sent continuously with no delay between them. This continuous packet stream can simulate a denial-of service (DOS) attack.
-starthop

This specifies what time-to-live (TTL) value to start with and therefore at which hop to start.
```
Device:/> traceroute server.example.com -starthop=3
```
The default value is 1.
-stop

This terminates any traceroute that is in progress .
```
Device:/> traceroute -stop
```
-timeout

This is the amount of time cOS Core will wait for a response from a router or the destination before it increases the time-to-live and tries again.
```
Device:/> traceroute server.example.com -timeout=2000
```
Any timeout conditions are indicated in the traceroute output. An example of this is shown below:
```
Device:/> traceroute example.com
				
Tracing example.com [10.120.184.11], 30 hops max, 32 bytes of data
Hop#    RTT      RTT      RTT   Host
  1     0 ms     0 ms    10 ms  10.4.16.1
  2    10 ms    10 ms    10 ms  10.4.0.2
  3    10 ms    10 ms    10 ms  10.131.48.2
  4     *        *        *     Request timed out
```
A timeout could occur because any of the following:
1. A router or the destination may not be set up to respond to ICMP ping messages.
2. The destination host may be offline.

Combining Options

Any of the above options can be combined in a single command. For example:

Device:/> traceroute server.example.com -count=2 -starthop=3 -maxhops=4
		
Hop#    RTT      RTT   Host
  3    10 ms    10 ms  10.131.48.2
  4    10 ms    10 ms  ge1-1-0-617.cty-pe3.una.se.ip.tzc.net [10.88.215.44]
  5    10 ms    10 ms  te2-1-80.zty-p2.sfl.se.ip.tzc.net [10.131.143.226]
  6   120 ms   120 ms  10.82.35.201
Maximum hops reached.

A complete description of all the command options can be found in the separate CLI Reference Guide.

2.6.8. The frags Command

IP datagram fragmentation results from the breaking down of larger packets into smaller datagram fragments that can fit within the Maximum Transmission Unit (MTU) size of the network equipment they must traverse. When such fragments are received by cOS Core, packet reassembly takes place to reconstruct the entire packet before it is forwarded.

The CLI command frags allows the administrator to examine the current status of the reassembly process. Using the frags command without any parameters lists the currently active reassemblies as shown in the example output below.

Device:/> frags
		
RecvIf Num  State   Source       Destination  Protocol Next  Timeout
------ ---- ------- ------------ ------------- -------- ----- -------
If1    886  Unknown 192.168.1.6  192.168.2.1  ESP          0 593/593
If2    890  Accept  10.0.1.60    192.168.3.1  UDP       7280 592/591
If3    345  Drop    192.168.0.2  10.1.1.2     UDP       1494 581/101

Each line of output from this command shows the status of an individual reassembly operation where at least one fragment of a packet has been received as an IP datagram. Each datagram in the reassembly process is uniquely identified by its source/destination IP address and its protocol number. A reassembly operation will remain in the output of the command as long as more fragments might be received.

Reassembly States

The State of reassembly shown by the frags command output can be one of the following:

Done

Reassembly is complete but reassembly is kept alive for a short period in case there are any duplicate fragments.
Drop

cOS Core has determined that the reassembled packet is to be dropped based on the configured rules. This is the opposite of the Accept state and all matching fragments received will be dropped.
Unknown

This indicates that it has not yet been determined if the packet is to be dropped or accepted.
Accept

This state indicates it has been determined to not drop the packet based on the configured rules. This is the opposite of Drop and matching fragments received are accepted.
Free

This indicates a reassembly slot that is available for starting a new reassembly.

Options for the frags Command

To see only the reassembly slots that are in the Free state, use the -free option:

Device:/> frags -free

To see reassembly operations that are complete use the -done option:

Device:/> frags -done

Maximum Length Settings

cOS Core allows the following settings to be used to control the maximum size of incoming packets for different protocols so that packets exceeding these sizes are dropped:

Max UDP Length - Maximum size of UDP packets (default: 60,000 bytes).
Max GRE Length - Maximum size of GRE packets (default: 2000 bytes).
Max ESP Length - Maximum size of IPsec ESP packets (default: 2000 bytes).
Max TCP Length - Maximum size of a TCP packet (default: 1480 bytes).

However, the MTU value of the individual cOS Core interfaces determines how the packet size is split. For example, the maximum UDP length could be set to 60,000 but the interface MTU size might be 1500 so packets would be split into 41 fragments (60,000/1500).

Keeping these maximum settings to the lowest possible value is beneficial since unreasonably large packets can be used as a form of attack and they are immediately rejected by cOS Core when they exceed the set maximum.

TCP Length

With TCP, all normally configured TCP stacks will limit the size of TCP packets to the Maximum Segment Size (MSS) value. This MSS value will normally be the MTU value minus the IP header size of 20 bytes. With an MTU value of 1500 bytes, the MSS will be 1480 bytes and this will normally never need to be fragmented.

Frags in Lingering State

When using the CLI command "stat" the output can in some situations show lingering fragments. as shown below:

Fragments : 1021 out of 1024 (1021 lingering)

This means that cOS Core has successfully reassembled all the fragments but they are kept in the buffer in case more fragments arrive. If new fragmented traffic arrives that needs to be tracked/processed, the fragments in lingering state will be replaced over time.

2.6.9. The selftest Command

It can be the case that operational problems are caused by a problem with the underlying hardware platform and not cOS Core. For this reason, the CLI command selftest is provided to perform tests on various aspects of hardware functioning.

	Warning: Do NOT conduct tests with live traffic!
	It is important to remember that the selftest command should not be used on a system that is carrying live traffic. The command can cause connections and associated data to be lost and the test results themselves could then become unreliable.

Preparing the System for Testing

To ensure the complete reliability of any selftest, it is recommended to take a complete backup of the current configuration and reset the entire system to the default cOS Core configuration as well as having the hardware platform disconnected from any networks.

This is also true for units in an HA cluster. The cluster should be broken up into two separate hardware units and they should each be reset to the base configuration.

Resetting to the base configuration can be done through the CLI or Web Interface or InControl. Using the CLI, the command is:

Device:/> reset -configuration

RAM Memory Selftest

An example of selftest usage is to test the volatile system RAM memory:

Device:/> selftest -memory

This repeatedly writes a one megabyte block of data across the RAM memory space and then reads it back to check all write/read operations were error free.

By default, the memory test is performed only once across the memory but this could be repeated using the -num option. For example, to repeat the test 10 times:

Device:/> selftest -memory -num=10

Media Selftest

Another example of selftest usage is to test if the non-volatile system disk media is functioning:

Device:/> selftest -media

The disk media is often solid state and usually physically separate from RAM. The test writes a single test file called medtest.tmp onto the media and then reads this file back to verify its contents before deleting it. It therefore tests that both the media disk memory and the media file system are functioning. The test does not affect any existing files.

The media test uses a default file size of one megabyte but this size can be changed using the -size option which takes a whole number of megabytes as its value. For example, to set the tested file size to 10 megabytes:

Device:/> selftest -media -size=10

Throughput Testing

The following command options test traffic throughput:

-throughout

This generates the maximum achievable traffic flow through all specified interfaces using the maximum packet size. This option does not validate the received packets.
-traffic

This test generates traffic of mixed packet sizes between 60 and 1,518 bytes and verifies the content of each received frame. Received packets are verified.

For either the -throughput or the -traffic option, the test should be run so that interfaces are connected together and the output from one interface is received by another. This can be achieved in one of two ways:

Connection Through a Switch

All the interfaces are connected to the same switch which is itself disconnected from any other networks. This is only satisfactory if the hardware platform provides the same maximum link speed on all Ethernet interfaces since faster interfaces can overwhelm slower ones.
Connecting Ethernet Interfaces In Pairs

With a hardware platform that provides mixed Ethernet interface speeds, it is necessary to connect interfaces with similar maximum link speeds together in adjacent pairs. A switch should not be used. Testing should be done on one pair of interfaces at a time. For example, if the -throughput option is to be applied between the If1 and If2 interfaces, the command would be:
```
Device:/> selftest -interfaces=If1,If2 -throughput
```

The -burnin Option

If the -burnin option is used, a set of tests, known as the test subset, is repeated continuously for a period of time. The default test period is two hours and the default subset is:

-memory
-ping
-throughput
-traffic
-cryptoaccel

The duration of the -burnin test can be changed, as can the test subset. For example, a test of the memory and media that is repeated for 30 minutes would be:

Device:/> selftest -burnin -minutes=30 -memory -media

The -burnin option could, for example, be used to detect errors that only occur sporadically or possibly only occur after some hardware component has reached a certain critical operating temperature.

The full list of options for selftest CLI command can be found in the separate CLI Reference Guide along with more command line examples.

2.6.10. The techsupport Command

The techsupport CLI command is a general command for problem troubleshooting that combines the output from several other troubleshooting commands into a single console output. This information can then used by the administrator but is usually sent to qualified support personnel to troubleshoot problems. It is the first and most helpful source of troubleshooting information and the output file is often referred to by support personnel as the TSF (Technical Support File).

The techsupport command can be run in one of the following ways:

By entering the techsupport command in a CLI console. This will cause output to be sent to the console. Since the contents can be long, this may not be the most appropriate way to run the command.
Run the command via the Web Interface. The cOS Core Web Interface provides a way to run the techsupport command by clicking a single button. The output from the command is saved to a file which is automatically saved to the local disk of the management computer running the browser.

This method of running the command is the best way to obtain the output as a file which can then be sent to qualified support personnel for analysis.

The file created by cOS Core will have a filename with the format techsupport-yyyymmdd.txt where yyyymmdd is the date of file creation. The header of the file also contains a timestamp which indicates the exact system time the file was created.

64 bit cOS Core Requires Manual Download of Crash Dumps

For older versions of cOS Core, the latest crash dump is included in the output from the techsupport command. For newer 64 bit versions this is not the case so the latest crash dump must be manually downloaded for inclusion in a support ticket submission. This is the case for the RSG-400.

All virtual cOS Core versions for ARM platforms are 64 bit versions. In addition, newer Clavister hardware such as the 100, 300, 500 and 6000 Series (and later) are running 64 bit cOS Core. If unsure about the version type, 64 bit versions will indicate this in the first line of the output from the about command:

Device:/> about
Clavister cOS Core 14.00.06.01-36868 (x86 64-bit)

Manual crash dump downloading can be done through the Web Interface by going to Status > Maintenance > System Error Reports (64 bit versions 14.00.06 and later). Alternatively, any suitable SCP client can be used (all 64 bit versions). Crash dump files always have the filetype .dmp and are stored in a folder called crashdumps under the firewall's filesystem root. The simplest SCP download method is to download all available files as a single zip file with an SCP command in the following form:

> scp admin@<fw-ip-address>:crashdumps/all.dmp .

Alternatively, an individual crash dump file could be downloaded with the following command form:

> scp admin@<fw-ip-address>:crashdumps/<filename>.dmp .

Example 2.37. Creating techsupport Output

This example shows the techsupport command can be invoked in the CLI and the Web Interface.

Command-Line Interface

Device:/> techsupport

Web Interface

Go to: Status > Maintenance > Technical Support
Click the button Download Support File
A file chooser dialog will appear to save the file with the given filename to the local disk.
Click OK

2.6.11. Creating an Anonymous Configuration Copy

The option exists to download a copy of the current cOS Core configuration to a file and to make the copy anonymous. This file will be the same as a normal configuration backup file with the one exception that all sensitive data is automatically anonymized.

This anonymizing process means that data such as passwords, pre-shared keys and certificates, are automatically replaced by random data in the downloaded file. Using an anonymous configuration copy is recommended when sending a configuration for review by a third party, such as external support personnel. However, this type of configuration copy should not be confused with a true configuration backup copy, which is discussed in Section 2.7.4, Backing Up Configurations.

An anonymous configuration copy file created by cOS Core will have a filename with following format:

 anonymous_config-<firewall-device-name>-yyyymmdd-v<version>.bak

Where yyyymmdd is the date of file creation and <version> is an integer which indicates how many times the configuration has been changed (the number of activations) since it was last in the default configuration state. For example:

 anonymous_config-MyFirewall-20210214-v081.bak

Note that creating an anonymous configuration copy can only be done using the Web Interface.

Example 2.38. Creating an Anonymous Configuration Copy File

This example shows how an anonymous copy of the current cOS Core configuration can be created using the Web Interface.

Web Interface

Go to: Status > Maintenance > Technical Support
Click the button Download configuration file
A file chooser dialog will appear to save the file with a given filename to the local disk.
Click OK

2.6.12. The Linktest Command

The Linktest CLI command in cOS Core is a diagnostic tool designed to verify the integrity and reliability of network links. This command establishes a connection to either a Measurement Lab (M-Lab) NDT7 server or a manually specified web server, conducting tests to assess network link health.

	Note: This is not a performance test tool
	While it provides insights into the network's status, it is not designed as a performance measurement tool. Its primary function is to confirm the operational status of the link rather than measuring its maximum throughput or bandwidth capacity.

Network Diagnostic Tool

Network Diagnostic Tool (NDT) is a single stream performance measurement of a connection's capacity for “bulk transport” (as defined in IETF's RFC-3148). NDT measures “single stream performance” or “bulk transport capacity”. NDT reports upload and download speeds.

NDT7, developed by Measurement Lab (M-Lab), is the seventh iteration of NDT. This protocol is known for its efficiency in diagnosing network issues, leveraging modern techniques like the BBR congestion control algorithm for accurate analysis.

In its simplest form, the CLI command to start the Linktest towards a random NDT server takes the form:

Device:/> linktest -start

To show the results or the current status of the test, the -show flag is used:

Device:/> linktest -show
Status:          Completed (2023-11-10 09:51:13)
Server URL:      https://ndt-mlab1-arn04.mlab-oti.measurement-lab.org

Result:
 Download                15.6 Mbps
 Upload                  24.1 Mbps

Using the -verbose flag

For a more verbose output of the results, the "-verbose" flag can be used:

Device:/> linktest -show -verbose
Status:          Completed (2023-11-10 09:51:13)
Duration:        22s
Protocol:        NDT7
Clients:         1
Interface:       if1
Server IP:       62.115.225.139
Server Region:   Sweden
Server URL:      https://ndt-mlab1-arn04.mlab-oti.measurement-lab.org

Download Statistic:
 NDT7 Server Stats       15.2 Mbps ( 17.7MB / 9.76s) (1.69 Kbps upload)
 Iface Stats             15.6 Mbps ( 20.7MB / 11.1s) ( 649 Kbps upload)
 HTTP Stats              15.2 Mbps ( 20.0MB / 11.1s) ( 420  bps upload)
Upload Statistic:
 NDT7 Server Stats       23.5 Mbps ( 28.0MB / 10.0s) (41.7 Kbps download)
 Iface Stats             24.1 Mbps ( 28.8MB / 10.0s) ( 659 Kbps download)
 HTTP Stats              23.1 Mbps ( 27.6MB / 10.0s) (37.1 Kbps download)

The NDT7, Iface and HTTP Stats are showing statistical data from three different measurement sources. The statistics are from the NDT7 server, the download (or upload) interface of the firewall and the HTTP data from the test connection itself.

	Note: The "Iface stats" may contain large differences in data
The Iface Stat information may contain a large amount of data not aligned with the NDT7 and HTTP statistical values. The reason for this is because the Iface Stat data is from the interface used for the test. This means that if the interface is processing other types of data, that data will be added into the measurement sample. This is normal and the reason why this particular statistic will not align with other statistical values.

Note: The "Iface stats" may contain large differences in data

The Iface Stat information may contain a large amount of data not aligned with the NDT7 and HTTP statistical values. The reason for this is because the Iface Stat data is from the interface used for the test. This means that if the interface is processing other types of data, that data will be added into the measurement sample. This is normal and the reason why this particular statistic will not align with other statistical values.

A clarification of the verbose output if using the Download Statistics from the NDT7 server stat output above as an example:

First Value (15.2 Mbps)

This is the average download speed in bps calculated from downloaded data amount over time.
Second Value (17.7MB)

This is the amount of data that was transferred in the test.
Third Value (9.76s)

This is the amount of time the test has been running according to the server (periodic reporting). The reason this value is different for e.g. NDT7 Server Stats is because the server reports this information periodically.
Fourth Value (1.69 Kbps upload)

This is how much control data that was sent between the client and server for verification purposes.

Additional Linktest command examples

Two additional examples on how the command can be executed along with explanation of what the syntax does:

Device:/> linktest -start -clients 2 -pbr=Table_2 -host=MyTestServer
 -port=7777 -ssl=True

This syntax initiates a Linktest towards a specific NDT7 server (MyTestServer) using two simultaneous data streams (clients). Additionally, the test uses a manually specified port (7777), a specific routing table (Table_2) and lastly the test will be done using Secure Sockets Layer (SSL).

Device:/> linktest -start_http GET 
http://myserver.example.com/MyFile.zip -clients 2 -limit 100 -timeout 100

This syntax initiates a custom Linktest using a HTTP GET operation towards a specific test server URL (http://myserver.example.com/MyFile.zip) using two simultaneous data streams(clients). Additionally, the test has two limits defined. The first limit is that a maximum of 100 MB of data should be downloaded and secondly the test should not be run for longer than 100 seconds. These limits can be useful to set to avoid the test is performed for a long time in case the connection speed is low or bad.

	Note: The URL determines if encryption should be used
	If the URL starts with HTTP:// unencrypted HTTP is used and if it starts with HTTPS:// encrypted HTTPS will be used.