cOS Core 14.00.13 Data Collection Guide

Target Audience

The target audience for this guide is the administrator of a Clavister firewall running cOS Core who is reporting troubleshooting information to support personnel by creating a ticket in the Clavister support ticket system with the relevant attached information files. The support ticket can then used as a means to track the progress and resolution of the issue.

The Scope of this Guide

This guide outlines the data collection procedures to be used with the cOS Core product when entering a Support Ticket into the online Clavister support system. The procedures it describes should be applied for each ticket submitted.

Following the recommendations in this document ensures that appropriate and sufficient information is provided to support personnel in order to achieve timely issue resolution.

Support Tickets

A Support Ticket is the basis for all issue reporting to Clavister support personnel. It is created online after logging in to the MyClavister section of the Clavister website and then selecting the Help Desk option. After creation, each ticket goes through a defined process towards a satisfactory resolution. Creating tickets and the resolution process are described in Chapter 2, Creating Tickets.

A ticket single relates to a single issue for which resolution is sought. Once entered, both the customer and support personnel can provide and exchange information via the ticket system with email notifications going to the customer when the ticket's status changes.

Providing the Required Information is Important

If a ticket is incorrectly entered or the necessary information and/or attachments are missing, processing can be delayed or may even require resubmission. It is therefore vital to provide all the mandatory information and any relevant optional information.

The information that is required and the information which is optional is described in Chapter 2, Creating Tickets as well as in Chapter 3, Attachments.

This chapter looks in detail at how a Support Ticket is created when reporting an issue to Clavister support personnel.

Using the Support Ticket System

To create a new ticket in the Clavister support ticket system, use the following steps:

Support Ticket Components

The following submitted components make up a support ticket:

A. The Issue Description

B. Attachments

These two components will now be described further.

A. The Issue Description

This part allows the entry of free text to describe the issue. The text description is in two parts:

The longer text description should cover the following:

B. Attachments

A support ticket usually also includes attachments which consist of files containing system output that help support personnel to understand the issue. These files should be compressed together into a single ZIP file. The ZIP file is then added after ticket creation by opening the ticket and then dragging and dropping the ZIP file into the attachment box on the webpage.

The attachment should be a single compressed ZIP file. Do not use tar + gzip.

Where the ZIP file exceeds a size 250 MBytes, it should not be attached to a ticket in the MyClavister system. Instead, support personnel should be contacted to get instructions for uploading to Clavister support's FTP server. A request for FTP upload of attachments could be made in the description text that is entered when the ticket is created.

Attachments are described in more detail in Chapter 3, Attachments.

Available Customer Options After Ticket Creation

Once a ticket is created, there are three ticket options available to the administrator:

The Ticket Life Cycle and Ticket States

A support ticket has a large number of possible states which indicate its progress in the ticket system. However, only a small number of states will be encountered by a typical ticket and those are the ones that will be discussed here.

The typical ticket life cycle and the associated states are as follows:

Email Notifications

Any change of support ticket state is reported to the customer automatically via an email. The email does not state what has changed in the ticket, only that there has been a change.

The addition of comments by support personnel will also cause a notification email to be sent even though there may be no state change.

Multiple Organization Users Can Track a Single Ticket

Multiple login accounts can be associated with a single organisation in the support system and it is possible for any of these to see a ticket and also get email notifications of status changes.

An Explanation of the MyClavister User Account Hierarchy

Each customer organization should have at least one initial top-level administrator MyClavister account. Each administrator account will have product licenses associated with it and also has a complete overview of tickets associated with those licenses. For smaller organizations this can be sufficient.

However, for larger organizations with many product installations, multiple new user accounts could be created later, with different login credentials and email addresses, and these can then be linked with the organization. Linking is done by an administrator account inviting in a new user so they become associated with the same organization as the administrator. The administrator can then assign access rights to the newly included user. A new user could be given administrator privileges too or, more typically, they might be given only restricted access to support tickets and/or licenses.

It is often beneficial for a larger organization to have one or a small number of administrator user accounts and then different user groups could create accounts which are invited in by an administrator to be part of an organization. This will mean that accounts can have a separate email address so that email traffic related to their support tickets will go only to the user who created the ticket. However, an administrator or any user with support ticket access privileges will still have the ability to see the ticket details for all the accounts associated with an organization.

This chapter reviews the attachments that could be included with a support ticket in a single ZIP file and describes how they are created and the folder structure in which they should be sent. Before beginning the process of gathering data, a number of points should be noted:

Mandatory and Optional Attachments

Attachments are considered to be either mandatory or optional. When submitting a new ticket, it is advisable to provide the mandatory attachments (configuration and techsupport files) since they are considered the minimum amount of information usually needed for timely issue resolution.

The optional attachments should be provided where the ticket creator judges they are relevant to the reported issue. Both types of attachments will now be described further.

A. Mandatory Attachments

B. Optional Attachments

The Format and Folder Structure for Submitted Data

All attachments for a given support ticket should be submitted in a single compressed ZIP file. The structure of the compressed files submitted should follow the hierarchical folder structure shown below, with subfolders included as needed:

	TIC-nnnn
		Eventn_FWnn_YYYYMMDD_HHMM
			Config
			Techsupport
			CLI
			Logs
			Pcap
			Statistics
			VM
			HW

At a minimum, the Config and Techsupport subfolders should always be included. Other folders should be included when relevant.

The folder naming scheme used in the above structure is the following:

Including Multiple Incidents/firewalls in One Ticket

When reporting several issues within the same support ticket, place each issue into its own Eventnnn_FWnn_YYYYMMDD_HHMM folder.

When the issue involves a high availability cluster of two firewalls, there should be one Eventnnn_FWnn_YYYYMMDD_HHMM folder for the master unit and one for the slave.

Each of the lower level subfolders is designed to hold those files related to a specific subject and the different subfolders and subjects are discussed next.

The Config Folder

This mandatory folder contains attachments describing the configuration and network setup. This includes the following:

The Techsupport Folder

This mandatory folder holds the techsupport file(s) created by running the techsupport CLI command in cOS Core through either the Web Interface or CLI. This is a key tool for problem troubleshooting that combines the output from several other troubleshooting CLI commands into a single text file. It is the most important source of troubleshooting information and the output file is often referred to by support personnel as the TSF (Technical Support File).

The techsupport tool can be run in one of the following ways:

Ideally, the techsupport tool should be run immediately after a problem occurs so that the state of the firewall after the event can be viewed. However, in some cases, running the command as late as possible prior to the event can provide additional information or may even be necessary if the tool cannot function after the critical event.

For HA configurations, the techsupport tool should be run on both the master and slave units.

64 bit cOS Core Requires Manual Download of Crashdumps

For older versions of cOS Core, the latest crashdump is included in the output from the techsupport command. For newer 64 bit versions this is not the case so the latest crashdump must be manually downloaded for inclusion in a support ticket submission.

All virtual cOS Core versions for ARM platforms are 64 bit versions. In addition, newer Clavister hardware such as the 100, 300, 500 and 6000 Series (and later) are running 64 bit cOS Core. If unsure about the version type, 64 bit versions will indicate this in the first line of the output from the about command:

System:/> about
Clavister cOS Core 14.00.06.01-36868 (x86 64-bit)

Manual crashdump downloading can be done through the Web Interface by going to Status > Maintenance > System Error Reports (64 bit versions 14.00.06 and later). Alternatively, any suitable SCP client can be used (all 64 bit versions). Crashdump files always have the filetype .dmp and are stored in a folder called crashdumps under the firewall's filesystem root. The simplest SCP download method is to download all available files as a single zip file with an SCP command in the following form:

> scp admin@<fw-ip-address>:crashdumps/all.dmp .

Alternatively, an individual crashdump file could be downloaded with the following command form:

> scp admin@<fw-ip-address>:crashdumps/<filename>.dmp .

The CLI Folder

This optional folder holds any relevant cOS Core console output which is additional to the output of the techsupport tool. An administrator console should be opened either locally or remotely to capture this output from the firewall.

The Logs Folder

This optional folder contains log event and snooping information. A text file of log messages in cOS Core's own memlog buffer can be downloaded as a text file through the Web Interface by going to Status > Logging > System Log and pressing the Download Logs button.

Logs can also be extracted from the memlog buffer as CLI console output using the logsnoop command with suitable options. This command can also provide log messages in real-time on the console. The console output could be provided in a text file which is included with the ticket. The logsnoop command are described further in the separate cOS Core Administration Guide and CLI Reference Guide.

A recommendation is to provide logs that cover at least 30 minutes before the issue occurred and, where possible, 30 minutes after the issue. The buffering provided by the memlog subsystem may be sufficient for this. Alternatively, log data in syslog format could be retrieved from a configured Syslog server and submitted.

For an HA configuration, log files should be provided for both the master and slave firewalls.

Output from snoop commands could also be included in the Logs folder, if relevant. Examples of these commands include the following:

The Pcap Folder

This optional folder contains output from packet dumps that are relevant to the problem. Packet dumps from the unprotected and protected side can give a detailed picture of the traffic pattern preceding the incident and may be essential for a root-cause analysis.

The easiest method for packet capture when the packet volumes are not large is to use the cOS Core pcapdump tool. When using this through the Web Interface, go to Status > Tools > Packet Capture and press Start Capture. After stopping the capture, a .cap file can be downloaded directly through the Web Interface for any interface however the filtering options are limited when using the tool in this way.

The pcapdump command in the CLI provides greater filtering options in order to restricting the memory resources required and this is described further in the separate cOS Core Administration Guide and CLI Reference Guide.

When capturing large packet volumes that might overwhelm the firewall, the recommendation is to use a computer with sufficient disk storage running Wireshark/TCPdump, possibly using a switch with a mirroring port.

It is recommended to include packet dumps within a time window of around 10 minutes before and 5 minutes after the incident, or long enough to cover the incident. If possible, keep each submitted file under 1 GB in size.

For an HA configuration, this operation should be done on both firewalls, if possible.

The Statistics Folder

This optional folder contains any additional statistical information which can be helpful in giving a complete picture of the system load and activity at the time of the incident.

The techsupport file already provides a snapshot of all the statistical counters. For retrieving additional statistical data, an SNMP client can be used along with the cOS Core MIB. An SNMP client can usually retrieve and store multiple statistical values over a period of time and then construct graphs or other presentations of the raw data. Using an SNMP client is more fully explained in the SNMP section of the separate cOS Core Administration Guide.

If additional statistical data is relevant to the issue, it is recommended, to record the statistics using SNMP during the period 12 hours before and then 2 hours after an incident.

The VM Folder

This optional folder contains a text file which describes the virtual environment in which the firewall is running.

Details should include the virtual environment name (for example, KVM), the hypervisor version number and any other details about the virtual environment that may be relevant.

If an HA configuration is using dissimilar virtual environments, both the master and slave environment should be described.

The HW Folder

This optional folder contains a text file which describes the firewall appliance involved in the issue if cOS Core is running on Clavister hardware and not in a virtual environment.

Details should include the product type and include details of any expansion modules being used. The serial number of the hardware involved may be useful.

If an HA configuration is using dissimilar hardware, both the master and slave should be described, although this is not officially supported by Clavister.