cOS Stream 3.90.00 Data Collection Guide


Table of Contents

1. Introduction & Scope
2. Creating Tickets
3. Attachments

Chapter 1: Introduction & Scope

[Note] Note: This document is also available in other formats

A PDF version of this document along with all current and older documentation in PDF format can be found at https://my.clavister.com.

It also available in a framed HTML version.

The Scope of this Guide

This guide outlines the data collection procedures to be used with the cOS Stream product when entering a Support Ticket into the Clavister support system. The procedures it describes should be applied for each ticket submitted.

Following the recommendations in this document ensures that appropriate and sufficient information is provided to support personnel in order to achieve timely issue resolution.

Support Tickets

A Support Ticket is the basis for all issue reporting to Clavister support personnel. It is created online after logging in to the MyClavister section of the Clavister website and then selecting the Help Desk feature. After creation, each ticket goes through a defined process towards a satisfactory resolution. Creating tickets and the resolution process are described in Chapter 2, Creating Tickets.

A ticket single relates to a single issue for which resolution is sought. Once entered, both the customer and support personnel can provide and exchange information via the ticket system with email notifications going to the customer when the ticket's status changes.

Providing the Required Information is Important

If a ticket is incorrectly entered or the necessary information and/or attachments are missing, processing can be delayed or may even require resubmission. It is therefore vital to provide all the mandatory information and any relevant optional information.

The information that is required and the information which is optional is described in Chapter 2, Creating Tickets as well as in Chapter 3, Attachments.

[Important] Important: This guide is subject to change

This document is subject to change between cOS Stream releases and will be updated whenever features affecting troubleshooting and logging are changed.

Chapter 2: Creating Tickets

This chapter looks in detail at how a Support Ticket is created when reporting an issue to Clavister support personnel.

Using the Support Ticket System

To create a new ticket in the Clavister support ticket system, use the following steps:

  1. Open a web browser and log into the relevant account at https://my.clavister.com.

    As explained at the end of this chapter, a new account may need to be created which will then need to be associated with an organization's administrator account.

  2. From the navigation menu, select Help Desk and then Register Support Ticket from the sub-menu.

  3. Drop-down boxes now allow the product, the product version and the product license to be selected. The MyClavister account entering the ticket must have the relevant product license assigned to it. It is not possible to create a ticket without an associated product license.

  4. The next step allows a short and longer textual description of the problem to be entered along with the estimated priority of the reported issue.

    As an option, it is possible to enter the user's own internal reference number for the ticket.

  5. The final step in ticket creation is to review the entered information and confirm it. Note that email notifications associated with the ticket are sent to the email address associated with the MyClavister account being used to create the ticket and this email address appears in this confirmation step.

    As described at the end of Chapter 1, Introduction & Scope, an organization could create several MyClavister accounts in order to send different sets of ticket notifications to different email addresses.

  6. At this point, the ticket is created and is automatically assigned a unique ticket number by the ticket system with the form TIC-NNNN.

  7. The ticket can now be opened in MyClavister by the user and the relevant attachments added. These attachments and the textual descriptions entered for a ticket are described further below.

Support Ticket Components

The following submitted components make up a support ticket:

A. The Issue Description

B. Attachments

These two components will now be described further.

A. The Issue Description

This part allows the entry of free text to describe the issue. The text description is in two parts:

  • A short one line summary description of the issue.

  • A longer, complete description of the issue.

The longer text description should cover the following:

  • Provide a further description of the issue.

  • Give a general description of the complete system. What is the computing platform and what is the software version?

  • Categorize the end-user impact by selecting a severity description from the list provided. The descriptions translate to one of the following severities:

    • Low
    • Medium
    • High
    • Emergency

  • Can the issue be easily reproduced?

  • What is the frequency of the issue?

  • Is the issue dependent on a specific change in the configuration?

  • Is the issue site specific?

  • What system activity was occurring when the issue appeared?

  • Have there been any changes to the system prior to the issue occurring.

  • Have there been any recent changes to the surrounding network prior to the issue occurring, such as the following:

    1. Have any other parts of the network been upgraded?

    2. Has there been significant changes in system loads?

    3. When dealing with remote networks or remote clients, has there been any recent change in those networks or clients?

B. Attachments

A support ticket usually also includes attachments which consist of files containing system output that help support personnel to understand the issue. These files should be compressed together into a single ZIP file. The ZIP file is then added after ticket creation by opening the ticket and then dragging and dropping the ZIP file into the attachment box on the webpage.

The attachment should be a single compressed ZIP file. Do not use tar + gzip.

Where the ZIP file exceeds 25 MBytes, it should not be attached to a ticket in the MyClavister system. Instead, support personnel should be contacted to get instructions for uploading to Clavister support's FTP server. A request for FTP upload of attachments could be made in the description text that is entered when the ticket is created.

Attachments are described in more detail in Chapter 3, Attachments.

Available Customer Options After Ticket Creation

Once a ticket is created, there are three ticket options available to the administrator:

  • Add attachments - As described above, these are troubleshooting files provided by the customer in a single ZIP file which is added to a ticket after creation.

  • Enter a comment - This is the primary means for discussion about the issue with support personnel.

  • Change the status to Closed - This should be done if the issue is resolved or the ticket doesn't need further action.

The Ticket Life Cycle and Ticket States

A support ticket has a large number of possible states which indicate its progress in the ticket system. However, only a small number of states will be encountered by a typical ticket and those are the ones that will be discussed here.

The typical ticket life cycle and the associated states are as follows:

  • Open

    When the ticket has been created and is being examined and discussed, the ticket state is Open.

  • Awaiting Information

    When support personnel have requested more information about a ticket, the state becomes Awaiting Information. This state could occur more than once as additional information is sought.

    Supplying all the mandatory and any relevant optional information at the time of ticket creation can help to avoid this state.

  • Fixed

    When support personnel believe the issue has been resolved, the state will become Fixed. This is not the same as Closed.

    When the state becomes Fixed, the ticket will remain in this state for two weeks. During these two weeks, the customer has the option to Reopen the ticket if they feel the issue has not been resolved.

    If, after two weeks, the customer has not reopened the ticket, support personnel will set the state to Closed. The customer can also set the state to Closed if they believe the issue has been resolved and requires no further action.

  • Closed

    The state reached when a ticket has been resolved. As described above, either support personnel or the customer will set the state to Closed.

Email Notifications

Any change of support ticket state is reported to the customer automatically via an email. The email will not state what has changed in the ticket, only that there has been a change.

The addition of comments by support personnel will also cause a notification email to be sent even though there may be no state change.

Multiple Organization Users Can Track a Single Ticket

Multiple login accounts can be associated with a single organisation in the support system and it is possible for any of these to see a ticket and also get email notifications of status changes.

An Explanation of the MyClavister User Account Hierarchy

Each customer organization should have at least one initial top-level administrator MyClavister account. Each administrator account will have product licenses associated with it and also has a complete overview of tickets associated with those licenses. For smaller organizations this can be sufficient.

However, for larger organizations with many product installations, multiple new user accounts could be created later, with different login credentials and email addresses, and these can then be linked with the organization. Linking is done by an administrator account inviting in a new user so they become associated with the same organization as the administrator. The administrator can then assign access rights to the newly included user. A new user could be given administrator privileges too or, more typically, they might be given only restricted access to support tickets and/or licenses.

It is often beneficial for a larger organization to have one or a small number of administrator user accounts and then different user groups could create accounts which are invited in by an administrator to be part of an organization. This will mean that accounts can have a separate email address so that email traffic related to their support tickets will go only to the user who created the ticket. However, an administrator or any user with support ticket access privileges will still have the ability to see the ticket details for all the accounts associated with an organization.

Chapter 3: Attachments

This chapter reviews the attachments that could be included with a support ticket and describes how they are created and the file format in which they should be sent. Before beginning the process, a number of points should be noted:

  • Create Attachments Directly After Issue Occurrence

    Try to create all attachments as soon as possible after the issue occurs and, where possible, before a system reboot.

  • Check the System Date and Time

    Before commencing the data collection necessary for support ticket attachments, make sure that the firewall system date and time are set correctly.

    This is important for the accuracy of the timestamps on submitted data.

  • Console Client Usage

    Access to the CLI is usually done via SSH from a third party console client product such as Putty. The CLI output can then be saved to a file for submission as an attachment. In Putty, this is done by choosing: Session > Logging > All Session Output.

    Some of the CLI commands used for data collection can generate a large amount of output. For this reason, the output buffer size of the console client being used for CLI access should be checked and increased where necessary.

    The same also applies if the CLI is being accessed with a client directly via the RS232 console port instead of via SSH.

Mandatory and Optional Attachments

Attachments can be considered to be either mandatory or optional. When submitting a new ticket, it is advisable to provide all mandatory attachments (where relevant) since they are considered to make up the minimum amount of information for issue resolution.

The optional attachments should be provided where the ticket creator judges they are relevant to the reported issue. Both types of attachments are listed below:

A. Mandatory Attachments

  • The firewall configuration in backup (.bkp) file form.

  • The techsupport.txt file generated by running the techsupport command.

B. Optional Attachments

  • A network diagram picture.

  • Any relevant CLI output in file form.

  • Crashdump files.

  • Log messages.

  • Packet dumps.

  • Any other relevant information.

Directory Structure for Submitted Data

All attachments for a given support ticket should be submitted in a single compressed file, preferably using the ZIP file format.

The structure of the compressed files submitted should follow the hierarchical directory structure shown below, with folders included as needed:

	TIC-nnnn
		Eventn_FWnn_YYYYMMDD_HHMM
			Config
			Techsupport
			CLI
			Crashdump
			Logs
			Pcap
			Statistics
			VM
			

At the very minimum, the Config and Techsupport subdirectories should be included.

The variable names used in this directory structure are:

  • TIC-nnn - nnn is the ticket ID. This ID is allocated automatically by the system when the ticket is created.

  • Eventn - n is the event ID. This is assigned by the customer. Information related to the first occurrence of an issue would use the name prefix Event1, the second Event2 and so on.

  • FWnn - nn is the firewall ID.

  • YYYYMMDD_HHMM - the date and time when the issue occurred. The date and time for Event1 will be before Event2 which is before Event3 and so on.

Multiple Incidents/firewalls Per Ticket

When reporting several issues within the same support ticket, place each issue into its own Eventnnn_FWnn_YYYYMMDD_HHMM folder.

When the issue involves a high availibility cluster of two Clavister Firewalls, there should be one Eventnnn_FWnn_YYYYMMDD_HHMM folder for the master unit and one for the slave.

Each of the lower level directories in the attachments file is designed to hold files related to a specific subject and the different subjects are discussed next.

The Config Directory

The subdirectory contains attachments describing the configuration and network setup.

  • A Copy of the Configuration

    This consists of a backup file of the configuration which is created using the CLI command:

    System:/> backup -create

    The backup file is created on the firewall and must then be downloaded to a management computer using SCP. If the file created is called config-20170121.bkp, then a typical SCP client command to download the file would be:

    c:\> scp admin@198.10.2.4:config-20170121.bkp

    Here, the management IPv4 address is assumed to be 198.10.2.4 and the account name is admin. This command could also be used to download other diagnostic files from the firewall.

  • A Network Diagram

    A diagram describing the network layout around the firewall should be provided. This should include IP numbers and the IP networks being used as well as the location of other equipment, including routers, clients, servers and other network devices.

The Techsupport Directory

This subdirectory holds the file(s) created by running the techsupport command:
System:/> techsupport
This command automatically runs a large series of individual diagnostic commands and stores the collective console output in a text file called techsupport.txt which is placed in the firewall's root directory. SCP can then be used to download this file to a management computer.

Ideally, the techsupport command should be run immediately after a problem occurs so that the state of the firewall after the event can be viewed. However, in some cases, running the command as late as possible prior to the event can provide additional information or may even be necessary if the CLI cannot function after the critical event.

For HA configurations, the techsupport command should be repeated for both the master and slave units.

The CLI Directory

This subdirectory holds any relevant CLI output which is additional to the output of the techsupport command.

The Crashdump Directory

This subdirectory contains any crashdump files downloaded from the system. To list all crashdumps files on the firewall disk, use the CLI command:
System:/> crashdump -list

The filenames of available crashdump files are also listed in the techsupport.txt file generated by the techsupport command.

Crashdump files are always saved on the firewall in a directory called crashdumps. A typical SCP client command on a management computer to download all the available crashdump files might be:

C:\> scp admin@198.10.2.4:crashdumps/* ./

Crashdump filenames have the following format:

			yyyy-mm-dd_hh.mm.ss_binary.dump

For example, 2017-05-30_12.57.12_logd.dump.

After successful download of all the existing crashdump files, it is recommended to clear those files from firewall storage. This is done with the CLI command:

System:/> crashdump -rm=all

The Logs Directory

This subdirectory contains log event and snooping information. Log data should be recovered from a configured Syslog server.

A recommendation is to provide logs that cover at least 30 minutes before the issue occurred and, where possible, 30 minutes after the issue.

For an HA configuration, log files should be provided for both the master and slave unit.

Output from snoop commands could also be included in the Logs directory, if relevant. Examples of these commands include the following:

  • ARP Snoop Data

    ARP data is gathered using the CLI command:

    System:/> arpsnoop <interface>

    This command should be repeated for each relevant interface.

  • IKE Snoop Data

    To monitor the activity of IPsec tunnel setup, the following command can be used:

    System:/> ike -snoop=full

  • Radius Snoop Data

    Where a RADIUS server is being used, its activity can be recorded with the command:

    System:/> radiussnoop -on -verbose

The Pcap Directory

This subdirectory contains output from packet dumps.

Packet dumps from the unprotected and protected side give a detailed picture of the traffic pattern preceding the incident and may be essential for the root cause analysis.

The amount of data a firewall is exposed to in a production network can be extensive. However, if packet dumps from the time of the incident are available, these should be included with the support ticket.

The recommendation from Clavister is to use a computer running Wireshark/TCPdump with sufficient disk storage for storing large amounts of data in order to capture packets from all relevant interfaces on the unprotected and the protected side at the same time.

The amount of data that needs to be stored depends on how much data each firewall is exposed to and how fast the incident appears.

Include packet dumps within an interval of around 10 minutes before and 5 minutes after the incident, or enough to cover the incident. If possible, keep each submitted file under 1 GB in size.

For an HA configuration, this operation should be done on both nodes, if possible.

The Statistics Directory

This subdirectory contains statistical information which can be helpful in giving a complete picture of the system load and activity at the time of the incident.

The techsupport command already provides a snapshot of all the statistical counters. For retrieving additional statistical data, SNMP or the CLI could be used. However, SNMP is the recommended method because SNMP clients can usually retrieve and store multiple statistical values over a period of time time and then construct graphs or other presentations of the raw data.

It is up to the creator of a ticket to include the relevant statistics. It is recommended, if possible, to provide statistical data covering the 12 hours before and the 2 hours after an incident.

The VM Directory

The subdirectory contains a text file which describes the virtual environment in which the firewall is running.

Details should include the virtual environment name (for example, ESXi), the version number and any other details about the virtual environment that may be relevant.

If an HA configuration is using dissimilar virtual environments, both the master and slave environment should be described.

Downloading All Files Using SCP

As described earlier for backup configuration files, use a suitable SCP client running on a management computer to download files from the firewall. To download all files, a typical SCP client command on the management computer would be:
C:\> scp admin@198.10.2.4:* ./
Here, the firewall's management interface IP is assumed to be 198.10.2.4 and the account name is admin..

A typical SCP command for downloading all files and all folders would be:

C:\> scp -r admin@198.10.2.4:* ./