|Note: This document is also available in other formats|
A PDF version of this document along with all current and older documentation in PDF format can be found at https://my.clavister.com.
It also available in a framed HTML version.
Target AudienceThe target audience for this guide is the administrator of a Clavister firewall running cOS Core who is reporting troubleshooting information to support personnel by creating a ticket in the Clavister support ticket system with the relevant attached information files. The support ticket can then used as a means to track the progress and resolution of the issue.
The Scope of this GuideThis guide outlines the data collection procedures to be used with the cOS Core product when entering a Support Ticket into the online Clavister support system. The procedures it describes should be applied for each ticket submitted.
Following the recommendations in this document ensures that appropriate and sufficient information is provided to support personnel in order to achieve timely issue resolution.
Support TicketsA Support Ticket is the basis for all issue reporting to Clavister support personnel. It is created online after logging in to the MyClavister section of the Clavister website and then selecting the Help Desk option. After creation, each ticket goes through a defined process towards a satisfactory resolution. Creating tickets and the resolution process are described in Chapter 2, Creating Tickets.
A ticket single relates to a single issue for which resolution is sought. Once entered, both the customer and support personnel can provide and exchange information via the ticket system with email notifications going to the customer when the ticket's status changes.
Providing the Required Information is ImportantIf a ticket is incorrectly entered or the necessary information and/or attachments are missing, processing can be delayed or may even require resubmission. It is therefore vital to provide all the mandatory information and any relevant optional information.
|Important: This guide is subject to change|
This document is subject to change between cOS Core releases and will be updated whenever features affecting troubleshooting and logging are changed.
This chapter looks in detail at how a Support Ticket is created when reporting an issue to Clavister support personnel.
Using the Support Ticket SystemTo create a new ticket in the Clavister support ticket system, use the following steps:
Open a web browser and log into the relevant account at https://my.clavister.com.
As explained at the end of this chapter, a new account may need to be created which will then need to be associated with an organization's administrator account.
From the navigation menu, select Help Desk and then Register Support Ticket from the sub-menu.
Drop-down boxes now allow the product, the product version and the product license to be selected. The MyClavister account entering the ticket must have the relevant product license assigned to it. It is not possible to create a ticket without an associated product license.
The next step allows a short and longer textual description of the problem to be entered along with the estimated priority of the reported issue.
As an option, it is possible to enter the user's own internal reference number for the ticket.
The final step in ticket creation is to review the entered information and confirm it. Note that email notifications associated with the ticket are sent to the email address associated with the MyClavister account being used to create the ticket and this email address appears in this confirmation step.
As described at the end of Chapter 1, Introduction, an organization could create several MyClavister accounts in order to send different sets of ticket notifications to different email addresses.
At this point, the ticket is created and is automatically assigned a unique ticket number by the ticket system with the form TIC-NNNN.
The ticket can now be opened in MyClavister by the user and the relevant attachments added. These attachments and the textual descriptions entered for a ticket are described further below.
Support Ticket ComponentsThe following submitted components make up a support ticket:
A. The Issue Description
These two components will now be described further.
A. The Issue DescriptionThis part allows the entry of free text to describe the issue. The text description is in two parts:
A short one line summary description of the issue.
A longer, complete description of the issue.
The longer text description should cover the following:
Provide a further description of the issue.
Give a general description of the complete system. What is the computing platform and what is the software version?
Categorize the end-user impact by selecting a severity description from the list provided. The descriptions translate to one of the following severities:
Can the issue be easily reproduced?
What is the frequency of the issue?
Is the issue dependent on a specific change in the configuration?
Is the issue site specific?
What system activity was occurring when the issue appeared?
Have there been any changes to the system prior to the issue occurring.
Have there been any recent changes to the surrounding network prior to the issue occurring, such as the following:
Have any other parts of the network been upgraded?
Has there been significant changes in system loads?
When dealing with remote networks or remote clients, has there been any recent change in those networks or clients?
B. AttachmentsA support ticket usually also includes attachments which consist of files containing system output that help support personnel to understand the issue. These files should be compressed together into a single ZIP file. The ZIP file is then added after ticket creation by opening the ticket and then dragging and dropping the ZIP file into the attachment box on the webpage.
The attachment should be a single compressed ZIP file. Do not use tar + gzip.
Where the ZIP file exceeds a size 250 MBytes, it should not be attached to a ticket in the MyClavister system. Instead, support personnel should be contacted to get instructions for uploading to Clavister support's FTP server. A request for FTP upload of attachments could be made in the description text that is entered when the ticket is created.
Attachments are described in more detail in Chapter 3, Attachments.
Available Customer Options After Ticket CreationOnce a ticket is created, there are three ticket options available to the administrator:
Add attachments - As described above, these are troubleshooting files provided by the customer in a single ZIP file which is added to a ticket after creation.
Enter a comment - This is the primary means for discussion about the issue with support personnel.
Change the status to Closed - This should be done if the issue is resolved or the ticket doesn't need further action.
The Ticket Life Cycle and Ticket StatesA support ticket has a large number of possible states which indicate its progress in the ticket system. However, only a small number of states will be encountered by a typical ticket and those are the ones that will be discussed here.
The typical ticket life cycle and the associated states are as follows:
When the ticket has been created and is being examined and discussed, the ticket state is Open.
When support personnel have requested more information about a ticket, the state becomes Awaiting Information. This state could occur more than once as additional information is sought.
Supplying all the mandatory and any relevant optional information at the time of ticket creation can help to avoid this state.
When support personnel believe the issue has been resolved, the state will become Fixed. This is not the same as Closed.
When the state becomes Fixed, the ticket will remain in this state for two weeks. During these two weeks, the customer has the option to Reopen the ticket if they feel the issue has not been resolved.
If, after two weeks, the customer has not reopened the ticket, support personnel will set the state to Closed. The customer can also set the state to Closed if they believe the issue has been resolved and requires no further action.
The state reached when a ticket has been resolved. As described above, either support personnel or the customer will set the state to Closed.
Note that when a ticket is placed in the Closed state it cannot be reopened nor can further comments be added to it. If an issue reoccurs, a new ticket must be created, but this new ticket could refer to the older closed ticket.
Email NotificationsAny change of support ticket state is reported to the customer automatically via an email. The email does not state what has changed in the ticket, only that there has been a change.
The addition of comments by support personnel will also cause a notification email to be sent even though there may be no state change.
Multiple Organization Users Can Track a Single TicketMultiple login accounts can be associated with a single organisation in the support system and it is possible for any of these to see a ticket and also get email notifications of status changes.
An Explanation of the MyClavister User Account HierarchyEach customer organization should have at least one initial top-level administrator MyClavister account. Each administrator account will have product licenses associated with it and also has a complete overview of tickets associated with those licenses. For smaller organizations this can be sufficient.
However, for larger organizations with many product installations, multiple new user accounts could be created later, with different login credentials and email addresses, and these can then be linked with the organization. Linking is done by an administrator account inviting in a new user so they become associated with the same organization as the administrator. The administrator can then assign access rights to the newly included user. A new user could be given administrator privileges too or, more typically, they might be given only restricted access to support tickets and/or licenses.
It is often beneficial for a larger organization to have one or a small number of administrator user accounts and then different user groups could create accounts which are invited in by an administrator to be part of an organization. This will mean that accounts can have a separate email address so that email traffic related to their support tickets will go only to the user who created the ticket. However, an administrator or any user with support ticket access privileges will still have the ability to see the ticket details for all the accounts associated with an organization.
This chapter reviews the attachments that could be included with a support ticket in a single ZIP file and describes how they are created and the folder structure in which they should be sent. Before beginning the process of gathering data, a number of points should be noted:
Create Attachments Directly After Issue Occurrence
Try to create all attachments as soon as possible after the issue occurs and, where possible, before a system reboot.
Check the System Date and Time
Before commencing the data collection necessary for support ticket attachments, make sure that the cOS Core system date and time are set correctly.
This is important for the accuracy of the timestamps on submitted data.
CLI Console Access
Access to the cOS Core CLI is usually done via an Ethernet interface using SSH. SSH access will require using third-party console client software such as PuTTY. The CLI output can then be saved to a file for submission as an attachment. In the PuTTY client this is done by selecting Session > Logging > All Session Output.
Some of the CLI commands used for data collection can generate a large amount of output. For this reason, the output buffer size of the console client being used for CLI access should be checked and increased if necessary.
The same also applies if the CLI is being accessed with a client directly via the local console port instead of via an Ethernet interface using SSH.
Mandatory and Optional AttachmentsAttachments are considered to be either mandatory or optional. When submitting a new ticket, it is advisable to provide the mandatory attachments (configuration and techsupport files) since they are considered the minimum amount of information usually needed for timely issue resolution.
The optional attachments should be provided where the ticket creator judges they are relevant to the reported issue. Both types of attachments will now be described further.
A. Mandatory Attachments
The firewall configuration in either anonymous or backup (non-anonymous) form.
The techsupport file generated by running the techsupport tool.
B. Optional Attachments
A network diagram picture to better explain the scenario.
Relevant CLI output in file form.
Relevant log messages.
Relevant snoop tool output.
Relevant packet dumps.
A description of the virtual environment (when relevant).
A description of the hardware platform (when relevant).
Any other information that is deemed relevant.
The Format and Folder Structure for Submitted DataAll attachments for a given support ticket should be submitted in a single compressed ZIP file. The structure of the compressed files submitted should follow the hierarchical folder structure shown below, with subfolders included as needed:
TIC-nnnn Eventn_FWnn_YYYYMMDD_HHMM Config Techsupport CLI Logs Pcap Statistics VM HW
At a minimum, the Config and Techsupport subfolders should always be included. Other folders should be included when relevant.
The folder naming scheme used in the above structure is the following:
TIC-nnn - Where nnn is the ticket ID. This ID is allocated automatically by the system when the ticket is created.
Eventn - Where n is the event ID. This is assigned by the customer. Information related to the first occurrence of an issue would use the name prefix Event1, the second Event2 and so on.
FWnn - Where nn is the firewall ID.
YYYYMMDD_HHMM - The date and time when the issue occurred. The date and time for Event1 will be before Event2 which is before Event3 and so on.
Including Multiple Incidents/firewalls in One TicketWhen reporting several issues within the same support ticket, place each issue into its own Eventnnn_FWnn_YYYYMMDD_HHMM folder.
When the issue involves a high availability cluster of two firewalls, there should be one Eventnnn_FWnn_YYYYMMDD_HHMM folder for the master unit and one for the slave.
Each of the lower level subfolders is designed to hold those files related to a specific subject and the different subfolders and subjects are discussed next.
The Config FolderThis mandatory folder contains attachments describing the configuration and network setup. This includes the following:
A Copy of the Configuration
This is mandatory and can be downloaded to the local disk as a single file using one of the following methods:
By creating an anonymous configuration copy file. This type of copy is created in the Web Interface by going to Status > Maintenance > Technical Support and clicking the Download configuration file button. This is the recommended method for making a configuration copy that will be reviewed by external support personnel.
An anonymous configuration copy means that sensitive information such as passwords, pre-shared keys and certificates, are automatically replaced by random data. This usually does not impair the ability to troubleshoot a problem.
By creating a non-anonymous backup config file. This is created in the Web Interface by going to Status > Maintenance > Backup & Restore and clicking the Backup Configuration button. This should only be needed if information missing from the anonymous copy is required for problem resolution.
A Network Diagram
A diagram describing the network layout around the firewall can be a very helpful aid to problem solving. This is not mandatory but can be very useful so it is recommended.
The diagram need not be professionally drawn but it should include IP numbers and the IP networks being used as well as the location of other equipment, including routers, clients, servers and other network devices.
The Techsupport FolderThis mandatory folder holds the techsupport file(s) created by running the techsupport CLI command in cOS Core through either the Web Interface or CLI. This is a key tool for problem troubleshooting that combines the output from several other troubleshooting CLI commands into a single text file. It is the most important source of troubleshooting information and the output file is often referred to by support personnel as the TSF (Technical Support File).
The techsupport tool can be run in one of the following ways:
Through the cOS Core Web Interface by going to Status > Maintenance > Technical Support and clicking the Download support file button. The output from the command can now be saved to a file on the local disk of the management computer running the browser. This is the recommended method.
The file created by cOS Core will have a filename with the format techsupport-yyyymmdd.txt where yyyymmdd is the date of file creation. The header of the file also contains a timestamp which indicates the exact system time the file was created.
By entering the techsupport command in a CLI console:
This will cause output to be sent to the console. Since the contents can be long, this may not be the most appropriate way to run the command.
Ideally, the techsupport tool should be run immediately after a problem occurs so that the state of the firewall after the event can be viewed. However, in some cases, running the command as late as possible prior to the event can provide additional information or may even be necessary if the tool cannot function after the critical event.
For HA configurations, the techsupport tool should be run on both the master and slave units.techsupport command. For newer 64 bit versions this is not the case so the latest crashdump must be manually downloaded for inclusion in a support ticket submission.
All virtual cOS Core versions for ARM platforms are 64 bit versions. In addition, newer Clavister hardware such as the 100, 300, 500 and 6000 Series (and later) are running 64 bit cOS Core. If unsure about the version type, 64 bit versions will indicate this in the first line of the output from the about command:
System:/>about Clavister cOS Core 14.00.03.01-36868 (x86 64-bit)
Manual crashdump downloading can be done using any suitable SCP client. Crashdump files always have the filetype .dmp and are stored in a folder called crashdumps under the firewall's filesystem root. The simplest download method is to download all available files as a single zip file with an SCP command in the following form:
> scp admin@<fw-ip-address>:crashdumps/all.dmp .
Alternatively, an individual crashdump file could be downloaded with the following command form:
> scp admin@<fw-ip-address>:crashdumps/<filename>.dmp .
The CLI FolderThis optional folder holds any relevant cOS Core console output which is additional to the output of the techsupport tool. An administrator console should be opened either locally or remotely to capture this output from the firewall.
The Logs FolderThis optional folder contains log event and snooping information. A text file of log messages in cOS Core's own memlog buffer can be downloaded as a text file through the Web Interface by going to Status > Logging > System Log and pressing the Download Logs button.
Logs can also be extracted from the memlog buffer as CLI console output using the logsnoop command with suitable options. This command can also provide log messages in real-time on the console. The console output could be provided in a text file which is included with the ticket. The logsnoop command are described further in the separate cOS Core Administration Guide and CLI Reference Guide.
A recommendation is to provide logs that cover at least 30 minutes before the issue occurred and, where possible, 30 minutes after the issue. The buffering provided by the memlog subsystem may be sufficient for this. Alternatively, log data in syslog format could be retrieved from a configured Syslog server and submitted.
For an HA configuration, log files should be provided for both the master and slave firewalls.
Output from snoop commands could also be included in the Logs folder, if relevant. Examples of these commands include the following:
ARP Snoop Data
ARP data is gathered using the CLI command:
This command should be repeated for each relevant interface.
IKE Snoop Data
To monitor the activity of IPsec tunnel setup, the following command can be used:
The Pcap FolderThis optional folder contains output from packet dumps that are relevant to the problem. Packet dumps from the unprotected and protected side can give a detailed picture of the traffic pattern preceding the incident and may be essential for a root-cause analysis.
The easiest method for packet capture when the packet volumes are not large is to use the cOS Core pcapdump tool. When using this through the Web Interface, go to Status > Tools > Packet Capture and press Start Capture. After stopping the capture, a .cap file can be downloaded directly through the Web Interface for any interface however the filtering options are limited when using the tool in this way.
The pcapdump command in the CLI provides greater filtering options in order to restricting the memory resources required and this is described further in the separate cOS Core Administration Guide and CLI Reference Guide.
When capturing large packet volumes that might overwhelm the firewall, the recommendation is to use a computer with sufficient disk storage running Wireshark/TCPdump, possibly using a switch with a mirroring port.
It is recommended to include packet dumps within a time window of around 10 minutes before and 5 minutes after the incident, or long enough to cover the incident. If possible, keep each submitted file under 1 GB in size.
For an HA configuration, this operation should be done on both firewalls, if possible.
The Statistics FolderThis optional folder contains any additional statistical information which can be helpful in giving a complete picture of the system load and activity at the time of the incident.
The techsupport file already provides a snapshot of all the statistical counters. For retrieving additional statistical data, an SNMP client can be used along with the cOS Core MIB. An SNMP client can usually retrieve and store multiple statistical values over a period of time and then construct graphs or other presentations of the raw data. Using an SNMP client is more fully explained in the SNMP section of the separate cOS Core Administration Guide.
If additional statistical data is relevant to the issue, it is recommended, to record the statistics using SNMP during the period 12 hours before and then 2 hours after an incident.
The VM FolderThis optional folder contains a text file which describes the virtual environment in which the firewall is running.
Details should include the virtual environment name (for example, KVM), the hypervisor version number and any other details about the virtual environment that may be relevant.
If an HA configuration is using dissimilar virtual environments, both the master and slave environment should be described.
The HW FolderThis optional folder contains a text file which describes the firewall appliance involved in the issue if cOS Core is running on Clavister hardware and not in a virtual environment.
Details should include the product type and include details of any expansion modules being used. The serial number of the hardware involved may be useful.
If an HA configuration is using dissimilar hardware, both the master and slave should be described, although this is not officially supported by Clavister.