cOS Stream 3.80.13 Administration Guide

This chapter provides details of how to work with cOS Stream's management interfaces.

Management Access Methods

By default, cOS Stream provides a default LocalUserDatabase object which can be used to authenticate management access if this method is selected (by default, it is selected). This database contains, at a minimum, one predefined user with the following default credentials:

Username: admin
Password: admin

This account has full administrative read/write privileges to all configuration data and is always the account used for initial SSH login (console access does not require any credentials in the default configuration).

	Important: Change the default administrator password
	For security, it is strongly recommended to change the password for the predefined admin account as soon as possible following the initial configuration of cOS Stream.

Displaying the Local User Database

System:/> show LocalUserDatabase

  Name
  ----------
  AdminUsers

System:/> cc LocalUserDatabase AdminUsers

System:/LocalUserDatabase/AdminUsers> show
			
User
			
   Name   Groups          Comments
   -----  --------------  --------
   admin  Administrators  <empty>

System:/LocalUserDatabase/AdminUsers> cc
System:/> 

Creating an Auditor Account

If, however, the group is specified as the text string Auditors, the user will only have read-only privileges and will not be able to make configuration alterations. The following commands create an auditor account with a username of audit and a password of audit:

System:/> cc LocalUserDatabase AdminUsers
System:/LocalUserDatabase/AdminUsers> add User audit
			Password=audit
			Groups=Auditors

Linking Logins to the Database

For example, SSH logins are controlled by a RemoteMgmtSSH object. This object refers to an AuthenticationProfile object that then refers to the database to be used for authentication.

AuthenticationProfile objects are discussed further in Section 17.1, Authentication Profiles. The diagram below illustrates the relationship between the various components.

Figure 2.1. Local User Database Usage with SSH

The CLI provides a comprehensive set of commands that allow the display and modification of a configuration.

This section only provides a summary for using the CLI. For a complete reference for all CLI commands, see the separate Clavister cOS Stream CLI Reference Guide.

String Delimiters, the Escape Character and Special Characters

add User my-user Password="pass word"

set User my-user Password="pass\"word"

Other special characters that might be needed are the following:

For example:

set RemoteManagement RemoteMgmtSSH ssh Banner="Welcome!\r\n - Type \"help\"

Note that the only other case that requires a backslash escape character is when using the dollar sign ($) in a CLI script when it is not part of a variable such as $1. When used in another context within a CLI script, it must be written as \$. This is discussed further in Section 2.1.4, CLI Scripting.

Case Sensitivity

CLI Access Methods

Enabling SSH Access

SSH is a protocol primarily used for secure communication over insecure networks, providing strong authentication and data integrity. SSH clients are freely available for almost all platforms. cOS Stream supports version 2 of the SSH protocol.

A RemoteMgmtSSH rule controls SSH access on the default management interface. A predefined RemoteMgmtSSH rule exists in the default configuration but it is disabled. It can be displayed with the CLI command:

System:/> show RemoteManagement RemoteMgmtSSH
	
  # Name
  - -------------
o 1 RemoteMgmtSSH

To enable this rule, use the command:

System:/> set RemoteManagement RemoteMgmtSSH RemoteMgmtSSH -enable

All such rules also require their SourceInterface and SourceNetwork properties to be set for any traffic to be accepted:

System:/> set RemoteManagement RemoteMgmtSSH RemoteMgmtSSH
			SourceInterface=if1
			SourceNetwork=if1_net

The following output from a show command now confirms that the rule has all the required properties set and is enabled:

System:/> show RemoteManagement RemoteMgmtSSH
	
# Name          SourceInterface Source Network
- ------------- --------------- --------------
1 RemoteMgmtSSH if1             if1_net

Authentication for SSH access is controlled by the AuthProfile property of the RemoteMgmtSSH rule. By default this is set to a predefined AuthenticationProfile object called MgmtAuthProfile. This profile points to the predefined local user database which contains a predefined administrator account with the credentials:

The AgentType property of the authentication profile for SSH access should always be set to Basic. All authentication profile properties are explained in Section 17.1, Authentication Profiles.

Processing Multiple RemoteMgmtSSH rules

Authentication Methods for SSH Management Access

SSH and SCP with Public Key Authentication

Public key authentication is set up with the following steps:

The following should be noted about the behavior of SSH authentication using public keys:

Troubleshooting SSH Access Problems

Controlling Direct Console Access to the CLI

show COMPortAccess 

Port Bps    Data bits Parity Stop bits Flow control Rows Cols
---- ------ --------- ------ --------- ------------ ---- ----
COM1 115200 8         None   1         None         40   80
		

An AuthenticationProfile is associated with the ComPortAccess object to specify authentication. In the default configuration, no profile is specified so there is no username/password pair required for console login. The same profile used by default for SSH could be associated with console access by using the command:

System:/> set COMPortAccess COM1 AuthProfile=MgmtAuthProfile

Basic Object Management Commands

CLI Command Structure

	<command> <object_category> <object_type> <object_name>

System:/> show Address IPAddress my_address

When typing commands, the object category can be left out where the command's meaning is unambiguous. For example, the show command above could have been entered as:

System:/> show IPAddress my_address

However, tab completion will always assume that the category is included. For example, tab completion would not be able to help complete the above command if the tab is pressed during or after the IPAddress object type.

The same object name could be used within two different categories or types although this is best avoided in order to avoid ambiguity when reading configurations.

	Note: The terms Category and Context
	When describing the CLI, the term object category is also sometimes referred to as the object context.

A command such as add can also include object properties. To add a new IPAddress object with an IPv4 address of 10.49.02.01, the command would be:

System:/> add Address IPAddress my_address Address=10.49.02.01

Setting the System Name and CLI Prompt

System:/> 

set System Name=my-system1

my-system1:/>

CLI Help

System:/> help time
COMMAND
        time (t). Display and set current system time.
		"
		"

Typing the CLI command:

System:/> help help

will give information about the help command itself.

Configuration Object Help

System:/> helpconfig EthernetInterface

Applying a Single Command to Multiple Objects

An example of an object that has an index number is the IPRule object which must be a member of the rule list in an IPRuleSet. Suppose that there are 100 IPRule objects in the main IP rule set. To change the rules with an index number between 65 and 80 (inclusive) so that their LogEnabled property is disabled, the command would be the following:

System:/> cc RuleSet IPRuleSet main
System:/IPRuleSet/main> set IPRule -Range=65-80 LogEnabled=No

Instead a more complex range could be specified:

System:/IPRuleSet/main> set IPRule -Range=10,11-15,60-65 LogEnabled=No

Deletion can be specified in the same way:

System:/IPRuleSet/main> delete IPRule -Range=90-100

The CLI Command History

Tab Completion

For example, consider the following, incomplete command to add a LogReceiver object called my_server to the configuration:

System:/> add LogReceiver LogReceiverSyslog my_receiver
			IPAddress=my_log_server_ip
			LogSeverity=

If the tab character is now entered, a list of available options is displayed for the LogSeverity property:

System:/> add LogReceiver LogReceiverSyslog my_receiver
			IPAddress=my_log_server_ip
			LogSeverity=(tab)
       Type:  String
Description:  Specifies with what severity log events
will be sent to the specified log receivers
    Default:  Emergency,Alert,Critical,Error,Warning,Notice,Information

List of all valid choices:

Name
-----------
<empty>
Debug
Information
Notice
Warning
Error
Critical
Alert
Emergency

Tab Completion Only Displays Allowed Properties

Special Characters Used with Tab Completion

Object Categories and Tab Completion

If a command such as add is entered and then the tab key is pressed, cOS Stream displays all the available categories. By choosing a category and then pressing tab again all the available object types for that category are displayed. Using categories means that there is a simple way to specify what kind of object is being sought and a manageable number of options are displayed after pressing the tab key.

Tab completion and its options are discussed further in the introduction to the separate Clavister NetShield Firewall CLI Reference Guide.

Selecting Object Categories

Suppose a route is to be added to the routing table main. The first command would be:

System:/> cc RoutingTable main
		
System:/RoutingTable/main> 

Notice that the command prompt changes to indicate the current category/context. We can now add the route:

System:/RoutingTable/main> add Route
			Interface=if1
			Network=if1_net
			Name=new_route1

To return to the default CLI context, the command is cc on its own:

System:/RoutingTable/main> cc
		
System:/> 

The categories that require an initial cc command before object manipulation have a "/" character following their names when displayed by a show command. For example: RoutingTable/.

Specifying Multiple Property Values

		AccountingServers=server1,server2,server3

Resetting to the Default Value

System:/> add OSPFProcess Name=my_process1 RouterID=ip_ospf1
Added OSPFProcess OSPF
System:/> set OSPFProcess OSPF1 RouterID=<empty>
Updated OSPFProcess OSPF

Inserting into Rule Lists

Referencing by Name

The cOS Stream CLI Reference Guide lists the options available for each object, including Name= and Index=.

To make reading configurations easier, it is strongly advised to always add unique a unique name to important objects in a configuration. For example, IP rules should always have an appropriate name specified so that their purpose can be immediately understood.

Where the Name property is optional, it can also have duplicates but this is never recommended and can be confusing.

Logging On to the CLI

After SSH communication is established, cOS Stream will respond with a login prompt to pressing the Enter key. Type in the username the login prompt followed by the Enter key. Type the password followed by the Enter key again. After the first startup, cOS Stream will allow administrator login with the username admin and the password admin. This default password should be changed as soon as possible.

If accessing the CLI through a local console connection to the firewall, authentication will only be required if an AuthenticationProfile is explicitly associated with the console's COMPortAccess object.

After a successful logon, the CLI command prompt will appear:

System:/> 

The default welcome message will also be displayed directly after the logon. This SSH welcome message can be removed with the CLI command:

System:/> set RemoteManagement RemoteMgmtSSH ssh Banner=

Where ssh is the name of the RemoteMgmtSSH object. To reset the message:

System:/> set RemoteManagement RemoteMgmtSSH ssh Banner="Welcome!"

SSH Connection Timeout

System:/> set RemoteManagement RemoteMgmtSSH ssh
			SessionIdleTime=1000000

Changing the admin User Password

To change the password, for example to my-password, the following CLI commands are used. First, change the current CLI context to be the relevant LocalUserDatabase object:

System:/> cc LocalUserDatabase AdminUsers

The password of the admin user can now be changed:

System:/LocalUserDatabase/AdminUsers> set User admin Password="my-password"

Finally, return to the default CLI context:

System:/LocalUserDatabase/AdminUsers> cc

Activating and Committing Changes

System:/> activate

System:/> commit

If the commit command is not issued within 30 seconds (the default time period) then the activated changes are automatically undone and the old configuration restored.

Symbols Indicating Object Status

- means the object has been deleted.
o means the object has been disabled.
! means the object has errors.
+ means the object has been newly created.
* means the object has been modified.

Checking Configuration Integrity

System:/> show -errors

All the changes made to a configuration can be shown before activation with the command:

System:/> show -changes

Examining the Configuration Log

System:/> cfglog -all
		
Notice  vsinit/Configure:1805  2011-06-17 13:53:40
  - Beginning system reconfigure. System starting.
		
Notice  vsinit/Configure:1805  2011-06-17 15:59:30
  - Beginning system reconfigure. Activating new configuration.

Notice  vsinit/Configure:2033  2011-06-17 15:59:31
  - Reconfigure completed successfully.

The output from this command also shows if cOS Stream is running in demonstration mode, without a valid license.

System Performance Commands

Using the CLI, the statistics command provides a means to examine individual or groups of values. To see all values for system cpu usage, a wild card can be used as in this example with some typical output:

System:/> statistics -add /system/cpu/*
		
Name | 2013-03-05 13:36:11              Value  Unit
--------------------------------------  -----  ----
/system/cpu/control_plane_usage         1      %
/system/cpu/cpus/ControlPlane_01/usage  1      %
/system/cpu/cpus/FastPath_01/usage      3      %
/system/cpu/cpus/FastPath_02/usage      11     %
/system/cpu/cpus/FastPath_03/usage      5      %
/system/cpu/cpus/FastPath_04/usage      31     %
/system/cpu/cpus/SlowPath_01/usage      7      %
/system/cpu/cpus/SlowPath_02/usage      3      %
/system/cpu/cpus/SlowPath_03/usage      23     %
/system/cpu/cpus/SlowPath_04/usage      14     %
/system/cpu/fast_path_usage             50     %
/system/cpu/slow_path_usage             47     %

The above output comes from a hypothetical processor with four cores. For each core, the fast path and slow path usage is shown with the total for each path at the bottom. The Control Plane handles configuration management and logging functions. The Slow Path deals with the setup of new flows. The Fast Path deals with packet forwarding functions.

The memory command provides a snapshot of memory utilization.

System:/> memory
			
	"
	"
  Free:                                   121306 KB
  Total (In listing + free):              135892 KB

The bulk of the above memory usage output has been omitted except for the final two lines. These give a summary of overall memory utilization.

The top command provides details of CPU utilization by control processes only. An example with truncated output is shown below.

System:/> top

Load avg: 0.94, 0.51, 0.35
CPU usage: 24.8%  CPUs: 1
Name          Time   CPU(%)
------------  -----  ------
statd         14:04  6.3
mdpd          12:52  5.1
vsinit        07:17  2.7
		"
		"

The format of this CPU usage summary follows the convention of the same command in UNIX. The Load avg figures are for the last 1, 5 and 15 minutes respectively. This list is ordered by CPU usage, with the most active processes placed at the top.

Multiple Administration Logins

For example, if one user deletes a configuration object then another user who tries to change it will receive a message to say that the operation is invalid.

Logging Out from the CLI

System:/> exit

Changing the Command View

The currently enabled views can be listed by using the cmdview command with no parameters. For example:

System:/> cmdview
Command view(s) enabled: default

In normal operation the Default view is sufficient. However, some situations may require the use of commands that should be used cautiously and for that reason they are hidden in alternate command "views".

For example, to turn on the Advanced view, enter the command:

System:/> cmdview advanced

Entering the Help command will now show that an additional set of commands are available to the administrator in addition to the default set. These additional commands include, for example, ruledb which provides a technically detailed listing of the current IP rules.

Similarly, an extra set of diagnostic commands becomes available with the command:

System:/> cmdview debug

The Debug, Default and Advanced command views all become available if the parameter all is used:

System:/> cmdview all

The original, default command view is returned by using the following command:

System:/> cmdview default

The service Command View

For this reason, the service view is protected by an additional one time password (OTP) which must be requested from Clavister support.

To allow the administrator to easily store and execute sets of CLI commands, the Clavister NetShield Firewall provides a feature called CLI scripting. A CLI script is a predefined sequence of CLI commands which can be executed after they are saved to a file and the file is then uploaded to the Clavister NetShield Firewall.

The steps for creating a CLI script are as follows:

The CLI script command is the tool used for script management and execution. The complete syntax of the command is described in the cOS Stream CLI Reference Guide and specific examples of usage are detailed in the following sections. See also Section 2.1.3, The CLI in this manual.

	Note
	Uploaded CLI script files are not held in permanent memory and will disappear after system restarts.

All CLI Commands are Allowed in Scripts

Executing Scripts

System:/> script -run my_script.sgs

Script Variables

$1, $2, $3, $4......$n

The values substituted for these variable names are specified as a list at the end of the script -run command line. The number n in the variable name indicates the variable value's position in this list. $1 comes first, $2 comes second and so on.

	Note: The symbol $0 is reserved
	Notice that the name of the first variable is $1. The variable $0 is reserved and is always replaced before execution by the name of the script file itself.

For example, a script called my_script.sgs is to be executed. The IPv4 address 126.12.11.01 is to replace all occurrences of $1 in the script file and the string if1 address is to replace all occurrences of $2.

The file my_script.sgs contains the single CLI command line:

add Address IPAddress if1_ip Address=$1 Comments=$2

To run this script file after uploading to cOS Stream with the given substitutions, the CLI command would be:

> script -run my_script.sgs 126.12.11.01 "if1 address"

Notice how the "if1 address" parameter is enclosed in quotes so any spaces can be included.

When the script file runs, the replacement of the variables means that the command becomes:

add Address IPAddress if1_ip Address=126.12.11.01 Comments="if1 address"

Escaping Characters

Script Validation and Command Ordering

Error Handling

System:/> script -run my_script2.sgs -force

Script Output

System:/> script -run my_script2.sgs -verbose

Scripts Are Lost After Restart

Listing Scripts

System:/> script
		
Name            Size
--------------  ----
my_script.sgs   8
my_script2.sgs  10

System:/> script -show my_script.sgs

Creating Scripts Automatically

If an installation already has the objects configured that need to be copied, then running the script -create command on that installation provides a way to automatically create the required script file. This script file can then be downloaded to the local management workstation and then uploaded to and executed on other Clavister NetShield Firewalls to duplicate the objects.

For example, suppose the requirement is to create the same set of IPAddress objects on several Clavister NetShield Firewalls that already exist on a single firewall. The administrator would connect to the single firewall with the CLI and issue the command:

System:/> script -create Address IPAddress -filename=new_script.sgs

This creates a script file called new_script_sgs which contains all the CLI commands necessary to create all IPAddress objects in that firewall's configuration. The created file's contents might, for example, be:

add Address IPAddress if1_ip Address=10.6.60.10
add IPAddress if1_net Address=10.6.60.0/24
add IPAddress if1_br Address=10.6.60.255
add IPAddress if1_dns1 Address=141.1.1.1
		"
		"
		"

Note that with the -create option, the output file must be specified with the -filename= parameter. This is the only time that -filename= is used.

The file new_script_sgs can then be downloaded with SCP to the local management workstation computer and then uploaded and executed on other Clavister NetShield Firewalls. The end result is that all units will have the same IPAddress objects in their address book.

In order to preserve any script files created in this way between system restarts, they must downloaded to the local storage of the management workstation using SCP and then uploaded again later.

Note: Automatically created scripts omit the object category

In the created script example above, adding an IP address is done with the command: add IPAddress... This is instead of the usual way of qualifying the object with its group name: add Address IPAddress... Both are valid forms of the command. If an object type can be uniquely identified with its name, its object category need not be specified. With automatically generated scripts, this is always the case. This shortened form can also be used when typing the entire command in a CLI console although tab completion will always include the object category.

Objects Excluded from Script Files

These node types are skipped when the script file is created and cOS Stream gives the message No objects of selected category or type.

	Tip: Listing commands at the console
	To list the created CLI commands on the console instead of saving them to a file, leave out the -filename= parameter in the script -create command.

Commenting Script Files

# The following line defines the if1 IPv4 address
add Address IPAddress if1_ip Address=10.6.60.10

Scripts Running Other Scripts

		script -run my_script2.sgs

To upload and download files to or from the Clavister NetShield Firewall, the Secure Copy (SCP) protocol can be used. SCP is based on the SSH protocol and many freely available SCP clients exist for almost all platforms. The command line examples given below are based on the most common command format found with SCP client software. Some SCP clients might use different syntax.

SCP Command Format

Upload is performed with the command:

> scp <local_filename> <destination_firewall>

Download is done with the command:

> scp <source_firewall> <local_filename>

The source or destination Clavister NetShield Firewall is specified using the format:

		<user_name>@<firewall_ip_address>:<filepath>

For example:

		admin@10.62.11.10:config.bkp

The <user_name> must be a defined user belonging to the administrator user group.

	Note: SCP examples do not show the password prompt
	SCP will normally prompt for the user password after the command line but that prompt is not shown in the examples given in this document.

Examples of SCP Uploading and Downloading

Using Wildcards with SCP

 > scp admin@192.168.3.1:*.mib .

 > scp admin@192.168.3.1:* .

 > scp *.bkp admin@192.168.3.1:.

For system management tasks in the default configuration, the administrator logs in to the CLI via SSH using a username and password that are checked against the credentials stored in a local database.

An alternative to using a local database is to have cOS Stream perform authentication of the entered credentials against an external RADIUS server. This is done by changing the properties of the AuthenticationProfile object assigned to the RemoteManagementSSH object used for CLI access via SSH. The option exists to use either a local database or a RADIUS server as the authentication source, or a combination of both. If a combination is selected, the administrator can then select which source to try first.

	Caution: RADIUS is ignored with public key authentication
	If cOS Stream is able to use public key authentication when an SSH client connects then RADIUS authentication will not also be attempted even though it might be configured in an associated AuthenticationProfile object.

Configuration Steps

RADIUS Servers Must Be Correctly Configured

The Login Message Changes for a Group Mismatch

However, there is one exception. When the user credentials are correct but the group name returned by the RADIUS server does not match a valid group (it is not Administrators or Auditors) the CLI will display the following message:

	Access denied.

Generated Log Event Messages

The types of log event messages that cOS Stream might generate are as follows:

 AUTHSYS: prio=notice id=00000 event=user_logged_in
 userid=33 user=jdoe ip=192.168.1.1 profile=MgmtAuthProfile agent=Basic
 userdb=RadiusServer1 usergroups="Administrators" action=none

AUTHSYS: prio=notice id=00000 event=user_logged_in
 userid=32 user=admin ip=192.168.1.1 profile=MgmtAuthProfile agent=Basic
 userdb=AdminUsers usergroups="Administrators" action=none

 AUTHSYS: prio=warning id=00000 event=received_radius_access-reject_message
 userid=33 user=jdoe ip=192.168.1.1 profile=MgmtAuthProfile
 agent=Basic userdb=RadiusServer1 action=none

 AUTHSYS: prio=warning id=00000 event=authentication_source_did_not_respond
 userid=33 user=jdoe ip=192.168.1.1 profile=MgmtAuthProfile
 agent=Basic userdb=RadiusServer1 action=none

Use the Local Console as a Fallback Option

System:/> add AuthenticationProfile my_mgmt_auth_profile
			LocalUserDB=""
			RemoteServer=my_radius_auth1,my_radius_auth2
			RemoteLoadBalance=RoundRobin
			AuthOrder=LocalLast
			MultipleLogins=AllowOne

System:/> set RemoteManagement RemoteMgmtSSH RemoteMgmtSSH
			AuthProfile=my_mgmt_auth_profile

Overview

Supported SNMP Versions

This section discussed overall MIB file support and setting up version 1 or 2c access. SNMPv3 access is described in Section 2.1.8, SNMPv3 Polling.

Supported Request Operations

MIB Files

An SNMP client connecting to cOS Stream can use two different types of MIB files:

Downloading MIB Files

> scp admin@192.168.1.17:/*.mib .

The selected MIB file should be transferred to the hard disk of the workstation that will run the SNMP client so it can be imported by the client software. When the SNMP client runs, the MIB files are read and they tell the client which values can be queried.

MIB Entries

IfPktsTotCnt OBJECT-TYPE
    SYNTAX      Counter32
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION 
        "Total number of packets transmitted by the interface"
    ::= { clvIfStatsEntry 10 }

All statistical values that can be accessed by SNMP are listed in the separate Statistics Reference Guide which provides all available information for each. This includes the description of the values together with the MIB OID, MIB name and MIB type.

Setting Up Version 1 or 2c SNMP Access

To ensure that the correct RemoteMgmtSNMP object triggers, different objects should not use overlapping filters. Instead, each object should specify a unique source interface and network combination. In addition, the source interface and network filters of RemoteMgmtSNMP and RemoteMgmtSNMP3 objects should also not overlap.

In addition to the filter parameters above, each RemoteMgmtSNMP rule also needs to have these two properties configured:

Other optional properties for the RemoteMgmtSNMP rule can be found in the separate CLI Reference Guide.

The Community String

System:/> add RemoteManagement RemoteMgmtSNMP my_snmp
			SourceInterface=lan
			SourceNetwork=mgmt-net
			GetCommunity=my-community-string

IP Rules are not Required with SNMP

Security Concerns with Version 1 and 2c

However, SNMPv3 polling can provide both encryption and authentication and this is described in the next section.

cOS Stream supports SNMPv3 polling by an external SNMPv3 client with an IPv4 or IPv6 address. SNMPv3 differs from the earlier SNMP versions in that it can provide both encryption and/or authentication.

Setting Up SNMPv3 Polling

All the available properties are listed under the RemoteMgmtSNMP3 object entry in the separate CLI Reference Guide.

To ensure that the correct RemoteMgmtSNMP3 object triggers, different objects should not use overlapping filters. Instead, each object should specify a unique source interface and network combination. In addition, the source interface and network filters of RemoteMgmtSNMP and RemoteMgmtSNMP3 objects should also not overlap.

SecurityLevel Property Options

SNMPv3 Encryption

SNMPv3 Authentication

By default, the algorithm used for creating the hash of the password is HMAC-SHA1-96. This can be changed by setting the property AuthenticationMethod to the alternative value of HMAC-MD5-96. However, HMAC-SHA1-96 is the recommended setting for strong security.

Note that SNMPv3 polling with authentication enabled will cease to function if the reboot counter has reached its maximum value. The reboot counter and how it is reset is explained further in Section 20.5, SNMP Traps.

Multiple Clients Are Allowed

However, it is important that the SourceNetwork property for multiple RemoteMgmtSNMP3 objects do not have IP addresses assigned to them which overlap. If overlapping occurs, the behavior will be unpredictable.

System:/> add RemoteManagement RemoteMgmtSNMP3 my-snmp3-rule
			SourceInterface=if1
			SourceNetwork=if1_net
			LocalUserDB=AdminUsers
			PrivacyPassword=mysecretpassword
			RequestLimit=20

Correctly setting the date and time is important for cOS Stream to operate properly. For example, certificates used in certificate based VPN tunnels depend on the system clock being accurately set.

In addition, log messages are tagged with time-stamps in order to indicate when a specific event occurred. Not only does this assume a working clock, but also that the clock is correctly synchronized with other equipment in the network.

The administrator can set the date and time manually and this is recommended when a new system installation is started for the first time.

The Local System Time

The system time is set with the following CLI command:

System:/> time -set YYYY-mm-DD HH:MM:SS

Where YYYY-mm-DD HH:MM:SS is the new date and time. Note that the date order is year, then month and then day.

System:/> time -set 2015-04-27 09:25:00

	Note: A reconfigure is not required
	A new date and time will be applied by cOS Stream as soon as it is set. There is no requirement to reconfigure or restart the system.

Time Zones

The DateTime object property TimeZone must be set to a value which reflects the time zone where the Clavister NetShield Firewall is physically located.

System:/> set DateTime Timezone=UTC+01:00

Daylight Saving Time

There is no specific setting for enabling DST in cOS Stream. Instead, DST is implicitly enabled by selecting the appropriate value for the Timezone property. For example, UTC+01:00 is the value for UTC plus one hour. However, UTC+01:00-EU is the value for UTC plus one hour which includes daylight saving. Tab completion can be used to identify all available values in the CLI.

System:/> set DateTime Timezone=UTC+01:00-EU

	Note: Switching off DST
	Even though DST is implicitly enabled through the time zone value specified, it is still possible to turn it off by disabling the DSTAutoAdjust property of the Timezone object. The same property can be used to then re-enable it. This might be done, for example, for testing purposes.

The system clock which cOS Stream uses can sometimes become fast or slow after a period of operation. This is normal behavior in most network and computer equipment and is solved by utilizing Time Servers.

cOS Stream is able to adjust the clock automatically based on information received from one or more time servers which provide a highly accurate time, usually using atomic clocks. Using time servers is highly recommended as it ensures the system will have its date and time aligned with other network devices.

For hardware platforms without battery backup of the system clock, using time servers is a useful way to automatically set the correct time after initial startup.

The SNTP Protocol

Configuring Time Servers

System:/> cc DateTime

System:/DateTime> add TimeServer IP=10.5.4.36 Name=my_tsrv1

System:/DateTime> add TimeServer IP=10.5.4.76 Name=my_tsrv2

System:/DateTime> cc

Manually Triggering Time Synchronization

System:/> time -sync

Time synchronization requested

System:/> 

Maximum Time Adjustment

For example, assume that the maximum adjustment value is set to 60 seconds and the current system time is 16:42:35. If a Time Server responds with a time of 16:43:38 then the difference is 63 seconds. This is greater than the maximum adjustment value so no update occurs for this response. The default value for the maximum adjustment is 600 seconds (ten minutes).

System:/> set DateTime TimeSyncMaxAdjust=4000

Forcing Synchronization

System:/> time -sync -force

The Time Synchronization Interval

System:/> set DateTime TimeSyncInterval=3600

When troubleshooting networking problems, one of the most useful tools is the CLI ping command. This command can send one of more ICMP ECHO, TCP or UDP packets to a specified IPv4 or IPv6 address and reports the reply that is received.

A Simple ICMP Example

System:/> ping 10.4.0.2

System:/> ping 10.4.0.2 -routingtable=my-rtable

Specifying the Protocol and the Message Load

System:/> ping -udp 10.4.0.2

System:/> ping -tcp 203.0.113.10 -port=80 -request="GET /HTTP/1.0\n\n"

Sending to IPv6 Addresses

System:/> ping c1a1::1:2

Sending 4-byte ICMP ping to c1a1::1:2:61083 from c1a1::1:1:128
4-byte ICMP reply from c1a1::1:2:61083 to c1a1::1:1:129

The -verbose Option

System:/> ping 10.4.0.2 -v

The verbose options are required when performing ping traffic simulation (described next) in order to see the configuration elements that are triggered during the simulation.

Incoming Packet Simulation Using the -srciface and -srcip Option

For example, the following command would simulate an incoming ICMP ping message arriving on the if2 interface that was coming from the source IP address 192.168.0.10:

System:/> ping 10.6.58.10 -srciface=if2 -srcip=192.168.0.10 -verbose

This command will construct an ICMP packet with a destination IP of 10.6.58.10 and cOS Stream will behave as though that packet had arrived at the interface if2 and had come from the external IP address 192.168.0.10.

Note that the simulated traffic received in this way will be treated by the configuration as normal traffic. This means that such configuration elements as IP rule sets will be applied so any relevant NAT or SAT actions will also be applied. The traffic might even be dropped. Also note that the -routingtable option cannot be used with the -srciface option as it has no relevance in this context.

If there is no route that matches the combination of source IP address and receiving interface, the packet will be dropped. Some example output when this happens is listed below:

System:/> ping 192.168.14.1 -srciface=if14 -srcip=192.168.14.12 -verbose
Simulating 4-byte ICMP ping to 192.168.14.1:44434
  from (if14) 192.168.14.12:8
(Response is not displayed when '-srciface' is used)
SNOOP:  POLICY-1  : Forward lookup matched PBR rule "RoutingRule_1"
  (id 0x3039) against <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:44434>
SNOOP:  POLICY-1  : Drop <ICMP
  if14;192.168.14.12:2048 -> 192.168.14.1:44434>:
	No route to source in table "Main" (ID: 0x20).

For the ping message not to be dropped, there must not only be a route that matches the source IP address and source interface combination but also an IP rule set entry that allows the traffic on that interface. If the administrator simulates the packet arriving on the if2 interface and going to some host on the network if1_net, the allowing IP rule set entry might be the following:

If there is no IP rule set entry that allows the traffic it will also be dropped. The following example shows some example console output when this happens:

System:/> ping 192.168.14.1 -srciface=if14 -srcip=192.168.14.12 -verbose
Simulating 4-byte ICMP ping to 192.168.14.1:2524
  from (if14) 192.168.14.12:8
(Response is not displayed when '-srciface' is used)
SNOOP:  POLICY-1  : Forward lookup matched PBR rule "RoutingRule_1"
(id 0x3039) against <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:2524>
SNOOP:  POLICY-1  : <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:2524>:
  Found reverse route "Route_3" (ID: 0x3ead62f) 192.168.14.0/24 ->
  if14 (Gateway: none) in table "Main" (ID: 0x20).
SNOOP:  POLICY-1  : <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:2524>:
  Found forward route "Route_3" (ID: 0x3ead62f) 192.168.14.0/24 ->
  if14 (Gateway: none) in table "Main" (ID: 0x20).
SNOOP:  POLICY-1  : Matched default rule "System::DefaultDrop"
SNOOP:  POLICY-1  : Dropped by default rule "System::DefaultDrop"
SNOOP:  POLICY-1  : <ICMP if14;192.168.14.12:2048 ->
  192.168.14.1:2524>: Dropped by IP rules

Once both the routing problem and the IP rule problems that were indicated in the above two output examples are fixed, the following is an example of some problem-free output:

System:/> ping 192.168.14.1 -srciface=if14 -srcip=192.168.14.12 -verbose
Simulating 4-byte ICMP ping to 192.168.14.1:24093
  from (if14) 192.168.14.12:8
(Response is not displayed when '-srciface' is used)
SNOOP:  POLICY-1  : Forward lookup matched PBR rule "RoutingRule_1"
  (id 0x3039) against <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:24093>
SNOOP:  POLICY-1  : <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:24093>:
  Found reverse route "Route_3" (ID: 0x3ead62f) 192.168.14.0/24 -> if14
  (Gateway: none) in table "Main" (ID: 0x20).
SNOOP:  POLICY-1  : <ICMP if14;192.168.14.12:2048 -> 192.168.14.1:24093>:
  Found forward route "Route_3" (ID: 0x3ead62f) 192.168.14.0/24 -> if14
  (Gateway: none) in table "Main" (ID: 0x20).
SNOOP:  POLICY-1  : Matched rule "iprule_all_services" with id 0xdaa96f5
SNOOP:  POLICY-1  : Allow <ICMP if14;192.168.14.12:2048 ->
  192.168.14.1:24093>: route via if14

Specifying Only the -srcip Option

System:/> ping 10.4.0.2 -srcip=192.168.0.5

All ping command options are listed in the separate CLI Reference Guide.

During system operation, important troubleshooting information is saved in system memory by the Dconsole subsystem. This information takes the form of time stamped messages which record both normal events such as system restarts but also any abnormal system events. These messages can be retrieved at any time using the CLI command:

System:/> dcon

The output generated by the dcon command provides an important diagnostic tool that can help to determine the cause of a system problem. However, it should be remembered that interpreting the output is often best done by sending the information to qualified support personnel.

Message Content

Some typical output from the dcon command is shown below:

System:/> dcon
Showing diagnose entries since 2012-11-09:
2015-11-07 12:02:15     01:51:29        INFO    SYSTEM,CONFIG   vsinit  
Beginning reconfigure [type=Activating new config][reason=] [09ce5f4f]
2015-11-07 12:02:16     01:51:30        INFO    SYSTEM,CONFIG   vsinit
Reconfigure completed successfully [0b1a5f4f]

Here, the dcon command is used without any parameters and by default only messages stored for today are displayed. The first message retrieved indicates the time of a reconfigure. The second indicates the reconfigure has been successfully completed.

All the recent dconsole messages could have been displayed by using the command with some previous date specified:

System:/> dcon -show -date=2016-01-01

The Message Buffering Mechanism

When the volatile memory buffer is full, the newest message overwrites the oldest message. At certain times, the entire buffer is copied to a non-volatile memory file called dcon.bin and then the buffer is cleared. These copy operations occur when:

When the copy operation occurs, the messages in the volatile memory buffer are appended to the existing messages in the non-volatile file dcon.bin. After the copying is complete, the volatile memory buffer is automatically cleared and begins to fill up again with new messages.

Messages can be appended to the non-volatile file until the configured maximum size of the file is reached. When this happens, cOS Stream erases the oldest 50% of the messages to clear space for new appended messages. The newest 50% of messages in the file are preserved.

During system startup the non-volatile file is allowed to temporarily grow beyond its configured maximum to accommodate any messages generated. This deals with the case that the file may be close to its size limit at startup but vital troubleshooting information still needs to be retained if a problem occurs.

The Volatile Memory Buffer is Circular

2015-11-06 13:11:26 368:23:42 INFO LOG dcon 801 Messages got Overwritten

Manually Clearing All Messages

System:/> dcon -clean

Filtering Messages

For example, to show all events that have a severity of Critical or higher, use the command:

System:/> dcon -show -severity=Critical

To show all events that were generated after and including 1st June 2011, use the command:

System:/> dcon -show -date=2016-06-01

To show all events related to the system vsinit component, use the command:

System:/> dcon -show -app=vsinit

To show all events related to the system dataplane component, use the command:

System:/> dcon -show -app=dataplane

For a full list of dcon command options, refer to the separate Clavister NetShield Firewall CLI Reference Guide.

A valuable diagnostic tool is the ability to examine the packets that enter and leave the interfaces of a Clavister NetShield Firewall. For this purpose, cOS Stream provides the CLI command pcapdump which allows the recording of packets entering and leaving all or certain interfaces. The captured packets can be stored in a file and/or displayed on the console in an easily readable text form. Filters can also be applied so that only packets meeting specific criteria are recorded.

This section focuses on some of the important options for using the command. The complete syntax of the pcapdump CLI command is described in the cOS Stream CLI Reference Guide.

Important: Pcapdump requires sufficient control plane memory

The pcapdump command requires that the control plane free memory space to be at least 100 MBytes. If this is not available then the pcapdump command will come back with a message saying that it has insufficient memory. This problem should only occur in a virtual environment and to resolve it, the virtual machine should be stopped and the memory allocated to cOS Stream should be increased. One Gigabyte of RAM for the whole virtual machine should be a minimum. The CLI memory command can show the amount of available memory in the control plane.

The Command View Needs to be Changed

System:/> cmdview debug

A Simple Example

Using -snaplen to Reduce File Size

System:/> pcapdump -start if3 -snaplen=100

The -snaplen option is also useful in reducing the space needed in the pcapdump buffer by shortening the packet information stored. The allocated buffer size is discussed later in this section.

The Memory Buffer Records All Packets Stored

System:/> pcapdump -show

System:/> pcapdump -write

System:/> pcapdump -write if3 -filename=if3.cap

System:/> pcapdump -start -out -nocap

System:/> pcapdump -status
             PCAP Status

Interface Mode   Packets(In)  Packets(Out) Filter
--------- ------ -----------  ------------ ------
if3       SNOOP  124          174

System:/> pcapdump -remove

As mentioned earlier, the buffer is usually cleared after packet recording stops and files are automatically written. However, if file writing is disabled by using the -nowrite option, the buffer is not cleared and subsequent pcapdump -start commands will append packet information to the current buffer contents.

The administrator has the option to examine the buffer contents or write the buffer contents to file. To clear the buffer when using the -nowrite option, the administrator must use the pcapdump -remove command.

Changing the Allocated Memory Buffer Size

	     Packet capture stopped (buffer size limit reached).

System:/> pcapdump -start if3 -bufsize=256 

Writing Directly to the Console

System:/> pcapdump -start if3 -out

System:/> pcapdump -start if3 -out -nowrite

As discussed earlier, if the desire was to write to the console but only to the console and not allocate and fill a buffer, the command would be:

System:/> pcapdump -start if3 -out -nocap

Filtering Output

The range of filters available is large and is fully documented on the online help (type help pcapdump) as well as in the separate CLI Reference Guide. This section will show examples of some of the filters to illustrate their use.

	Caution: Pcapdump can degrade performance
	Running pcapdump will have some impact on overall system performance. The degree of impact depends on the options chosen. The impact can be reduced by capturing on the smallest number of interfaces possible and through using filters to reduce the amount of packets captured. Raising the values of the -bufsize and/or -count options too high can have a particularly adverse effect on performance.

Compatibility with Wireshark

The administrator must use SCP in order to download the .cap files created by pcapdump from cOS Stream disk storage to a local management workstation for reading by Wireshark.

Writing in Hex and K12 Format

System:/> pcapdump -start if3 -out -hex

System:/> pcapdump -start if3 -out -k12

cOS Stream can also perform a conversion from the current buffer contents in .cap format to hexadecimal or K12 format on the console. To display the current buffer contents in hexadecimal, the command would be:

System:/> pcapdump -show if3 -hex

Writing hexadecimal to the console provides an alternative way to get packet data into Wireshark or another tool. The hexadecimal output displayed in an SSH client console can be copy and pasted into a local text file on the management workstation and then that file can be read by Wireshark.

The techsupport CLI command is a general command for problem troubleshooting that combines the output from several other troubleshooting commands into a single console output. This information can then used by the administrator but is usually sent to qualified support personnel to troubleshoot problems.

The simplest form of the command is the following:

System:/> techsupport

This will create a file in the top level (root) directory of the system file system called techsupport.txt. Each usage of the command will overwrite any previous techsupport.txt file.

Once created, there are two ways of examining the file's contents:

The first few lines of the techsupport.text file include the system time when the contents were created. A typical example of this header is the following:

Technical Support Information
Please verify the contents before sending to Technical Support.
System Time: 2016-09-08 12:13:48 (UTC+00:00)

In contrast to System Hardware Monitoring, the feature called User Hardware Monitoring is used to generate log event messages for specific sensors and conditions that are defined by the administrator.

Enabling User Hardware Monitoring

Assume that monitoring of the CPU temperature is required. If the sensor name is CPU_Temp and the new object is to be called mon1, user monitoring for this sensor can be enabled with the following CLI command:

System:/> add HWMONMonitor mon1 SensorID=CPU_Temp

The HWMONMonitor object created will generate an event message when its sensor has a value that moves outside the thresholds defined for the object (in the case of temperatures and fan speeds) or when the status changes (in the case of a failed power supply).

Another log message is generated as the sensor value moves back to a normal value and becomes equal to the threshold value. In the above command, no minimum or maximum threshold values are specified so cOS Stream will use the default thresholds for this sensor. All sensors have default thresholds already defined. Note that the SensorID property must always be specified when creating an HWMONMonitor object.

By default, a severity of Warning is used for all log messages generated. The severity can be explicitly set by changing the Severity property of an HWMONMonitor object. For example, to set the severity to Emergency for the mon1 object, the CLI command would be:

System:/> set HWMONMonitor mon1 Severity=Emergency

Sensor Types

Understanding the thresholds of a numeric sensor is straightforward but binary sensors require some additional explanation. Binary sensors have a value which is either 1 or 0. However, it depends on the sensor which value indicates a "normal" condition. For example, one sensor could indicate that a PSU is installed and its normal value is 1 (the PSU is present). Another sensor might indicate that a PSU has failed and its normal value is 0 (the PSU is operational and has not failed).

The following should be noted about HWMONMonitor object thresholds for binary sensors:

Changing Threshold Values

System:/> set HWMONMonitor mon1 LowThresh=-5 HighThresh=100

	Important: Do not change thresholds of binary sensors
	The thresholds associated with a binary sensor, such as PSU failure, should not be changed by the administrator. The default threshold values should always be used.

Changing the Monitoring Interval for the HWMONMonitor Object

System:/> set HWMONMonitor mon1 Interval=30

Log Messages

The log messages generated by HWMONMonitor objects all belong to the HWMON log message category. As described previously, the severity of the message is Warning by default but this can be changed by setting the Severity property of the HWMONMonitor object generating the message. Below are some examples of the log messages created by user hardware monitoring.

A sensor value above a maximum threshold:

SYSTEM,HWMON: prio=warning id=1081
event=sensor_value_above_monitor_threshold sensorid=”CPU_Temp”
description="CPU Temperature" name="n2" value=81
threshold=80 action=none

A sensor value below a minimum threshold:

SYSTEM,HWMON: prio=warning id=1079
event=sensor_value_below_monitor_threshold sensorid=”CPU_Temp”
description="CPU Temperature" name="n2" value=29
threshold=30 action=none

A sensor value crossing a threshold back to a normal value:

SYSTEM,HWMON: prio=warning id=1082
event=sensor_returned_to_normal sensorid=”CPU_Temp”
description="CPU Temperature" name="n2" value=80
action=none

The Log Repeat Interval

The default log repeat interval value of 6 hours can be changed by assigning a new value to the LogRepeatInterval property of the HWMONMonitor object. The value is specified as an integer number of seconds. For example, the following command would change the log repeat interval for the monitor called mon1 to become 12 hours:

System:/> Set HWMONMOnitor mon1 LogRepeatInterval=43200

Note that the minimum allowed value for the LogRepeatInterval property is 30 seconds and the maximum is 86,400 seconds (24 hours).

Displaying the Available Sensors

System:/> hwmon -sensorlist
		
Sensor ID       Description               Unit  Value  Monitor Min    Max
--------------  ------------------------  ----  -----  ------- -----  -----
CPU_Temp        CPU Temperature           C     58     yes     0      98    
System_Temp     System Temperature        C     24     no      0      55    
System_Power    System Power Consumption  W     224    no      0      500   
System_12V      System Internal Power     mV    12126  no      11500  13000 
FAN1_RPM        System FAN1 Speed         RPM   4200   no      1500   6800   
FAN2_RPM        System FAN2 Speed         RPM   4000   no      1500   6800  
FAN3_RPM        System FAN3 Speed         RPM   4100   no      1500   6800  
PSU1_Avail      PSU1 Available            -     1      no      1      1     
PSU1_Fail       PSU1 Failure Detected     -     0      no      0      0    
PSU1_Input_Lost PSU1 Power Input Lost     -     0      no      0      0     
PSU2_Avail      PSU2 Available            -     1      no      1      1   
PSU2_Fail       PSU2 Failure Detected     -     0      no      0      0   
PSU2_Input_Lost PSU2 Power Input Lost     -     0      no      0      0

The following should be noted about the above output:

Displaying the Current HWMONMonitor List

System:/> hwmon
		
Name  Sensor    Description       Value  Low  High  Status  #Low  #High
----  --------  ----------------  -----  ---  ----  ------  ----  -----
mon1  CPU_Temp  CPU Temperature   59     0    98    NORMAL  0     0
mon2  CPU_Temp  CPU Temperature   59     20   70    NORMAL  0     0

Note that in the above list, there are two HWMONMonitor objects defined for the same sensor. This is permissible and means that separate log messages can be generated for different sensor ranges.

The Status column in the above output indicates the status of the sensor value when it was last examined by the HWMONMonitor object and can show the following values:

Displaying HWMONMonitor Properties

System:/> hwmon mon1
Name                       : mon1
Sensor id                  : CPU_Temp
Log severity setting       : warning
Description                : CPU Temperature
Value                      : 58
Low Threshold              : 0
Recommended low threshold  : 0
High Threshold             : 98
Recommended high threshold : 98
Action interval (seconds)  : 5
ALARM:LOW counter          : 1
ALARM:HIGH counter         : 3
Last status                : NORMAL

Note that the Recommended low threshold and the Recommended high threshold in the above output are the default threshold values for the associated sensor.

Sensor Alarms

The following should be noted about the alarm counters:

The sensor information that is gathered by cOS Stream is available to an SNMP client using the SNMP protocol. This type of monitoring is known as System Monitoring.

System monitoring only has one option available that can be set by the administrator, and that is if it is enabled or disabled. All hardware monitoring is enabled by default, but this can be disabled by the following CLI command:

System:/> set Settings HWMONSettings MonitorEnable=No

This will disable both system and user monitoring. Both can be re-enabled with the following command:

System:/> set Settings HWMONSettings MonitorEnable=Yes

This setting affects both system monitoring and user monitoring

The following are the available sensors for the current range of Clavister hardware products running cOS Stream. All fan speeds are given in RPM and temperatures are in degrees centigrade.