Chapter 18: Log Servers

As well as log messages sent by external nodes, the InCenter system itself generates log messages as various events occur. All these log messages can be sent to the following types of Syslog receivers:

Note that when the InCenter software is upgraded, internal log files will not be preserved. For this reason, configuring an external log server can provide the advantage of preserving old log information.

Configuring External Nodes

For an external node to send log messages to InCenter, a LogReceiverSyslog object must be added in the node configuration in the normal way. For example:

Device:/> add LogReceiverSyslog Name=system_log_server
		IPAddress=203.0.113.5

Note that the IP address specified for the server is the same IP address used for InCenter management access.

18.1. The Internal Log Server

The InCenter internal log server is available as a standard component of the InCenter installation. The internal log server will store events from both InCenter itself as well as any external nodes that are configured to send Syslog messages to InCenter.

All log messages generated by InCenter itself will be automatically stored by the configured log server. The source IP address for these events will always be the loopback address 127.0.0.1.

Getting the Current Log Server Status

The log -status command will show the current server status:

admin@InCenter:/> log -status

Querying the Server Database

The log -query command is used to query the server database. The most basic form of the command has no additional parameters:

admin@InCenter:/> log -query
Time                 Source     Severity  Category  Message
-------------------  ---------  --------  --------  --------
2017-10-21 12:14:08  127.0.0.1  info      CONFIG    commit_started
user=admin ip=127.0.0.1 port=8000
Showing 1 out of 1.

The above output shows a single log message (wrapped to fit on the page) to illustrate the message format. The Source column shows the IP address of the node or the IP address 127.0.0.1 if the message is from InCenter itself.

In practice, too many messages may be displayed and the number can be limited with the -num option:

admin@InCenter:/> log -query -num=10

It is also possible to display log messages without any column formatting using the -no-table option. The following is an example with some typical output:

admin@InCenter:/> log -query -num=1 -no-table
2017-06-08 16:41:02.321 192.168.111.11 ARP,VALIDATE:
prio=warning id=00240 event=disallowed_by_access_rule
recviface=if2 srchw=10:00:00:02:10:00 srcip=127.0.0.3
destip=127.0.0.3 pkt_rec vif=if2
pkt_srchw=10:00:00:02:10:00 pkt_enetproto=ARP
rule=System_Block127Net action=drop logtrace=066089df

Filtering Queries

Filtering is possible to display only the log messages that match a set of criteria. For example, the source IP could be the criteria:

admin@InCenter:/> log -query -num=10 source=192.168.111.11

The criteria could be the log message severity:

admin@InCenter:/> log -query severity=warning

Alternatively the criteria could be the log message severity category. There can be more than one category associated with a log message but only one needs to match:

admin@InCenter:/> log -query category=ARP
Time                 Source          Severity  Category
-------------------  --------------  --------  ------------ 
2017-06-08 16:41:02  192.168.111.11  warning   ARP,VALIDATE
prio=warning id=00240 event=disallowed_by_access_rule
recviface=if2 srchw=10:00:00:02:10:00 srcip=127.0.0.3
destip=127.0.0.3 pkt_rec

Multiple criteria can be combined:

admin@InCenter:/> log -query severity=warning category=ARP

Pattern Filtering

A powerful method of filtering log messages is to use free text. This can be combined with the following wildcards:

  • * - Asterisk means any combination of characters.
  • ? - Question mark means any single character.

The following example will find all log messages that contain the word "warning" followed by any number of characters before the text "ip=10.11.12.?00", where the ? represents any character:

admin@InCenter:/> log -query
			-pattern="warning *ip=10.11.12.?00"

Log Server Settings

All log servers have the following adjustable settings:

  • LogServerPort - Listening port number of the log server.

  • LogDatabaseCleanupEnabled - This enables clean up of the log message database. The cleanup process is controlled using the other settings in this list. Default value: Yes.

  • MaximumLogDatabaseSize - Maximum size of the database in megabytes. If enabled, the oldest log messages are deleted if the total index size exceeds this amount. Default value: 10,000.

  • MaximumLogEventsAge - Delete log messages if they are older than this many days. Setting this to the minimum value of 1 will mean only the current day's logs are retained. Default value: 365.

  • MaximumAggregatedDatabaseSize - Aggregation indices will be deleted until the total disk used by them is lower than this limit, starting with the oldest index. Default value: 10,000.

  • MaximumAggregatedDataAge - Aggregation indices older than this age will be deleted. Default value: 365.

  • RateLimitBurst - Total number of logs allowed to be received over the time period specified by RateLimitInterval.

  • RateLimitInterval - The interval during which rate limiting counts logs and rejects logs once the number exceeds RateLimitBurst. The average number of logs that the log server accepts each second is the RateLimitBurst divided by the RateLimitInterval. For the default system this number is 1000/5 = 200 logs/second.

It should be noted that the automatic cleanup process runs once every hour.

Displaying the Current Server Properties

The command show LogServer in the Settings CLI context will display the current values of above settings for the currently configured log server. Below is example output for the internal server:

admin@InCenter:/> cc Settings 
admin@InCenter:/Settings> show LogServer
                      Property  Value
------------------------------  --------
                      Enabled:  Yes
                     Location:  Internal
    LogDatabaseCleanupEnabled:  Yes
              LogReceiverPort:  514
     MaximumAggregatedDataAge:  365
MaximumAggregatedDatabaseSize:  10000
       MaximumLogDatabaseSize:  10000
          MaximumLogEventsAge:  365
               RateLimitBurst:  1000
            RateLimitInterval:  5