6.3. Email Control

Home Prev	cOS Core 15.00.02 Administration Guide	Next

Email traffic can be a major concern for a system administrator, both because of its volume and because of the security threats it can carry. Unsolicited email, often referred to as Spam, is sent out by groups known as spammers in massive quantities and can waste resources, transport malware, as well as try to direct the recipient to webpages that might present further security risks.

The cOS Core anti-spam feature can be set up of the following ways:

IP Policy based Email Filtering for IMAP, POP3 and SMTP

Email filtering can be enabled on an IP Policy which targets the email traffic and this is the recommended method for setup. It makes available all the email filtering tools available in cOS Core. This includes a comprehensive anti-spam capability.

This type of email filtering is described in Section 6.3.1, Email Control Profiles with IP Policies.
IP Rule Based Email Filtering for SMTP

Email filtering using IP rules provides email filtering through an SMTP ALG object. Email filtering using IP rules can only be performed on SMTP traffic and is normally only used for compatibility with older cOS Core versions.

This type of email filtering is described in Section 6.3.3, SMTP Anti-Spam with IP Rules.

In both of the above ways of configuring email filtering, DNSBL servers might be used and DNSBL server usage with cOS Core is described generally for both methods in Section 6.3.2, DNSBL Processing.

6.3.1. Email Control Profiles with IP Policies

With IP policies, email filtering can be applied to IMAP, POP3 and SMTP traffic using IP policies. This is done by creating an Email Control Profile and associating it with an IP Policy object. With IMAP and POP3 filtering, emails cannot be dropped when they fail filtering but only marked as failed. With SMTP, emails can be dropped or forwarded.

Setting Up Email Filtering with IP Policies

IP policy based email filtering is set up with the following steps:

Create an Email Control Profile object which defines how email is to be filtered. If anti-spam filtering is required, it must be explicitly enabled in the profile (by default, it is disabled).
Optionally add one or more Email Filter objects as children to the Email Control Profile object. Each will specify an email address (or addresses using wildcards) which are to be blacklisted (automatically rejected before filtering) or whitelisted (never subject to filtering).
Associate the Email Control Profile object created above with an IP Policy object which triggers on the email traffic. Only a single profile can be associated with an IP policy.
The Service property for this IP policy must trigger on the IMAP, POP3 or SMTP protocols so it must be set to an appropriate Service object. The Service object used must have its Protocol property set to IMAP, POP3 or SMTP (whichever applies).

The predefined IMAP, POP3 and SMTP services could be used by setting their Protocol property to be IMAP or POP3 or SMTP. However, it is recommended to instead create a new custom Service object and this is done in the setup example found at the end of this section.
Optionally enable anti-virus scanning on the IP policy. This will scan any email attachments for viruses and will function with the IMAP, POP3 or SMTP protocol. Anti-virus scanning is discussed further in Section 6.4, Anti-Virus Scanning.

	Note: A service group cannot be used for email filtering
	It is not possible to use a custom Service Group object for IP policy based email filtering which combines any of the IMAP, POP3 or SMTP protocols. Separate IP policies with separate services for IMAP, POP3 or SMTP must be configured.

General Email Control Profile Settings

The following Email Control Profile options are available for all types of traffic.

Anti-Spam

When enabled, anti-spam scoring is applied to emails to determine if it could be spam. The anti-spam feature is described further later in this section.

By default, this option is disabled.
Blacklist Subject Tag

This is the text inserted into the subject field of an email if it is found on the blacklist for this profile.

Protocol Specific Options

The following additional options are available for when the profile is used with specific email related protocols:

SMTP Settings

These settings relate to processing by the SMTP ALG and are described further in Section 6.1.5, SMTP ALG.
POP3 Settings

These settings relate to processing by the POP3 ALG and are described further in Section 6.1.6, POP3 ALG.
IMAP Settings

These settings relate to processing by the IMAP ALG and are described further in Section 6.1.7, IMAP ALG.

Encryption and Allowing STARTTLS

By default, an Email Control Profile object will drop encrypted traffic for SMTP, POP3 or IMAP since none of the profile's checks can be applied when encryption is used.

If the profile property Allow STARTTLS is enabled, encrypted SMTP, POP3 or IMAP traffic will be allowed, provided that the encryption results from a STARTTLS command to change over from plain text. However, none of the profile's checks will be applied to this encrypted traffic.

Email Address Whitelist/Blacklist

One or more Email Filter objects can be added as children to an Email Control Profile object. Each filter specifies one address or, using wildcards, a series of addresses that are to be always dropped (blacklisted) or always allowed (whitelisted).

The wildcard asterisk ("*") character can be used to specify any string. For example, the string *.@example.com represents any email address from the example.com domain. The question mark ("?") character can also be used to represent any single character.

Whitelisting and blacklisting takes precedence over all other email filtering. Whitelisted emails will never be subject to anti-spam or anti-virus processing if either of those is enabled.

If an email comes from a blacklisted domain, the mail is not dropped. Instead, the subject line has a configurable text string inserted at the beginning. By default, this string is:

			*** BLACK LISTED ***

However, this text can be set by the user, as previously described.

Note that some IMAP clients may download and display the headers of emails before they download the email body. This can mean that the blacklist text may not be displayed correctly in the subject line with such clients. This is discussed further in the later section on anti-spam since the same problem can exist with flagging spam in the email subject line.

Anti-Spam and Filter Scoring

The anti-spam feature of email filtering using IP policies must be explicitly enabled in the Email Control Profile object used with an IP policy. Once enabled, a score is calculated by adding together the sub-scores of the enabled anti-spam filters. The score is used in the following ways:

If the total score is equal to or greater than the specified Tagging Threshold (the default value is 10) then an email is considered to be SPAM and tagged as such.
For SMTP only, if the total score is equal to or greater than the specified Reject Threshold (the default value is 20) then an email is dropped.

Each anti-spam filter can be enabled or disabled and they are comprised of the following list:

Domain Verification

When enabled, cOS Core will perform a DNS MX query, requesting the IP address of a mail server associated with the email sender's domain. For example, the sender email address might be some.name@example.com so cOS Core will send an MX query to the configured DNS server for the mail server at example.com.

If the DNS server recognizes the domain name, cOS Core takes no action. If the DNS server does not recognize the domain, cOS Core will add the sub-score for this filtering option to the overall anti-spam score.

Note that at least one DNS server must be configured in cOS Core for this option to work. If no DNS server is configured, this test will not be performed and its sub-score will not be added. Configuring DNS servers is described in Section 3.10, DNS.

This filtering option is enabled by default.

Malicious Link Protection

This option neutralizes undesirable or malicious HTML links inside emails. It is enabled by default and is discussed later in this section in more detail. Finding a single triggering link will add this option's sub-score (a default value of 10) to the total score. Malicious link protection is described in depth later in this section.

DCC

The Distributed Checksum Clearinghouses (DCC) method of filtering involves cOS Core sending anonymous checksums that identify an email to an external Clearing House server. The server returns a value to indicate how many other email recipients have reported identical checksums. If the returned value is greater than the DCC Threshold property set by the administrator, the sub-score for this filter option is added to the total anti-spam score.

DCC filtering is enabled by default and the default sub-score is 10. The administrator does not need to define any DCC servers because cOS Core uses Clavister's own server network for this function.

The DCC option is a subscription based feature and it will only function if the current date is prior to the DCC subscription date specified in the cOS Core license. If the DCC feature is enabled but the license does not allow it, the behavior is described in Appendix A, Subscription Based Features under the heading Subscription Expiry Behavior.

The DCC feature has its own parameter and expiry date in a Clavister license file. For this reason, if cOS Core is upgraded from a version that does not have DCC (prior to version 11.00) to a version that does (11.00 or later), a new cOS Core license with DCC parameter must be created and installed.

DNS Blacklists

By enabling the DNS Blacklists option, up to 10 different DNS Blacklist (DNBL) servers can be specified, each with its own sub-score (the default value is 10 per server) that will be added to the total score if that server flags an email as spam.

If a DNSBL server is not responding then its sub-score is not included in the final score calculation. How DNSBL functions is explained further in Section 6.3.2, DNSBL Processing. For an example of configuring DNSBL anti-spam filtering, see Example 6.10, “IMAP ALG Setup”.

Email Tagging

If an email score exceeds the Tagging Threshold property, it is marked as being SPAM. This is done in either or both of two ways:

Tag Subject

This is enabled by default and adds text to the subject line of an email. The default text is "*** SPAM ***" but this can be set to any string.

An example of this kind of tagging is if the original Subject field is:
```
 Buy this stock today!
```
If the tag text is defined to be "*** Probably SPAM ***", then the modified email's Subject field will become:
```
 *** Probably SPAM *** Buy this stock today!
```
The line above is what the email's recipient will see in the summary of their inbox contents. The individual user could then decide to set up their own filters in the local client to deal with such tagged emails, possibly sending it to a separate folder.
Tag Header

This is also enabled by default and it inserts X-Spam information into the bottom of the email header data which describes the anti-spam processing that has been performed on the email. When enabled, this indicates if the email is considered spam or not. This information may not be immediately visible to the recipient but many email clients will provide the option to view it and many will also allow it to be part of the client's filtering criteria.

The X-Spam fields inserted by cOS Core are as follows:
1. X-Spam-Checker-Version - The software that tagged the email.
2. X-Spam-Status - A list that includes the scoring and the filters applied.
3. X-Spam-Flag - Included only for emails marked as spam and always Yes.
4. X-Spam-Report - A detailed list of the results from applied filters.
An example of inserted X-SPAM information for an email that was flagged as spam is the following:
```
X-Spam-Checker-Version: Clavister cOS Core on Device
X-Spam-Status:
     Yes, score=30 required=10 tests=LINK_PROTECTION,DCC,DNS_BLACKLIST_1
X-Spam-Flag: Yes
X-Spam-Report:
* 10 LINK_PROTECTION: Contained undesirable web links
* 10 DCC: Checksum reported >= 16777200 times
* 10 DNS_BLACKLIST_1: Blacklisted source IP address
```
An example of inserted X-SPAM information for an email that was not flagged as spam is the following:
```
X-Spam-Checker-Version: Clavister cOS Core on Device
X-Spam-Status: No, score=5 required=10 tests=DNS_BLACKLIST_1
```

Malicious Link Protection

The Malicious Link Protection filter for anti-spam allows undesirable links in email traffic to be neutralized as the email passes through cOS Core. It is part of anti-spam filtering and can only be enabled when anti-spam is enabled. Links within emails are evaluated by cOS Core using the Web Content Filtering subsystem which is described in Section 6.2, Web Content Filtering.

The following points should be noted for the link protection:

Link protection will only examine the first 10 links in an email. After that no more link processing is performed for subsequent links.
When a link triggers web content filtering, the link will still appear in the mail but will be disabled so it cannot be clicked.
If one or more links trigger the specified link protection score (see properties below) will be added just once for an email to the overall anti-spam score.

The following properties of an Email Control Profile are used to control link protection:

Link Protection

By default, link protection is enabled if anti-spam is enabled. Disable this property to switch it off for all emails.
Link Protection Score

This is the score that is added to the total anti-spam score if just one link is found that triggers web content filtering. The default value is 10.
Link Protection Categories

A subset of the list of all web content filtering categories can be defined and this subset will be disallowed. By default, this is set to a limited subset. All available categories are described in Section 6.2, Web Content Filtering.
Non-Managed Action

This determines what to do with a link that could not be determined to belong to one of the web content filtering categories. The choice is either to allow such links (the default) or to disallow them.

	Note: A valid web content filtering subscription is required
	The malicious link protection filter will only function if cOS Core also has a valid subscription for the web content filtering feature since this is used to evaluate links. This is described further in Appendix A, Subscription Based Features.

IMAP Clients May Display Incorrect Header Information

Email clients using IMAP to retrieve email details from an email server, can download and display the headers of emails before they download the email body. This means the user can only download the body of emails they want to read based on the header information.

Since cOS Core may perform some of its spam scoring based on the email body, the initial header information displayed by the client may not yet correctly show the spam text in the subject line. When the email body is eventually downloaded, cOS Core examines it and the header information will be updated if the mail is now considered to be spam. However, some email clients may not update the displayed header information and will continue to display the original unflagged header information. This is not something that can be controlled by cOS Core.

In all cases, the spam related log messages generated by cOS Core, and the spam related statistics it keeps, will correctly reflect which emails it has classified as spam.

As mentioned previously, a similar problem can arise with blacklisting and IMAP clients. The initial header information displayed by the client may not correctly show the blacklist text.

6.3.2. DNSBL Processing

DNSBL servers can be used with both IP policy based anti-spam and also with SMTP ALG based anti-spam. This section explains the way they function and how they are used with cOS Core anti-spam and is applicable to both DNSBL usage with email control profiles and IP policies as well as to the SMTP ALG being used with IP rules. Further explanation of DNSBL usage with IP rules can be found in Section 6.3.2, DNSBL Processing.

Public DNSBL Databases

A number of trusted organizations maintain publicly available databases of the origin IP address of known spamming SMTP servers and these can be queried over the Internet. These lists are known as DNS Black List (DNSBL) databases and the information is accessible using a standardized query method supported by cOS Core.

DNSBL Server Usage By cOS Core

When the cOS Core anti-spam filtering feature is configured, the IP address of the email's sending server is sent to one or more DNSBL servers to find out if any DNSBL servers think the email is from a spammer or not. cOS Core examines the IP packet headers to do this. The diagram below illustrates cOS Core's interaction with DNSBL servers.

Figure 6.9. DNSBL Anti-Spam Processing

The reply sent back by a DNSBL server is either a not listed response or a listed response. In the latter case of being listed, the DNSBL server is indicating the email might be spam and it will usually also provide information known as a TXT record which is a textual explanation for the listing.

Note that the TXT record is not used by IP policy based anti-spam. If anti-spam is configured with IP rules using an SMTP ALG object then the record is part of the X-SPAM information which is optionally inserted into the body of an email.

	Tip: Finding a list of available DNSBL servers
	A list of public DNSBL servers can be found at: https://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists.

Creating a DNSBL Consensus

Multiple DNSBL servers can be specified in an Email Control Profile object which is then assigned to an IP policy that triggers on the targeted traffic. A weight is also assigned to each DNSBL server and this must be an integer greater than zero. A weighted sum can then be calculated based on all server responses and the following thresholds for the sum can be specified in the email control profile:

Tag Threshold

If the sum is greater than or equal to this value then the email is considered as probably being spam but forwarded to the recipient with notifying text inserted into it.
Reject Threshold

If the sum is greater than or equal to this value then the email is considered to be definitely spam and is discarded or alternatively sent to a single, special mailbox. This value will only apply to SMTP traffic.

If it is discarded then the administrator has the option that an error message is sent back to the sending SMTP server (this error message is similar to the one used with blacklisting).

A DNSBL Threshold Calculation Example

As an example, suppose that three DNSBL servers are configured: dnsbl1, dnsbl2 and dnsbl3. Weights of 3, 2 and 2 are assigned to these respectively. The spam threshold is then set to be 5.

If dnsbl1 and dnsbl2 say an email is spam but dnsbl3 does not, then the total calculated will be 3+2+0=5. Since the total of 5 is equal to (or greater than) the threshold then the email will be treated as spam.

If the Drop threshold in this example is set at 7 then all three DNSBL servers would have to respond in order for the calculated sum to cause the email to be dropped (3+2+2=7).

Allowing for Failed DNSBL Servers

If a query to a DNSBL server times out then cOS Core will consider that the query has failed and the weight given to that server will be automatically subtracted from both the spam and drop thresholds for the scoring calculation done for that email.

If enough DNSBL servers do not respond then this subtraction could mean that the threshold values become negative. Since the scoring calculation will always produce a value of zero or greater (servers cannot have negative weights) then all email will be allowed through if both the Spam and Drop thresholds become negative.

A log message is generated whenever a configured DNSBL server does not respond within the required time. This is done only once at the beginning of a consecutive sequence of response failures from a single server to avoid unnecessarily repeating the message.

DNSBL Server Query Logging

The following types of log messages are generated by cOS Core as a result of DNSBL server queries:

Logging of dropped or spam tagged emails - These log messages include the source email address and IP as well as its weighted points score and which DNSBLs caused the event.
DNSBLs not responding - DNSBL query timeouts are logged.
All defined DNSBLs stop responding - This is a high severity event since all email will be allowed through if this happens.

6.3.3. SMTP Anti-Spam with IP Rules

This section describes anti-spam setup using an SMTP ALG object configured using IP rules.

Anti-Spam Setup with IP Rules

To set up spam filtering in the SMTP ALG, the following list summarizes the steps:

Create a new SMTP ALG object.
Specify the DNSBL servers that are to be used. There can be one or multiple. Multiple servers can act both as backups to each other as well as confirmation of a sender's status.
Specify a weight for each server which will determine how important it is in deciding if email is spam or not in the calculation of a weighted sum.
Specify the thresholds for designating any email as spam. If the weighted sum is equal or greater than these then an email will be considered to be spam. Two thresholds are specified:
1. Spam Threshold - The threshold for tagging mail as spam.
2. Drop Threshold - The threshold for dropping mail.
The Spam Threshold should be less than the Drop Threshold. If the two are equal then only the Drop Threshold applies.
Specify a textual tag to prefix to the Subject field of email designated as spam.
Optionally specify an email address to which dropped email will be sent (as an alternative to simply discarding it). Optionally specify that the TXT messages sent by the DNSBL servers that failed are inserted into the header of these emails.

Creating a DNSBL Consensus

The administrator can configure the cOS Core SMTP ALG to consult multiple DNSBL servers in order to form a consensus opinion on an email's origin address. For each new email, configured servers are queried to assess the likelihood that the email is spam, based on its origin address.

With the SNMP ALG, the administrator assigns a weight greater than zero to each configured DNSBL server so that a weighted sum can then be calculated based on all responses. The administrator can then configure one of the following actions based on the weighted sum calculated:

Dropped

If the sum is greater than or equal to a predefined Drop threshold then the email is considered to be definitely spam and is discarded or alternatively sent to a single, special mailbox.

If it is discarded then the administrator has the option that an error message is sent back to the sending SMTP server (this error message is similar to the one used with blacklisting).
Flagged as Spam

If the sum is greater than or equal to a predefined Spam Threshold then the email is considered as probably being spam but forwarded to the recipient with notifying text inserted into it.

Alternative Actions for Dropped Spam

If the calculated sum is greater than or equal to the Drop threshold value then the email is not forwarded to the intended recipient. Instead the administrator can choose one of two alternatives for dropped email:

A special email address can be configured to receive all dropped email. If this is done then any TXT messages sent by the DNSBL servers (described next) that identified the email as spam can be optionally inserted by cOS Core into the header of the forwarded email.
If no receiver email address is configured for dropped emails then they are discarded by cOS Core. The administrator can specify that an error message is sent back to the sender address along with the TXT messages from the DNSBL servers that failed the email.

Tagging Spam

If an email is considered to be probably spam because the calculated sum is above the spam threshold but it is below the drop threshold, then the Subject field of the email is changed and prefixed with a message and the email is forwarded on to the intended recipient. The tag message text is specified by the administrator but can be left blank (although that is not recommended).

An example of tagging might be if the original Subject field is:

Buy this stock today!

And if the tag text is defined to be "*** SPAM ***", then the modified email's Subject field would become:

*** SPAM *** Buy this stock today!

And this is what the email's recipient will see in the summary of their inbox contents. The individual user could then decide to set up their own filters in the local client to deal with such tagged emails, possibly sending it to a separate folder.

Adding X-Spam Information

If an email is determined to be spam and a forwarding address is configured for dropped emails, then the administrator has the option to Add TXT Records to the email. A TXT Record is the information sent back from the DNSBL server when the server thinks the sender is a source of spam. This information can be inserted into the header of the email using the X-Spam tagging convention before it is sent on. The X-Spam fields added are:

X-Spam-Flag - This value will always be Yes.
X-Spam-Checker-Version - The software that tagged the email.
X-Spam-Status - This will always be DNSBL.
X-Spam-Report - A list of the DNSBL servers that flagged the email as spam.
X-Spam-TXT-Records - A list of TXT records sent by the DNSBL servers that identified the email as spam.
X-Spam_Sender-IP - IP address used by the email sender.

These fields can be referred to in filtering rules set up by the administrator in mail server software.

Verifying the Sender Email

As part of the anti-spam module, the option exists to check for a mismatch of the "From" address in the SMTP protocol command with the actual email header "From" address. Spammers can deliberately make these different to get email past filters so this feature provides an extra check on email integrity.

If a mismatch is detected, one of two actions can be configured:

The email is dropped.
Allow the email to pass but tag it using the configured spam tag.

When sender address verification is enabled, there is an additional option to only compare the domain names in the "From" addresses.

Real-time Monitoring

The following values for spam filtering can be monitored in real time using the real-time monitoring functionality of InControl.

For the DNSBL subsystem overall:

Number of emails checked.
Number of emails spam tagged.
Number of dropped emails.

For each DNSBL server accessed:

Number of positive (is spam) responses from each configured DNSBL server.
Number of queries sent to each configured DNSBL server.
Number of failed queries (without replies) for each configured DNSBL server.

The dnsbl CLI Command

The dnsbl CLI command provides a means to control and monitor the operation of the anti-spam subsystem when the SMTP ALG is set up with IP rules (it does not track DNSBL usage with IP policies).

The dnsbl command on its own without options shows the overall status of all ALGs. If the name of the SMTP ALG object on which DNSBL spam filtering is enabled is my_smtp_alg then the output would be:

Device:/> dnsbl
	
DNSBL Contexts:

Name                      Status      Spam     Drop    Accept  
------------------------  -------- -------- -------- --------
my_smtp_alg                 active      156       65    34299
alt_smtp_alg              inactive        0        0        0

The -show option provides a summary of the spam filtering operation of a specific ALG. It is used below to examine activity for my_smtp_alg although in this case, the ALG object has not yet processed any emails.

Device:/> dnsbl my_smtp_alg -show
 
Drop Threshold        : 20
Spam Threshold        : 10
Use TXT records       : yes
IP Cache disabled
Configured BlackLists : 4
Disabled BlackLists   : 0
Current Sessions      : 0
Statistics:
Total number of mails checked : 0
Number of mails dropped       : 0
Number of mails spam tagged   : 0
Number of mails accepted      : 0
BlackList                 Status   Value Total    Matches  Failed  
------------------------- -------- ----- -------- -------- --------
b.barracudacentral.org    active      25        0        0        0 
cbl.abuseat.org           active      20        0        0        0 
dnsbl.sorbs.net           active       5        0        0        0 
asdf.egrhb.net            active       5        0        0        0

To examine the statistics for a particular DNSBL server, the following command can be used.

Device:/> dnsbl smtp_test b.barracudacentral.org -show

BlackList: b.barracudacentral.org
Status       : active
Weight value : 25
Number of mails checked                     : 56
Number of matches in list                   : 3
Number of failed checks (times disabled)    : 0

To clean out the dnsbl cache for my_smtp_alg and to reset all its statistical counters, the following command option can be used:

Device:/> dnsbl my_smtp_alg -clean

Email Addresses are Cached for Performance

To speed processing, the SMTP ALG maintains a cache of the most recently looked-up sender "From" IP addresses in local memory. If the cache becomes full then the oldest entry is written over first. There are two SMTP ALG object properties which can be configured for this address cache:

Cache Size

This is the number of entries that the cache can contain. If set to zero, the cache is not used. Increasing the cache size increases the amount of cOS Core memory required for anti-spam.
Cache Timeout

The timeout determines how long any address will be valid for once it is saved in the cache. After this period of time has expired, a new query for a cached sender address must be sent to the DNSBL servers.

The default value is 600 seconds.

The address cache is emptied when cOS Core restarts or a reconfiguration operation is performed.