6.3. Email Control

Home Prev	cOS Core 15.00.04 Administration Guide	Next

Email traffic can be a major concern for a system administrator, both because of its volume and because of the security threats it can carry. Unsolicited email, often referred to as Spam, is sent out by groups known as spammers in massive quantities and can waste resources, transport malware, as well as try to direct the recipient to webpages that might present further security risks.

6.3.1. Configuring Email Control Profiles

With IP Policies, email filtering can be applied to IMAP, POP3 and SMTP traffic using IP Policies. This is done by creating an Email Control Profile and associating it with an IP Policy object. With IMAP and POP3 filtering, emails cannot be dropped when they fail filtering but only marked as failed. With SMTP, emails can be dropped or forwarded.

Setting Up Email Filtering with IP Policies

IP policy based email filtering is set up with the following steps:

Create an Email Control Profile object which defines how email is to be filtered. If anti-spam filtering is required, it must be explicitly enabled in the profile (by default, it is disabled).
Optionally add one or more Email Filter objects as children to the Email Control Profile object. Each will specify an email address (or addresses using wildcards) which are to be blacklisted (automatically rejected before filtering) or whitelisted (never subject to filtering).
Associate the Email Control Profile object created above with an IP Policy object which triggers on the email traffic. Only a single profile can be associated with an IP policy.
The Service property for this IP policy must trigger on the IMAP, POP3 or SMTP protocols so it must be set to an appropriate Service object. The Service object used must have its Protocol property set to IMAP, POP3 or SMTP (whichever applies).

The predefined IMAP, POP3 and SMTP services could be used by setting their Protocol property to be IMAP or POP3 or SMTP. However, it is recommended to instead create a new custom Service object and this is done in the setup example found at the end of this section.
Optionally enable anti-virus scanning on the IP policy. This will scan any email attachments for viruses and will function with the IMAP, POP3 or SMTP protocol. Anti-virus scanning is discussed further in Section 6.4, Anti-Virus Scanning.

	Note: A service group cannot be used for email filtering
	It is not possible to use a custom Service Group object for IP policy based email filtering which combines any of the IMAP, POP3 or SMTP protocols. Separate IP Policies with separate services for IMAP, POP3 or SMTP must be configured.

General Email Control Profile Settings

The following Email Control Profile options are available for all types of traffic.

Anti-Spam

When enabled, anti-spam scoring is applied to emails to determine if it could be spam. The anti-spam feature is described further later in this section.

By default, this option is disabled.
Blacklist Subject Tag

This is the text inserted into the subject field of an email if it is found on the blacklist for this profile.

Protocol Specific Options

The following additional options are available for when the profile is used with specific email related protocols:

SMTP Settings

These settings relate to processing by the SMTP ALG and are described further in Section 6.1.5, SMTP ALG.
POP3 Settings

These settings relate to processing by the POP3 ALG and are described further in Section 6.1.6, POP3 ALG.
IMAP Settings

These settings relate to processing by the IMAP ALG and are described further in Section 6.1.7, IMAP ALG.

Encryption and Allowing STARTTLS

By default, an Email Control Profile object will drop encrypted traffic for SMTP, POP3 or IMAP since none of the profile's checks can be applied when encryption is used.

If the profile property Allow STARTTLS is enabled, encrypted SMTP, POP3 or IMAP traffic will be allowed, provided that the encryption results from a STARTTLS command to change over from plain text. However, none of the profile's checks will be applied to this encrypted traffic.

Email Address Whitelist/Blacklist

One or more Email Filter objects can be added as children to an Email Control Profile object. Each filter specifies one address or, using wildcards, a series of addresses that are to be always dropped (blacklisted) or always allowed (whitelisted).

The wildcard asterisk ("*") character can be used to specify any string. For example, the string *.@example.com represents any email address from the example.com domain. The question mark ("?") character can also be used to represent any single character.

Whitelisting and blacklisting takes precedence over all other email filtering. Whitelisted emails will never be subject to anti-spam or anti-virus processing if either of those is enabled.

If an email comes from a blacklisted domain, the mail is not dropped. Instead, the subject line has a configurable text string inserted at the beginning. By default, this string is:

			*** BLACK LISTED ***

However, this text can be set by the user, as previously described.

Note that some IMAP clients may download and display the headers of emails before they download the email body. This can mean that the blacklist text may not be displayed correctly in the subject line with such clients. This is discussed further in the later section on anti-spam since the same problem can exist with flagging spam in the email subject line.

Anti-Spam and Filter Scoring

The anti-spam feature of email filtering using IP Policies must be explicitly enabled in the Email Control Profile object used with an IP policy. Once enabled, a score is calculated by adding together the sub-scores of the enabled anti-spam filters. The score is used in the following ways:

If the total score is equal to or greater than the specified Tagging Threshold (the default value is 10) then an email is considered to be SPAM and tagged as such.
For SMTP only, if the total score is equal to or greater than the specified Reject Threshold (the default value is 20) then an email is dropped.

Each anti-spam filter can be enabled or disabled and they are comprised of the following list:

Domain Verification

When enabled, cOS Core will perform a DNS MX query, requesting the IP address of a mail server associated with the email sender's domain. For example, the sender email address might be some.name@example.com so cOS Core will send an MX query to the configured DNS server for the mail server at example.com.

If the DNS server recognizes the domain name, cOS Core takes no action. If the DNS server does not recognize the domain, cOS Core will add the sub-score for this filtering option to the overall anti-spam score.

Note that at least one DNS server must be configured in cOS Core for this option to work. If no DNS server is configured, this test will not be performed and its sub-score will not be added. Configuring DNS servers is described in Section 3.10, DNS.

This filtering option is enabled by default.

Malicious Link Protection

This option neutralizes undesirable or malicious HTML links inside emails. It is enabled by default and is discussed later in this section in more detail. Finding a single triggering link will add this option's sub-score (a default value of 10) to the total score. Malicious link protection is described in depth later in this section.

DCC

The Distributed Checksum Clearinghouses (DCC) method of filtering involves cOS Core sending anonymous checksums that identify an email to an external Clearing House server. The server returns a value to indicate how many other email recipients have reported identical checksums. If the returned value is greater than the DCC Threshold property set by the administrator, the sub-score for this filter option is added to the total anti-spam score.

DCC filtering is enabled by default and the default sub-score is 10. The administrator does not need to define any DCC servers because cOS Core uses Clavister's own server network for this function.

The DCC option is a subscription based feature and it will only function if the current date is prior to the DCC subscription date specified in the cOS Core license. If the DCC feature is enabled but the license does not allow it, the behavior is described in Appendix A, Subscription Based Features under the heading Subscription Expiry Behavior.

The DCC feature has its own parameter and expiry date in a Clavister license file. For this reason, if cOS Core is upgraded from a version that does not have DCC (prior to version 11.00) to a version that does (11.00 or later), a new cOS Core license with DCC parameter must be created and installed.

DNS Blacklists

By enabling the DNS Blacklists option, up to 10 different DNS Blacklist (DNBL) servers can be specified, each with its own sub-score (the default value is 10 per server) that will be added to the total score if that server flags an email as spam.

If a DNSBL server is not responding then its sub-score is not included in the final score calculation. How DNSBL functions is explained further in Section 6.3.2, DNSBL Processing. For an example of configuring DNSBL anti-spam filtering, see Example 6.8, “IMAP ALG Setup”.

Email Tagging

If an email score exceeds the Tagging Threshold property, it is marked as being SPAM. This is done in either or both of two ways:

Tag Subject

This is enabled by default and adds text to the subject line of an email. The default text is "*** SPAM ***" but this can be set to any string.

An example of this kind of tagging is if the original Subject field is:
```
 Buy this stock today!
```
If the tag text is defined to be "*** Probably SPAM ***", then the modified email's Subject field will become:
```
 *** Probably SPAM *** Buy this stock today!
```
The line above is what the email's recipient will see in the summary of their inbox contents. The individual user could then decide to set up their own filters in the local client to deal with such tagged emails, possibly sending it to a separate folder.
Tag Header

This is also enabled by default and it inserts X-Spam information into the bottom of the email header data which describes the anti-spam processing that has been performed on the email. When enabled, this indicates if the email is considered spam or not. This information may not be immediately visible to the recipient but many email clients will provide the option to view it and many will also allow it to be part of the client's filtering criteria.

The X-Spam fields inserted by cOS Core are as follows:
1. X-Spam-Checker-Version - The software that tagged the email.
2. X-Spam-Status - A list that includes the scoring and the filters applied.
3. X-Spam-Flag - Included only for emails marked as spam and always Yes.
4. X-Spam-Report - A detailed list of the results from applied filters.
An example of inserted X-SPAM information for an email that was flagged as spam is the following:
```
X-Spam-Checker-Version: Clavister cOS Core on Device
X-Spam-Status:
     Yes, score=30 required=10 tests=LINK_PROTECTION,DCC,DNS_BLACKLIST_1
X-Spam-Flag: Yes
X-Spam-Report:
* 10 LINK_PROTECTION: Contained undesirable web links
* 10 DCC: Checksum reported >= 16777200 times
* 10 DNS_BLACKLIST_1: Blacklisted source IP address
```
An example of inserted X-SPAM information for an email that was not flagged as spam is the following:
```
X-Spam-Checker-Version: Clavister cOS Core on Device
X-Spam-Status: No, score=5 required=10 tests=DNS_BLACKLIST_1
```

Malicious Link Protection

The Malicious Link Protection filter for anti-spam allows undesirable links in email traffic to be neutralized as the email passes through cOS Core. It is part of anti-spam filtering and can only be enabled when anti-spam is enabled. Links within emails are evaluated by cOS Core using the Web Content Filtering subsystem which is described in Section 6.2, Web Content Filtering.

The following points should be noted for the link protection:

Link protection will only examine the first 10 links in an email. After that no more link processing is performed for subsequent links.
When a link triggers web content filtering, the link will still appear in the mail but will be disabled so it cannot be clicked.
If one or more links trigger the specified link protection score (see properties below) will be added just once for an email to the overall anti-spam score.

The following properties of an Email Control Profile are used to control link protection:

Link Protection

By default, link protection is enabled if anti-spam is enabled. Disable this property to switch it off for all emails.
Link Protection Score

This is the score that is added to the total anti-spam score if just one link is found that triggers web content filtering. The default value is 10.
Link Protection Categories

A subset of the list of all web content filtering categories can be defined and this subset will be disallowed. By default, this is set to a limited subset. All available categories are described in Section 6.2, Web Content Filtering.
Non-Managed Action

This determines what to do with a link that could not be determined to belong to one of the web content filtering categories. The choice is either to allow such links (the default) or to disallow them.

	Note: A valid web content filtering subscription is required
	The malicious link protection filter will only function if cOS Core also has a valid subscription for the web content filtering feature since this is used to evaluate links. This is described further in Appendix A, Subscription Based Features.

IMAP Clients May Display Incorrect Header Information

Email clients using IMAP to retrieve email details from an email server, can download and display the headers of emails before they download the email body. This means the user can only download the body of emails they want to read based on the header information.

Since cOS Core may perform some of its spam scoring based on the email body, the initial header information displayed by the client may not yet correctly show the spam text in the subject line. When the email body is eventually downloaded, cOS Core examines it and the header information will be updated if the mail is now considered to be spam. However, some email clients may not update the displayed header information and will continue to display the original unflagged header information. This is not something that can be controlled by cOS Core.

In all cases, the spam related log messages generated by cOS Core, and the spam related statistics it keeps, will correctly reflect which emails it has classified as spam.

As mentioned previously, a similar problem can arise with blacklisting and IMAP clients. The initial header information displayed by the client may not correctly show the blacklist text.

6.3.2. DNSBL Processing

Public DNSBL Databases

A number of trusted organizations maintain publicly available databases of the origin IP address of known spamming SMTP servers and these can be queried over the Internet. These lists are known as DNS Black List (DNSBL) databases and the information is accessible using a standardized query method supported by cOS Core.

DNSBL Server Usage By cOS Core

When the cOS Core anti-spam filtering feature is configured, the IP address of the email's sending server is sent to one or more DNSBL servers to find out if any DNSBL servers think the email is from a spammer or not. cOS Core examines the IP packet headers to do this. The diagram below illustrates cOS Core's interaction with DNSBL servers.

Figure 6.9. DNSBL Anti-Spam Processing

The reply sent back by a DNSBL server is either a not listed response or a listed response. In the latter case of being listed, the DNSBL server is indicating the email might be spam and it will usually also provide information known as a TXT record which is a textual explanation for the listing.

	Tip: Finding a list of available DNSBL servers
	A list of public DNSBL servers can be found at: https://en.wikipedia.org/wiki/Comparison_of_DNS_blacklists.

Creating a DNSBL Consensus

Multiple DNSBL servers can be specified in an Email Control Profile object which is then assigned to an IP policy that triggers on the targeted traffic. A weight is also assigned to each DNSBL server and this must be an integer greater than zero. A weighted sum can then be calculated based on all server responses and the following thresholds for the sum can be specified in the email control profile:

Tag Threshold

If the sum is greater than or equal to this value then the email is considered as probably being spam but forwarded to the recipient with notifying text inserted into it.
Reject Threshold

If the sum is greater than or equal to this value then the email is considered to be definitely spam and is discarded or alternatively sent to a single, special mailbox. This value will only apply to SMTP traffic.

If it is discarded then the administrator has the option that an error message is sent back to the sending SMTP server (this error message is similar to the one used with blacklisting).

A DNSBL Threshold Calculation Example

As an example, suppose that three DNSBL servers are configured: dnsbl1, dnsbl2 and dnsbl3. Weights of 3, 2 and 2 are assigned to these respectively. The spam threshold is then set to be 5.

If dnsbl1 and dnsbl2 say an email is spam but dnsbl3 does not, then the total calculated will be 3+2+0=5. Since the total of 5 is equal to (or greater than) the threshold then the email will be treated as spam.

If the Drop threshold in this example is set at 7 then all three DNSBL servers would have to respond in order for the calculated sum to cause the email to be dropped (3+2+2=7).

Allowing for Failed DNSBL Servers

If a query to a DNSBL server times out then cOS Core will consider that the query has failed and the weight given to that server will be automatically subtracted from both the spam and drop thresholds for the scoring calculation done for that email.

If enough DNSBL servers do not respond then this subtraction could mean that the threshold values become negative. Since the scoring calculation will always produce a value of zero or greater (servers cannot have negative weights) then all email will be allowed through if both the Spam and Drop thresholds become negative.

A log message is generated whenever a configured DNSBL server does not respond within the required time. This is done only once at the beginning of a consecutive sequence of response failures from a single server to avoid unnecessarily repeating the message.

DNSBL Server Query Logging

The following types of log messages are generated by cOS Core as a result of DNSBL server queries:

Logging of dropped or spam tagged emails - These log messages include the source email address and IP as well as its weighted points score and which DNSBLs caused the event.
DNSBLs not responding - DNSBL query timeouts are logged.
All defined DNSBLs stop responding - This is a high severity event since all email will be allowed through if this happens.