Phone Lines icon GFI Support Home

Articles in this section

See more

Compromised Servers - Troubleshooting Article

Author: Benji Bizzell Updated

Overview

This article explains how to troubleshoot a potentially compromised account or server. Common symptoms related to these issues are:

  • Users report issues with sending or receiving emails, and your Mail Queue appears to be filled with suspicious emails with unexpected senders or recipients.
  • Your server performance is reduced, and the server appears to struggle to process the growing number of emails.

 

Workflow

code2flow_s135ey__5_.png

 

Instructions

Each blue rectangle represents a troubleshooting procedure and links to a section in this article:

  1. Enable Message Queue Sender Details
  2. Run Malware Scans
  3. Review the Sender and Recipient Pairs
  4. Contact Sender to Confirm Legitimacy
  5. Change Passwords
  6. Clear Connections and Message Queue
  7. Review Relay Settings
  8. Restrict Access to the Relay
  9. Remove Unexpected IPs from IP Address Groups
  10. Perform Blacklist Check
  11. Add DNS Security Records and Harden Server Security
  12. Investigate SMTP Delivery Issue
  13. Disable All IP Group Exceptions

 

Enable Message Queue Sender Details

As a first step when you suspect the server may have become compromised, you should enable additional Message Queue Columns and analyze the emails built up within the queue.

 

mceclip0.jpg

  1. Navigate to WebAdmin > Status > Message Queue.
  2. Right-click any Column Header > Columns:
    1. Enable Authenticated Sender and Sender IP.

Review the emails within the queue and confirm whether any of the emails appear suspicious based on their Authenticated Sender, Unexpected Sender IPs, or Sender and Recipient pairings.

  • If you identify a mismatched Authenticated Sender and “From” field, Change the Password of the “Authenticated Sender.”
  • If you identify a single Internal Sender IP Address for most emails, Run a Malware Scan on the associated devices.
  • If you identify a single External Sender IP Address for most emails, Change the Password of the “Authenticated Senders.”
  • If you identify several unrecognized External Sender IP Addresses, Review the Relay Settings.

 

 

Run Malware Scans

In situations where a single Internal Sender IP Address is appearing for the majority of the suspicious or unexpected emails within your mail queue, this may be a sign of the identified host machine containing Malicious software (such as a Trojan).

It is suggested that you run a Malware/AV Scan against any device that the host machine or associated user may have accessed. There is no harm in performing such a scan against all machines connected to the Network for added safety. 

In addition to running these Malware scans, it is suggested that you enforce a Password Change for the users within the server for added security.

 

 

Review the Sender and Recipient Pairs

In some situations, you may not see any clear signs of a single compromised user or IP but notice some odd traffic. For example, emails are being sent to domains in locales that your company typically does not email. When encountering this, you should compare the Sender and Recipients within the message queue to isolate the exact User Account used to email these individuals.

Suppose you identify a clear pattern, such as a single user unexpectedly emailing irregular foreign email addresses. In that case, you can Contact the Sender directly to help confirm the legitimacy of the emails.

Otherwise, if the domains appear valid, but the volume raises concerns, this may not be a case of a compromised account. Instead, you can take a closer look at a potential SMTP Delivery Issue.

 

 

Contact Sender to Confirm Legitimacy

The easiest way to validate the potential for odd email patterns is to contact the user where the emails originated. While unexpected delivery to unknown or irregular emails may be a sign of a compromised account, they may be reaching a new client with a foreign domain.

If the user confirms that the emails are legitimate, but you continue to see a build-up within the queue, there may be an SMTP Delivery Issue at play.

If the user confirms that they never sent the suspicious emails, you should immediately change the user’s password to prevent further spam.

 

 

Change Passwords

While it is often most important to target the specific users responsible for the identified spam within your Message Queue, we recommend enforcing a server-wide password change. In some situations, you may find that multiple accounts may have become compromised, and changing a single user will shift the traffic to another within the server. 

Once you have updated all of the passwords for your users, the next step will be to validate the solution by Clearing Connections and the Message Queue.

 

 

Clear Connections and Message Queue

While reviewing suspected compromised servers or accounts, it is crucial that you properly validate that any changes made have fully resolved the issue. The most reliable method to confirm this is to Clear the Message Queue Manually. This process involves stopping the Kerio Connect services, which forces any existing connections to be closed, and “resetting” the message queue.  

Further, as an overloaded message queue can place a heavy strain on your server, this can allow normal email flow to be restored while you continue your mitigation. Once cleared, monitor the situation and confirm that the issue does not persist.

If you detect that further unexpected messages appear in high volume, this could be the sign that your Relaying Settings are too lax.

 

 

Review Relay Settings

A common cause for compromised mail servers is when the Relay is under-secured, such as when it has been set to “Open Relay.” If the initial account-level mitigation steps have not addressed the message queue issues, this is often the next best place to look.

  1. Navigate to WebAdmin > Configuration > SMTP Server.
  2. Check the Relay Control Tab:
    mceclip1.png
    1. Verify that the “Open Relay” setting is not being used.
    2. Review the “Allow Relay Only From” section to determine potential flaws, such as:
      1. Unexpected IP Address Group Exceptions.
      2. Allowing Unauthenticated Relaying.
  3. Check the Security Options Tab:
    1. In most environments, it is beneficial to use most of these settings.
    2. If you have disabled all of these, it may be exposing you to common attack vectors:mceclip2.png
  4. Check the SMTP Delivery Tab:
    mceclip3.png
    1. Confirm that you do not see any unexpected, unauthenticated Relay Rules.
    2. If you use a Relay, verify that it is adequately secured and the settings entered are valid.
      1. When possible, it may be worth temporarily disabling any of these rules to isolate them as a cause for the odd traffic patterns seen.

If you identify any clear signs of the Relay security being lax, it is suggested that you restrict access further.

If you don’t see any settings that would indicate the Relay is more open than it should be, the exception IP Groups in use may themselves contain unexpected entries.

 

 

Restrict Access to the Relay

We suggest using all of the security options available to help rule out particular attack vectors. In particular, consider using the following settings and the associated default values. Many of these are explicitly targeting malicious behavior patterns:

  • SMTP Relay Control:
    mceclip4.png 
  • Security Options:
    mceclip5.png
    • Max Messages per Hour from One IP.
    • Max Concurrent Connections from One IP.
    • Max Unknown Recipients.
    • Block if Sender domain not in DNS.
    • Block if Client IP has no Reverse DNS.
    • Max Recipients in Message.
    • Max Failed Commands in SMTP Session.

Once you have hardened the security of your SMTP Relay settings, it is suggested that you once again Clear the Connections and Mail Queue to verify the solution.

If you are using IP Address Group Exceptions, Review the IP groups themselves directly to confirm no errant IP addresses or ranges are defined.

 

 

Remove Unexpected IPs from IP Address Groups

One of the often-overlooked settings within Kerio Connect environments is the IP Address Groups configured. Many admins will only use the default “Local Clients” group. However, the default group may have been inadvertently edited, resulting in an exposed attack vector.

These IP Address Groups can be assigned to the following security-related menus, which should each be reviewed to confirm whether an exception is in place and the Group the exception is applied to:

  • Security > Security Policy
  • Security > Sender Policy
  • SMTP Server > Relay Control
  • SMTP Server > Security Options
  • Spam Filter > Blacklists

Once you have verified the IP Address Groups you are using in the above rules, you can directly look at the Groups within the WebAdmin > Configuration > IP Address Groups menu. Consider the following example of a common mistake within the Relay configuration exceptions. The user attempted to define a local client mask but inadvertently created an extensive range of IPs:

mceclip6.png

 

Once you have confirmed that the IP Address Groups are configured correctly, and any changes are saved, you should verify the solution by Clearing the Connections and Mail Queue.

 

 

Perform Blacklist Check

After you have confirmed that the issue is resolved and the Message Queue is no longer filling with unexpected or malicious emails, you are now ready to perform a bit of clean-up. The most apparent issue with spam is the reputation damage it can cause to your Domain and Brand. When many Spam emails are sent from your mail server, this can result in your IP being blacklisted, which causes future delivery issues.

We suggest that you run a scan of your domains using a tool such as MxToolbox, which checks a variety of blacklists and can provide you with some proposed handling. Many blacklists are temporary, but you may need to contact the blacklist operators or your ISP to investigate your options.

Regardless of the outcome of the above check, the next step will be to further Secure your domain and Server Security.

 

 

Add DNS Security Records and Harden Server Security

Many compromised servers have not adequately defined essential security-focused DNS records, such as SPF, PTR, and DKIM. Each of these serves a slightly different role but accomplishes the same end goal - allowing external Mail Servers to “validate” whether a spam email originated from your server, protecting your domain’s reputation.

In addition to adding any missing essential DNS Records, consider the following Security Adjustments based on the cause of your issue:

 

 

Investigate SMTP Delivery Issue

You may have encountered what initially appeared to be an issue with a compromised mail server, but you see an SMTP Delivery Issue upon further investigation. When your Message Queue begins to fill up with unsent emails, it is appropriate to raise some alarms and run the checks to verify this is not the result of some malicious activity. 

If you have verified that this is not the case, it is suggested that you pivot to Troubleshooting the Outgoing Email Delivery Issues instead.

 

 

Disable All IP Group Exceptions

In some situations, you may notice that all security settings appear to be correct, and no unexpected IP Addresses are set to include an exception, but you continue to see Spoofed Spam Messages filling the queue. When this occurs, you should recheck to verify whether the existing messages in the queue are showing a Sender IP of Localhost (127.0.0.1) or another Local IP. In these situations, this most often means there may be a compromised device in the network.

You can reference the steps within Anti-Spoofing not preventing Spoofed Spam Messages from a certain IP to prevent further messages by disabling all IP Group Exceptions. Once this is completed, you should re-run your Malware Scans and expand this to all devices in your network.

 

 

Related articles