Overview

This article covers a common scenario in GFI Kerio Connect where internal servers or scripts (such as PowerShell Send-MailMessage, SQL-driven email jobs, or monitoring tools) fail to authenticate via SMTP, logging errors such as:

• Failed SMTP login from [hostname] with SASL method NTLM
• User [account] performed NTLM authentication in NT domain [DOMAIN], this domain is not valid.
• 550 5.7.1 Authentication Required
• Only application passwords are allowed since 2FA is enabled

These errors typically arise when a script or application uses Windows NTLM authentication to send mail through Kerio Connect, but the Windows NT domain name configured in Kerio does not match what the client presents, or the sending tool cannot use PLAIN/LOGIN authentication (required for MFA/app-password accounts). This article explains the root causes and provides step-by-step guidance to resolve the failures and securely harden service accounts.

In This Article

• Symptoms
• Root Causes
• Resolution
  • Fix 1: Correct the Windows NT Domain in Kerio Connect
  • Fix 2: Switch the Sending Script to PLAIN/LOGIN Authentication
  • Fix 3: Use an Active Directory-Backed Account with Kerberos/NTLM
  • Fix 4: Use Port 587 (SMTP Submission) with STARTTLS
• Hardening Service Accounts
  • Create a User Access Policy to Restrict the Service Account
  • Allowlist SMTP Access by IP (Service-Level)
• Additional Information
• FAQ

Symptoms

• Kerio Connect security or debug logs show repeated entries like:
  Failed SMTP login from [hostname] with SASL method NTLM
• Log entry: User [account] performed NTLM authentication in NT domain [DOMAIN], this domain is not valid.
• Scripts using PowerShell Send-MailMessage return: Mailbox unavailable. The server response was: 5.7.1 Authentication Required
• Sending works only when SMTP authentication bypass (trusted relay) is enabled — which is not compliant for sensitive communications
• SSL errors when scripts use the server's internal IP address instead of its FQDN
• Mail sends successfully on port 25 but not on port 587

Root Causes

1. Windows NT domain mismatch: Kerio Connect's domain Advanced tab has the Windows NT domain set to the FQDN form (e.g., DOMAIN.ORG), but Windows clients present the short NetBIOS domain name (e.g., DOMAIN) during NTLM handshake. This causes Kerio to reject the authentication.
2. PowerShell Send-MailMessage cannot specify auth method: This built-in cmdlet does not allow forcing PLAIN or LOGIN authentication. On domain-joined hosts it defaults to NTLM, which is incompatible with Kerio local accounts that have MFA/app passwords enabled.
3. SSL certificate mismatch: Scripts configured to use the server's internal IP address fail TLS handshake because the SSL certificate is issued for the FQDN (e.g., webmail.example.com), not the IP.
4. Port misconfiguration: Port 587 (SMTP Submission) may not deliver mail if firewall rules or local endpoint security block outbound TCP/587 from the sending host.

Resolution

Fix 1: Correct the Windows NT Domain in Kerio Connect

This resolves the "this domain is not valid" error for NTLM-authenticating clients.

1. From a domain-joined machine, open a command prompt and run:
   echo %USERDOMAIN%
   Note the returned value — this is the short (NetBIOS) domain name that Windows presents during NTLM authentication (e.g., DOMAIN, not DOMAIN.ORG).
2. In Kerio Connect Administration, go to Configuration > Domains.
3. Double-click your email domain and open the Advanced tab.
4. Leave Kerberos realm as the full FQDN (e.g., DOMAIN.ORG).
5. Change Windows NT domain from the FQDN value to the short NetBIOS domain name you confirmed in step 1 (e.g., change DOMAIN.ORG to DOMAIN).
6. Click OK to save, then restart the Kerio Connect service during a maintenance window.
7. When testing authentication from scripts, format credentials using the short domain: DOMAIN\username 

Fix 2: Switch the Sending Script to PLAIN/LOGIN Authentication

PowerShell's built-in Send-MailMessage cmdlet does not allow specifying the SMTP authentication mechanism. On domain-joined hosts it may default to NTLM, which is incompatible with Kerio Connect local accounts (especially when MFA is enabled). Replace it with a library that supports explicit PLAIN/LOGIN authentication:

• PoshSMTP (github.com/jordanreiter/PoshSMTP) — PowerShell module with explicit auth method support
• MailKit / MimeKit (.NET library) — supports PLAIN, LOGIN, OAUTH2; recommended for production use
• Any SMTP client or library that lets you specify AuthenticationMechanism = PLAIN or LOGIN

When using these alternatives, configure them to connect to the server's FQDN (the name matching the SSL certificate, e.g., webmail.example.com), port 587, with STARTTLS enabled.

Fix 3: Use an Active Directory-Backed Account with Kerberos/NTLM

If switching the script library is not immediately possible, using an AD-backed Kerio account resolves NTLM failures without requiring a new auth library:

1. Create a corresponding Active Directory user account
2. In Kerio Connect Administration, go to Accounts > Users and edit the service account.
3. Change the account's authentication source from Internal user database to Kerberos 5 (Active Directory).
4. Ensure the Windows NT domain in Configuration > Domains > Advanced matches the NetBIOS domain (see Fix 1 above).
5. Update scripts to use the AD password and domain-qualified credentials: DOMAIN\username.

Note: With an AD-backed account, MFA can still be enabled in Kerio for interactive logins (webmail, clients). MFA enforcement in Kerio primarily applies to interactive logins — NTLM/Kerberos SMTP submission by scripts does not prompt for or require the second factor, so scripts continue to work using the AD password.

Fix 4: Use Port 587 (SMTP Submission) with STARTTLS

Port 25 is intended for server-to-server relay, not authenticated client submission. Port 587 with STARTTLS is the correct and secure configuration for authenticated senders.

1. Verify that SMTP Submission on port 587 is enabled in Kerio Connect: go to Configuration > Services and confirm the SMTP Submission (587) service is running.
2. On the sending host, check outbound firewall rules to ensure TCP/587 to the Kerio server is permitted. Endpoint security (antivirus/DLP) that performs SMTP inspection may also interfere with STARTTLS negotiation.
3. Update the script/application to use:
   • SMTP server: FQDN matching the SSL certificate (e.g., webmail.example.com) — not the internal IP address
   • Port: 587
   • Encryption: STARTTLS (not implicit TLS/SSL on port 465, unless your client explicitly requires it)
   • Authentication: PLAIN or LOGIN over the encrypted connection

Important: Using the internal IP address (e.g., 192.168.0.16) for SSL/TLS connections will cause certificate handshake failures because SSL certificates are issued for hostnames, not IP addresses. Always use the FQDN.

Hardening Service Accounts

Once SMTP sending is working, apply the following hardening steps to minimize the security footprint of service/automation accounts — especially important when MFA cannot be enforced on the account for script compatibility, or when the account handles sensitive data.

Create a User Access Policy to Restrict the Service Account

A User Access Policy lets you restrict which protocols a specific account can use, preventing interactive access (webmail, Outlook, mobile sync) while still allowing SMTP submission.

1. In Kerio Connect Administration, go to Configuration > Definitions > User Access Policies.
2. Click Add to create a new policy (e.g., named "Service Account — SMTP Only").
3. Add restrictions to deny all interactive/client protocols:
   • Kerio Connect Client (webmail)
   • ActiveSync
   • EWS (Exchange Web Services)
   • IMAP
   • POP3
   • WebDAV / CalDAV / CardDAV
   • Kerio Outlook Connector (online and offline)
   • XMPP
4. Leave SMTP and SMTP Submission allowed (or restricted by IP — see below).
5. Save the policy, then assign it to the service account: go to Accounts > Users, open the service account, go to the Rights tab, and set Access policy to the policy you just created.

Note: SMTP is not listed under User Access Policies in some Kerio Connect versions. SMTP access restriction is enforced separately at the service level (see below). User Access Policies control interactive/client protocols only.

Allowlist SMTP Access by IP (Service-Level)

To further restrict which servers are allowed to use SMTP/SMTP Submission, apply an IP address group restriction at the service level. This applies globally to the service (all accounts), so use it for the submission port used exclusively by your internal senders.

1. Go to Configuration > Definitions > IP Address Groups and create a group (e.g., "Internal Mail Senders") containing the IP addresses of all internal hosts authorized to submit mail (application servers, monitoring systems, Hyper-V hosts, etc.).
2. Go to Configuration > Services and double-click SMTP Submission (587).
3. On the Access tab, select "Allow access only from the selected IP address group" and choose your newly created IP group.
4. Optionally, apply the same restriction to the SMTP (25) service if internal hosts also submit via port 25.
5. Click OK to save.

Note: Restricting SMTP Submission by IP at the service level affects all accounts that use that service. Ensure all legitimate senders (including user mail clients, if they use port 587) are included in the IP group, or use a broad internal subnet.

Additional Information

Why NTLM fails with MFA/app passwords: Kerio Connect requires PLAIN or LOGIN authentication when app passwords are used for MFA-enabled local accounts. NTLM authentication bypasses the app-password mechanism and is only valid for AD-backed accounts when the NT domain mapping is correct. Mixing a local Kerio account (not AD-backed) with an NTLM-authenticating client will always fail.

SMTP authentication methods: Kerio Connect supports PLAIN, LOGIN, CRAM-MD5, DIGEST-MD5, and NTLM. PLAIN and LOGIN transmit credentials in plaintext, so they must only be used over an encrypted (STARTTLS/TLS) connection. NTLM is intended for AD-integrated environments. Kerio's Security Policy setting "Require encrypted connection" (under Configuration > Security > Security Policy) enforces TLS before authentication for all clients.

PRTG and monitoring tools: Tools like PRTG Network Monitor that use simple SMTP sensors may not support modern authenticated SMTP. If the tool cannot be configured to use SASL PLAIN/LOGIN over TLS, add its IP to a dedicated IP group and permit unauthenticated relay from that group at the service level — or configure the tool to use authenticated SMTP with a supported method.

References:

• Using Kerberos Authentication with Kerio Connect
• Securing Kerio Connect
• Troubleshooting SMTP-Related Issues

FAQ

Q1: Can I enable MFA on a service account used by scripts and still have those scripts send mail successfully?
A1: Yes — if the account is Active Directory-backed and the script authenticates via NTLM or Kerberos using the AD password. Kerio Connect's MFA enforcement applies to interactive logins (webmail, Outlook connector, ActiveSync), not to NTLM/Kerberos SMTP submission. Scripts using the AD password continue to work even when MFA is enabled on the account. MFA with app passwords only works if the SMTP client can use PLAIN or LOGIN authentication (not NTLM).

Q2: Why can't I find "SMTP" or "SMTP Submission" in User Access Policies?
A2: This is expected behavior. Kerio Connect's User Access Policies only control interactive/client protocols (Kerio Connect Client, ActiveSync, EWS, IMAP, POP3, WebDAV/CalDAV/CardDAV, Kerio Outlook Connector, XMPP). SMTP services are controlled separately at the service level under Configuration > Services > [SMTP service] > Access. To restrict SMTP by IP, apply an IP address group restriction directly to the SMTP or SMTP Submission service.

Q3: Restricting SMTP Submission by IP at the service level — will this break regular users sending mail via Outlook or webmail?
A3: It depends on which port your end users' mail clients use for sending. If users' Outlook profiles or mobile devices are also configured to use port 587, restricting that port by IP will affect them. To avoid this, either (a) add all user IP ranges to the allowed group, (b) use a broad internal subnet, or (c) use a separate submission port for scripted senders versus end users. Webmail (Kerio Connect Client) does not use SMTP Submission — it submits mail internally — so webmail users are unaffected by service-level SMTP restrictions.