Tag Archives: Exchange 2016

Exchange, Office365

Email not delivered to DANE enabled domains in Office 365

Leave a comment

The servicedesk got complaints that email was not delivered to an organization, and that an NDR was never generated. The sending user never knew this, until the sender and receiver talked on the phone. Our organization was Exchange 2016 on-premises with a Cisco IronPort as a gateway to the Internet. The other (receiving) organization was in Exchange Online.

The receiving organization was in Exchange Online, and had already enabled DANE for inbound email messages (see my previous blog post on this). I checked multiple organizations in Exchange Online that have DANE enabled (including hotmail.nl) and they all failed.

One organization has a Fortra Clearswift in front of their environment that has DANE enabled, and we were able to send email to this particular domain.

And to make it more complex, other organizations with an IronPort gateway were able to successfully send email messages to these domains.

At this point still no clue whether it is a Microsoft issue, or an IronPort issue or something specific to our organization.

When checking the DANE SMTP service for the domains involved everything looks fine as shown in the following screenshot:

When checking the IronPort logs, the following cryptic and non-explaining error was logged:

MID 1614647 (DCID 600113) DANE failed for Hotmail.nl. Reason: 4.0 - Other network problem.

Also shown in the following screenshot:

The same error was logged for the other DANE enabled domains as well.

So, DANE fails on the IronPort, but the tools to check DANE all reported DANE was good. Also, I was able to send mail to these domains using Gmail. I always say when it works in Gmail, everything is ok.

When checking the DANE configuration on the command prompt on the IronPort it looks more like a DNS issue as shown in the following screenshot:

But when checking the DNS record (_25._tcp.exchangelabs-nl.y-v1.mx.microsoft)
with MXToolbox, everything is green again as shown in the following screenshot:

After checking with the network department it turned out that there was an IPS solution implemented and the network engineer knew about an old CVE, dating back to 2013 (CVE-2013-4466) that warns for a situation where a buffer flow can occur in the dane_query_tlsa function when more that four DANE entries are returned.

The CVE is 11 years old, but the IPS still had this implemented, and when more than four entries were returned, everything was discarded and the email was lost. Removing this from the IPS and everything works fine.

Note. The RFC for DANE does not mention the amount of entries that can be returned, so more than four should not be a problem.

Exchange

Hotfix Update for Exchange 2016 and Exchange 2019

Leave a comment

Wait, what? On April 23, 2024 Microsoft has released a hotfix update for Exchange 2016 and Exchange 2019 and as MVP’s we only learned about this last week.

A hotfix update or HU contains fixes for issues that might arise with a security update in Exchange server. For example, the March 2024 SU for Exchange server introduced a number of issues, and these are fixed with this HU. Besided hotfixes, a HU can also contain new features that did not make it in the last security update (SU) or Cumulative Update (CU). In this HU for example, Hybrid Modern Authentication for OWA and ECP is introduced as a new feature. Another new feature introduced in this HU is the support for ECC (Elliptic Curve Cryptography) certificates. ECC certificates however are not supported for the federation trust certificate, the Exchange server OAuth certificate and ECC certificates cannot be used when ADFS claims-based authentication is used.

The following issues are fixed in this HU:

  • “We can’t open this document” error in OWA after installing March 2024 SU
  • Search error in Outlook cached mode after installing March 2024 SU
  • OwaDeepTestProbe and EacBackEndLogonProbe fail after installing March 2024 SU
  • Edit permissions option in the ECP can’t be edited
  • Outlook doesn’t display unread message icon after installing Exchange Server March 2024 SU
  • My Templates add-in doesn’t work after installing Exchange Server March 2024 SU
  • Download domains not working after installing the March 2024 SU

You can download this hotfix update for Exchange server here:

Exchange 2019 CU14 HU2 – https://www.microsoft.com/en-us/download/details.aspx?id=106021
Exchange 2019 CU13 HU6 – https://www.microsoft.com/en-us/download/details.aspx?id=106022
Exchange 2016 CU23 HU13 – https://www.microsoft.com/en-us/download/details.aspx?id=106023

Be aware that the filename for all versions of this HU is the same (Exchange2019-KB5037224-x64-en.exe) so when downloading multiple versions make sure you store them at different locations.

A hotfix update is cumulative and includes all security features and fixes from the previous security updates. When running Exchange 2019 CU14 and you have not installed the March 2024 security update then there’s no need to install this first. Just continue with the immediate installation of this HU.

More information

Exchange Hybrid

HCW8078 – Migration Endpoint could not be created

Leave a comment

When running the hybrid configuration wizard on an Exchange 2016 server (in an Exchange 2010 to Exchange 2016 migration) to create a (classic) hybrid configuration, the wizard failed with several error messages as shown in the following screenshot:

Or in plaint text:

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException. The connection to the server ‘mail.contoso.com’ could not be completed.

Microsoft.Exchange.MailboxReplicationService.MRSRemoteTransientException
The call to ‘https://mail.contoso.com/EWS/mrsproxy.svc’ failed. Error details: The HTTP request was forbidden with client authentication scheme ‘Negotiate’. –> The remote server returned an error: (403) Forbidden..

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException
The HTTP request was forbidden with client authentication scheme ‘Negotiate’.

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException. The remote server returned an error: (403) Forbidden. 

I did the following troubleshooting steps:

  • Manually creating a migration endpoint using the Exchange Online Admin console. This failed as well.
  • Firewall restrictions were in place, but the Microsoft IP ranges were all configured correctly
  • Checked the MRS. It was enabled (well, that’s wat the Exchange PowerShell said) and authentication was set correctly

Eventually it turned out that the MRS was the problem. Although Exchange Management Shell returned that the MRS Proxy was enabled, it was not functioning correctly. I disabled the MRS proxy, enabled it again and restarted IIS using the following PowerShell commands:

[PS] C:\> Set-WebServicesVirtualDirectory -Identity EXCH01\EWS -MRSProxyEnabled $False
[PS] C:\> Set-WebServicesVirtualDirectory -Identity EXCH01\EWS -MRSProxyEnabled $True
[PS] C:\> IISRESET

When finished, using the Get-WebServicesVirtualDirectory command still returned that MRS Proxy was enabled, as shown here:

[PS] C:\>Get-WebServicesVirtualDirectory -server exch01 | select Name,MRSProxyEnabled
 
Name                   MRSProxyEnabled
----                   ---------------
EWS (Default Web Site)            True

But this time it was working correctly and the Hybrid Configuration Wizard finished successfully.

Exchange, Exchange Hybrid

Hybrid configuration wizard hangs when adding federated domains

Leave a comment

I am working on an Exchange 2010 project (no typo!) where all mailboxes need to be moved to Exchange Online. Since Exchange 2010 is out-of-support for years, creating an Exchange 2010 hybrid environment is not a good idea. So, instead I installed Exchange 2016 and created a hybrid Exchange 2016 environment. But that brings it challenges as well.

The first issue we ran into when creating the hybrid configuration was the TLS 1.2 issue. Exchange 2016 (on Windows 2016) supports TLS 1.2, but it needs to be enabled. To enable TLS 1.2 on Exchange 2016, follow the instructions on the Exchange Server TLS configuration best practices article on the Microsoft website.

When running the Hybrid Configuration Wizard for the first time you must add the domains for federation with Office 365. DNS TXT records were added for all four domains but validating failed after validating the first domain. The wizard hangs on validating subsequent domains.

Removing the other three domains from the wizard and thus validating with only one domain solved our problem and the wizard successfully finished. Later on we were able to add the other domains, but only one at a time.

Exchange

Exchange Security Updates June 2023

Leave a comment

On June 13, 2023 Microsoft has released Security Updates for:

  • Exchange 2019 CU13
  • Exchange 2019 CU12
  • Exchange 2016 CU23

There are no Security Updates released for older versions of Exchange 2016 and Exchange 2019, these are the only supported versions. There are also no Security Updates for Exchange 2013 since this is completely out-of-support. If you are still running on Exchange 2013 you must seriously consider upgrading to Exchange 2019 or Exchange Online.

The following vulnerabilities are addressed with these Security Updates:

VulnerabilityImpactSeverity
CVE-2023-28310Remote Code ExecutionImportant
CVE-2023-32031Remote Code ExecutionImportant

More information regarding CVE’s can be found in the Security Update Guide.

The Security Update downloads en knowledgebase articles can be found here:

Exchange versionDownloadKB article
Exchange 2019 CU13https://www.microsoft.com/en-us/download/details.aspx?id=105280KB5026261
Exchange 2019 CU12https://www.microsoft.com/en-us/download/details.aspx?id=105281KB5026261
Exchange 2016 CU23https://www.microsoft.com/en-us/download/details.aspx?id=105282KB5025903

Some remarks about these Security Updates:

  • When possible, try to run the latest Cumulative Update for Exchange 2016 or Exchange 2019.
  • Exchange Security Updates are cumulative, so a Security Update contains all fixes that were released in earlier Security Updates (for a specific Exchange Cumulative Update).
  • Exchange Security Updates are specific for an Exchange Cumulative Update, so you cannot install an Exchange Security Update for Exchange 2019 CU13 on an Exchange 2019 CU12 server.
  • Security Updates must be installed on hybrid servers as well, even if there are no mailboxes anymore on these hybrid servers.
  • If you have a management server with only the Exchange server management tools installed, you must install Security Updates as well.
  • Of course, test Security Updates in a test environment first.
  • Use the Microsoft Exchange Healthchecker script (https://microsoft.github.io/CSS-Exchange/Diagnostics/HealthChecker/) to check the status of your Exchange server and if additional actions are needed.