Category Archives: Exchange

Exchange, Office365

Email not delivered to DANE enabled domains in Office 365

Leave a comment

The servicedesk got complaints that email was not delivered to an organization, and that an NDR was never generated. The sending user never knew this, until the sender and receiver talked on the phone. Our organization was Exchange 2016 on-premises with a Cisco IronPort as a gateway to the Internet. The other (receiving) organization was in Exchange Online.

The receiving organization was in Exchange Online, and had already enabled DANE for inbound email messages (see my previous blog post on this). I checked multiple organizations in Exchange Online that have DANE enabled (including hotmail.nl) and they all failed.

One organization has a Fortra Clearswift in front of their environment that has DANE enabled, and we were able to send email to this particular domain.

And to make it more complex, other organizations with an IronPort gateway were able to successfully send email messages to these domains.

At this point still no clue whether it is a Microsoft issue, or an IronPort issue or something specific to our organization.

When checking the DANE SMTP service for the domains involved everything looks fine as shown in the following screenshot:

When checking the IronPort logs, the following cryptic and non-explaining error was logged:

MID 1614647 (DCID 600113) DANE failed for Hotmail.nl. Reason: 4.0 - Other network problem.

Also shown in the following screenshot:

The same error was logged for the other DANE enabled domains as well.

So, DANE fails on the IronPort, but the tools to check DANE all reported DANE was good. Also, I was able to send mail to these domains using Gmail. I always say when it works in Gmail, everything is ok.

When checking the DANE configuration on the command prompt on the IronPort it looks more like a DNS issue as shown in the following screenshot:

But when checking the DNS record (_25._tcp.exchangelabs-nl.y-v1.mx.microsoft)
with MXToolbox, everything is green again as shown in the following screenshot:

After checking with the network department it turned out that there was an IPS solution implemented and the network engineer knew about an old CVE, dating back to 2013 (CVE-2013-4466) that warns for a situation where a buffer flow can occur in the dane_query_tlsa function when more that four DANE entries are returned.

The CVE is 11 years old, but the IPS still had this implemented, and when more than four entries were returned, everything was discarded and the email was lost. Removing this from the IPS and everything works fine.

Note. The RFC for DANE does not mention the amount of entries that can be returned, so more than four should not be a problem.

Exchange, Exchange Hybrid

Remove the last Exchange server

7 Comments

Removing the last Exchange server in a hybrid environment was announce more than 2 years ago with Exchange 2019 CU12 and Exchange 2016 CU23 (Released: 2022 H1 Cumulative Updates for Exchange Server) but finally I’m working with a customer that wants to do this, remove that last Exchange server in a hybrid environment.

This customer not in the process of decommissioning their datacenters yet, but they do want to decommission their last Exchange 2019 server (all their mailboxes are in Exchange Online). The Domain Controllers are still running on-premises, so the Exchange 2019 server is only used for management purposes (SMTP Relay is already moved elsewhere).

The first thing is to remove the Hybrid Configuration. I wrote about that in 2020, but the article is still valid: https://jaapwesselius.com/2020/12/15/remove-exchange-hybrid-configuration/.

The second step is to install the Exchange 2019 management tools. This can be on a Domain Controller or on a Management Server. To install the Exchange 2019 management tools, the Active Directory management tools, the Visual C++ Redistributable Package for Visual Studio 2012 and the IIS6 management tools must be installed first. Execute the following commands in an elevated PowerShell window:

PS C:\> Install-WindowsFeature rsat-adds,telnet-client
PS C:\> Enable-WindowsOptionalFeature -Online -FeatureName IIS-IIS6ManagementCompatibility, IIS-Metabase -All

The Exchange Management Shell is installed, when starting this it connects to an Exchange 2019 server. But when this server is not available the Exchange Management Shell fails. To work with Exchange PowerShell, open a regular Windows PowerShell and execute the following command:

PS C:\> Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn

This uses the Exchange PowerShell module that’s installed on the management server, but it does not connect anywhere, and it does not use any RBAC configuration.
When using this, the following Exchange PowerShell commands are available:

  • *-MailUser
  • *-MailContact
  • *-RemoteMailbox
  • *-DistibutionGroup
  • *-DistributionGroupMember
  • *-EmailAddressPolicy
  • Set-User and Get-User

When enabling an existing user account with a mailbox in Exchange Online, you can use the Get-User command and pipe it to the Enable-RemoteMailbox command, like this:

PS C:\ > Get-User JaapICT | Enable-RemoteMailbox -RemoteRoutingAddress JaapICT@contoso.mail.onmicrosoft.com

The Remote Mailbox will be created and all properties will be set correctly, including the email address conforming to the existing Email Address Policy. This is clearly visible when requesting the properties of the mailbox:

PS C:\> Get-RemoteMailbox -Identity JaapICT | select name -ExpandProperty EmailAddresses

Name               : JaapICT
AddressString      : C=NL;A= ;P=Corporation;O=Contoso;S=JaapICT;
ProxyAddressString : X400:C=NL;A= ;P=Corporation;O=Contoso;S=JaapICT;
Prefix             : X400
IsPrimaryAddress   : True
PrefixString       : X400

Name               : JaapICT
SmtpAddress        : JaapICT@Contoso.com
AddressString      : JaapICT@Contoso.com
ProxyAddressString : SMTP:JaapICT@Contoso.com
Prefix             : SMTP
IsPrimaryAddress   : True
PrefixString       : SMTP

Name               : JaapICT
SmtpAddress        : jaapict@Contoso.mail.onmicrosoft.com
AddressString      : jaapict@Contoso.mail.onmicrosoft.com
ProxyAddressString : smtp:jaapict@Contoso.mail.onmicrosoft.com
Prefix             : SMTP
IsPrimaryAddress   : False
PrefixString       : smtp

Now everything is working the last Exchange 2019 server can be removed. What I normally recommend is to disable all Exchange services on the server and leave it running for one or two weeks. If any issues arise it is easy to start the Exchange services again and fix the problem.

When nothing bad happens, you can remove the last Exchange server. After disabling the last mailboxes and removing the Mailbox Databases and Send Connectors, turn off the Exchange 2019 server. DO NOT UNINSTALL Exchange 2019, but shutdown the Exchange 2019 and remove the server. That is, delete the VM or reconfigure the bare metal server with something else, but DO NOT UNINSTALL the Exchange 2019 server. The management tools that are installed still use configuration settings in Active Directory.

From this moment on you must manage your recipient in Exchange Online using the on-premises management server where the Exchange Management Tools are installed (I say on-premises server, but this can also be a VM in Azure of course, as long as it is a domain joined server you are good).

Exchange

Exchange vNext will be Exchange Server Subscription Edition

Leave a comment

Today Microsoft silently released an update to their Exchange roadmap, which includes information regarding Exchange 2019 CU15 and Exchange vNext. You can read all the Microsoft marketing stuff on the Exchange Server Roadmap Update article.

What’s new is that vNext is rebranded to Exchange Server Subscription Edition, just like we have Sharepoint Subscription Edition.

The most important part about Exchange Server Subscription Edition is that it is ‘code equivalent’ to Exchange 2019 CU15. So, if you have Exchange 2019 CU15 running later this year, then updating to vNext is just a matter of an in-place upgrade. There’s one thing we need to look out for, the underlying Operating System. If you install CU15 on Windows Server 2022 (or worse, on Windows Server 2019) and SE only supports Windows Server 2025 we will be very unhappy 🙂

What are new features in Exchange 2019 CU15 and thus Exchange Server SE?

  • Support for TLS 1.3 (which was planned for CU14).
  • Certificate management in the Admin Center.
  • Removal of the UCMA (makes sense, since there won’t be any support for Unified Messaging.
  • Removal of the MSMQ components in the setup application (MSMQ components are not needed in earlier versions of Exchange 2019, please check the Exchange 2019 requirements article).
  • Re-introducing certificate management in the Admin Center.

So, when can we expect Exchange Server Subscription Edition? As Exchange Server SE is identical to Exchange 2019 CU15 (in will include the necessary security updates of course) the only difference is the licensing of Exchange Server SE. You need a subscription license for the server, and old Client Access Licenses are no longer supported and you can use the regular Office 365 licenses for clients.

Microsoft states it will be available early Q3 2025, which means early July 2025. Since support for Exchange 2016 and Exchange 2019 will end in October 2025 Microsoft cannot afford to slip this date since you need sufficient time to upgrade from earlier versions of Exchange server.

What’s also interesting is that Microsoft is already releasing information about Exchange Server SE CU1, which should be released by the end of 2025 (can slip though).

The most interesting features in Exchange Server SE are Kerberos authentication for server-to-server authentication, the removal of Outlook Anywhere and the deprecation of Remote PowerShell. This brings Exchange server SE nicely inline with Exchange Online.

There’s one very important announcement Microsoft makes: Exchange server SE CU1 will stop supporting co-existence with ALL PREVIOUS VERSIONS of Exchange server. So, this means that in that timeframe, only Exchange Server SE CU1 (and later) will be supported and all previous versions of Exchange server must be removed from your environment.

Exchange Server SE is still approx 18 months away from now, but it is time to start thinking about your Exchange environment. Do you want to fully move to Exchange Online, or do you want to keep mailboxes on-premises in Exchange Server? If so, it’s time to start working on moving to Exchange 2019 CU14 and upgrade to CU15 later this year (or skip CU14 and move directly to CU15).

It is not a strange idea, I’m currently working with three large Exchange 2016 on-premises deployments to move them to Exchange 2019 and prepare for Exchange server SE.

So, lots of work to do the upcoming 18 months 🙂

Exchange

Hotfix Update for Exchange 2016 and Exchange 2019

Leave a comment

Wait, what? On April 23, 2024 Microsoft has released a hotfix update for Exchange 2016 and Exchange 2019 and as MVP’s we only learned about this last week.

A hotfix update or HU contains fixes for issues that might arise with a security update in Exchange server. For example, the March 2024 SU for Exchange server introduced a number of issues, and these are fixed with this HU. Besided hotfixes, a HU can also contain new features that did not make it in the last security update (SU) or Cumulative Update (CU). In this HU for example, Hybrid Modern Authentication for OWA and ECP is introduced as a new feature. Another new feature introduced in this HU is the support for ECC (Elliptic Curve Cryptography) certificates. ECC certificates however are not supported for the federation trust certificate, the Exchange server OAuth certificate and ECC certificates cannot be used when ADFS claims-based authentication is used.

The following issues are fixed in this HU:

  • “We can’t open this document” error in OWA after installing March 2024 SU
  • Search error in Outlook cached mode after installing March 2024 SU
  • OwaDeepTestProbe and EacBackEndLogonProbe fail after installing March 2024 SU
  • Edit permissions option in the ECP can’t be edited
  • Outlook doesn’t display unread message icon after installing Exchange Server March 2024 SU
  • My Templates add-in doesn’t work after installing Exchange Server March 2024 SU
  • Download domains not working after installing the March 2024 SU

You can download this hotfix update for Exchange server here:

Exchange 2019 CU14 HU2 – https://www.microsoft.com/en-us/download/details.aspx?id=106021
Exchange 2019 CU13 HU6 – https://www.microsoft.com/en-us/download/details.aspx?id=106022
Exchange 2016 CU23 HU13 – https://www.microsoft.com/en-us/download/details.aspx?id=106023

Be aware that the filename for all versions of this HU is the same (Exchange2019-KB5037224-x64-en.exe) so when downloading multiple versions make sure you store them at different locations.

A hotfix update is cumulative and includes all security features and fixes from the previous security updates. When running Exchange 2019 CU14 and you have not installed the March 2024 security update then there’s no need to install this first. Just continue with the immediate installation of this HU.

More information

Exchange

Exchange 2019 CU14

Leave a comment

On February 13, 2024 Microsoft has released a new Cumulative Update for Exchange 2019, the 2024 H1 Cumulative Update (or CU14). One major ‘new’ feature and some minor features.

Extended Protection

One of the interesting things in this update is that it by default enables Extended Protection on all virtual directories of your Exchange 2019 server.

Extended protection is not new, it was introduced in August 2022 Security Updates. At the time I already wrote about, which you can read on https://jaapwesselius.com/2022/08/09/exchange-security-updates-august-2022/.

Extended Protection is an enhancement on the existing Windows Authentication. Extended Protection mitigates authentication relay or ‘man in the middle’ attacks. It is implemented by using channel binding information using a Channel Binding Token (CBT). There’s no need to configure anything, Extended Protection is automatically configured by the CU14 setup. It is a per server setting, so it is only enabled on the server you have installed CU14 on.

There are some prerequisites that need to be met before you can enable Extended Protection. Run the healthchecker.ps1 script and check the prequisites page for Extended Protection on the Microsoft page.

If you want to install CU14, but you environment does not meet the prerequisites, you can install CU14 unattended with the /DoNotEnableEP or /DoNotEnableEPFEEWS (EPFEEWS is Extended Protection FrontEnd EWS) switch.

Please be aware that this is not the recommended approach. The recommended approach is to prepare your environment for Extended Protection, enable it on older server and then install CU14 with automatic enabling Extended Protection.

TLS 1.3

Unfortunately TLS 1.3 (which is supported on Windows 2022) did not make it in this update, and is now scheduled to be included in CU15 (2024 H2 Cumulative Update, so by the end of this year).

CVE-2024-21410

CVE-2024-21410 was also released today, and this CVE is addressed in CU14 by Extended Protection. This is also true for Exchange 2016 CU23 (EP was introduced in an earlier Security Update for Exchange 2016).

Installing Exchange 2019 CU14

CU14 does not come with an Active Directory Schema update; the Schema is still at version 17003. The Exchange configuration version is increased by one (now 16762) and the Domain version has not changed (still 13243).

Use the following commands to upgrade the Configuration and Domain Partition in your environment:

Z:\>Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms_DiagnosticDataOn

And:

Z:\>Setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms_DiagnosticDataOn

You’ll notice the following output on the console:

Exchange Setup has enabled Extended Protection on all the virtual directories on this machine. For more information visit: https://aka.ms/EnableEPviaSetup. 

We recommend running Exchange HealthChecker script to evaluate if there are any configuration issues which can cause feature breakdowns. HealthChecker script can be downloaded from https://aka.ms/ExchangeSetupHC.

At this point, nothing has changed on the virtual directories of your Exchange server so the information is not needed at this point. Running the HealthChecker script is always a good thing before running any upgrade. When using the unattended upgrade, use the following command:

Z:\>Setup.EXE /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataOn

And when needed, you can add the /DoNotEnableEP switch here. Or you can also use the GUI setup of course if you prefer to do that.

Some remarks about CU14

  • Make sure you fully understand the impact of Extended Protection. For more information check the Configure Extended Protection in Exchange server page on the Microsoft site.
  • CU14 supports the .NET Framework 4.8.1 on Windows Server 2022 (and only on Windows Server 2022)
  • This Cumulative Update contains all binaries from previous Cumulative Updates, including all previously released Security Updates.
  • Download CU14 at https://www.microsoft.com/en-us/download/details.aspx?id=105878
  • Knowledgebase article KB5035606 contains more information, including known issues with this release.
  • Before bringing in production, please test thoroughly in your test environment.
  • The only supported versions of Exchange 2019 are Exchange 2019 Cumulative Update 14 and Cumulative Update 13. Earlier versions are no longer supported (and no longer receive Security Updates)
  • Exchange 2013 is old, and out of extended support. No more Security Updates will be released for Exchange 2013. If you are still running Exchange 2013, make sure you migrate to Exchange 2019 or Exchange Online anytime soon!
  • Exchange 2016 is out of mainstream support, which means that Microsoft will no longer release new Cumulative Updates for Exchange 2016. Security Updates will be released until Extended Support for Exchange 2016 ends.