So I’m working on a nice Public Folder migration from Exchange 2016 to Exchange Online. Last week all preparations were performed and this morning I was planning to start the migrationbatch. Fourteen Public Folder mailboxes in Exchange 2016 to eighteen Public Folder mailboxes in Exchange Online. What could possibly go wrong….
Creating the new endpoint for the migrationbatch failed with an error saying “Multiple mailbox users match identity “PFMailbox1″. Specify a unique value” as shown in the following screenshot:
This is strange since there’s no mailbox user named “Mailbox1”, there’s only one Public Folder mailbox named “Mailbox1” and that is holding the Public Folder hierarchy. But, last week I have been preparing the Public Folder migration, and one step was to create a Public Folder mailbox, just to see if it would work. I deleted the Public Folder mailbox, but it is a soft deleted state, and this conflicts with the creation of the Public Folder migration endpoint.
To find the Public Folder mailboxes including the soft deleted mailboxes, execute the following command against Exchange Online:
After the latest Windows Updates (June 2022) I could not logon to Office 365 using a federated domain (Microsoft Teams in particular, the mailboxes are in Exchange 2019 for this domain), regular domains did not experience any issues.
Trying the regular ADFS URLs did not help, the site was not available:
When checking the WAP server I noticed the WAP service was not running and was not willing to start. At the same time, EventID 224 was logged in the eventlog with the “The federation server proxy configuration could not be updated with the latest configuration on the federation service” error messages as shown in the following screenshot:
Especially the additional data reveals a lot:
Retrieval of proxy configuration data from the Federation Server using trust certificate with thumbprint ‘76426A7DB45871F25A7BD5D883F2C5196B82E0DA’ failed with status code ‘Unauthorized’. The remote server returned an error: (401) Unauthorized.
At the same time, Event ID 276 is logged on the internal ADFS Server:
Obviously, the trust between the proxy server and the ADFS server is broken (it has been some time when I look at the timestamps, this happens in a test environment 😊) so the trust relationship needs to be re-established.
This can be done using the wizard in the Remote Access Management Console:
If you get a warning message like “Web Application Proxy could not connect to the AD FS configuration storage and could not load the configuration” you must change the ProxyConfigurationStatus in the registry (HKLM\Software\Microsoft\ADFS) from “2” to “1” as shown in the following screenshot.
Follow the wizard, select the appropriate certificate, check the changes and click the Configure button as shown in the following two screenshots:
When you check the eventlog, you’ll see Event ID 252 with the configuration changes:
And you can see that the ADFS Proxy server can authenticate successfully:
On August 9, 2022 Microsoft has released important Security Updates for Exchange 2013, Exchange 2016 and Exchange 2019 that are rated ‘critical’ (Elevation of Privileges) and ‘important’ (Information Disclosure).
This security update rollup resolves vulnerabilities found in Microsoft Exchange Server. To learn more about these vulnerabilities, see the following Common Vulnerabilities and Exposures (CVE):
CVE-2022-21979 – Microsoft Exchange Information Disclosure Vulnerability
CVE-2022-21980 – Microsoft Exchange Server Elevation of Privilege Vulnerability
CVE-2022-24477 – Microsoft Exchange Server Elevation of Privilege Vulnerability
CVE-2022-24516 – Microsoft Exchange Server Elevation of Privilege Vulnerability
CVE-2022-30134 – Microsoft Exchange Server Elevation of Privilege Vulnerability
This Security Update introduces support for Extended Protection. Extended protection enhances authentication to mitigate ‘man in the middle’ attacks. Extended protection is supported on the latest version of Exchange 2016 and Exchange 2019 (2022H1) and the August 2022 Security Update (this one) so it is vital to bring your Exchange servers up-to-date.
Be aware of the following limitations:
Extended protection is only supported on the current and previous versions of Exchange (i.e. Exchange 2016 CU21/CU21 and Exchange 2019 CU12/CU11) and Exchange 2013 CU23 with the August 2022 SU installed
Extended protection is not supported on hybrid servers with the hybrid agent.
Extended protection is not supported with SSL Offloading. SSL Re-encrypt (also knows as SSL Bridging) is supported, as long as the SSL certificate on the load balancer is identical to the SSL certificate on the Exchange servers.
If you still have Exchange 2013 in your environment and you are using Public Folders, make sure your Public Folders are hosted on Exchange 2016 or Exchange 2019.
Note. Make sure you have your Exchange server properly configured with all related security settings. Use the latest HealthChecker.ps1 script to find any anomalies in your Exchange configuration. If you fail to do so, the script to enable Extended Protection will fail with numerous error messages.
Enable Extended Protection
First off, make sure you have the latest Cumulative Update installed on all your Exchange servers and install the August 2022 Security Updates on all your servers, including the Exchange 2013 servers.
Another important thing is that you must make sure that TLS settings across all Exchange servers are identical. You can use the healthchecker.ps1 script to figure out if this is the case. Personally, it took me quite some time to get this right.
The easiest way to configure Extended Protection is by using the ExchangeExtendedProtectionManagement.ps1 script (which can be found on github). This script can enable Extended Protection on all Exchange servers in your organization, but by using the -SkipExchangeServerNames option you can exclude certain Exchange servers (for example, Exchange 2013 servers or servers running the hybrid agent). There’s also the -ExchangeServerNames option which lets you specify which servers to enable the Extended Protection on.
As long as I can remember I have been creating, updating, renewing, exporting and importing Exchange certificates on Exchange servers.
This morning I had to renew my own Exchange certificate, and my PowerShell command Export-ExchangeCertificate failed on the -FileName option so it would not accept the option to store the file somewhere. This is strange, because in our Exchange 2016/2019 book that was released less then a year ago we were able to use the -FileName option.
It turned out that for the Export-ExchangeCertificate and Import-Certificate the -FileName option was removed because of security concerns. In more detail, the -FileName option accepts a UNC path which makes it possible for compromised servers to access other servers using UNC paths.
The way to export a certificate in Exchange 2016 CU23 and Exchange 2019 CU12 (and higher) is to import the certificate in a variable and store this in a file:
The last couple of days I have been working with multiple customers on SMTP relay in Exchange 2016 during a migration from Exchange 2010 to Exchange 2016. The last time I did that was with Exchange 2010, almost 9 years ago https://jaapwesselius.com/2012/05/25/smtp-relay-in-exchange-2010/ and things have changed over the years. The changes in Exchange 2016 are carried forward in Exchange 2019…. Oh, and this is true for Exchange 2013 as well, but since Exchange 2013 is already out of support and none of my customers is using Exchange 2013 I’ll skip this….
Exchange 2010 has the Hub Transport service for SMTP, and this is using port 25 for communicating with other SMTP hosts. Exchange 2016 and Exchange 2019 have two services for SMTP transport:
Front-End Transport service (FETS) listening on port 25. This is where other SMTP hosts connect to. In the Exchange 2016 Admin Center the FETS Receive Connector is identified as Default Frontend <server> . In Exchange 2013, this service was running on the Client Access Server.
Hub Transport Service listening on port 2525. This is a back-end service used by FETS and other Exchange Hub Transport back-end services. Clients are not expected to use the Hub Transport service on port 2525. In the Exchange Admin Center this Receive Connector is identified as Default <server> . In Exchange 2013, this service was running on the Mailbox server.
These connectors are shown in the following screenshot. The Default Frontend Receive Connector (on port 25) is selected, the red arrow points to the Hub Transport Receive Connector on port 2525.
Note. The Client Frontend Receive Connector in the screenshot is listening on port 587 and is used for authenticated SMTP clients like Mozilla Thunderbird.
SMTP Relay in Exchange 2016 and 2019
The Default Frontend Receive Connector allows all SMTP clients to connect to it and drop email messages for local delivery. You don’t want to configure this connector to relay SMTP message to external domains, this is known as an ‘open relay’ and this is the number one reason to be put on every blacklist available on the Internet. You can do this and restrict access based on IP addresses, but I strongly recommend against changing the default connectors. Leave the inbound SMTP traffic end up on the Default Frontend Receive Connector and create a dedicated connector for SMTP relay traffic.
There are two ways to create such a relay connector:
Create a dedicated receive connector (on Frontend Transport, not on Transport Service), restrict by IP address and add the Ms-Exch-SMTP-Accept-Any-Recipient permission on the NT AUTHORITY\ANONYMOUS LOGON security principal. This is what I have shown in the blog mentioned earlier, and this is only possible using Exchange PowerShell. Sending hosts are considered anonymous, and anti-spam and message size limits are applied.
Create a dedicated receive connector (again on the Frontend Transport), restrict by IP address, and add the Exchange Servers and Externally Secured authentication mechanism to the connector. In this scenario, sending hosts are considered as authenticated senders, and email messages bypass anti-spam and message size limits. And it’s easy to configure using the Exchange Admin Console.
Since the first option is already documented, I will continue with the second option. Personally, I like to do this with PowerShell and the command to create such a connector and configure it are like these:
When you check the connector using the Exchange Admin Center, you can see that the authentication mechanism is set correctly as shown in the following screenshot:
It is also possible to create a new connector using the Exchange Admin Center. In the EAC, navigate to mail flow and select the receive connectors tab and click the + icon. Follow the wizard, give the new connector a proper name, select Frontend Transport and Custom, and restrict by IP address as shown in the following screenshots:
When created, open the new receive connector, select security and configure the authentication mechanmism to Externally secured and Exchange servers as shown in a previous screenshot.
It is now possible to relay SMTP messages from the server with IP address 10.38.96.15. Using Telnet on port 25, you will see something like this:
When trying to relay from another server (which is not listed in the Remote Network Settings) it will fail with the 550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain error as shown in the following screenshot:
So in short, do not configure the default receive connector in such a way that it will relay messages outside of the Exchange server. When you need to use SMTP relay, create a dedicated connector.
The first and most secure option is to create a new receive connector, restrict by IP address and configure the Ms-Exch-SMTP-Accept-Any-Recipient permission. Anti-spam and message size limits are applied, but it can only be configured using PowerShell (and thus more complex).
The second on is to create a new receive connector, restrict by IP address and configure the authentication mechanisms. Easier to configure (using EAC) but less secure: anti-spam and message limits are not applied.