Category Archives: Exchange

Exchange Server OWA and ECP not working

Since a couple of days my OWA and ECP are not working anymore. This happens on both Exchange 2019 CU11 and Exchange 2016 CU22, in two AD sites both with external access. I didn’t notice before since Outlook Mobile and Outlook on the desktop continue to work. After logging in, the Something went wrong message appears, in the navigation bar you can see it is an Error 500 message.

I think (but I’m not sure) that this started after applying the latest November 2021 Security Updates and this is usually caused by starting the Security Update without elevated privileges. There’s an Microsoft article about this: OWA or ECP stops working after you install a security update – Exchange | Microsoft Docs.

I am pretty sure that I installed the Security Update with elevated privileges, but also after reinstalling the Security Update (with elevated privileges!) the error continues. The Microsoft article also mentions settings in IIS Manager (Application settings > BinsearchFolder), but that was not the issue (settings were ok).

When authentication fails, two entries are written in the Application Eventlog, EventID 1003 (MSExchange Front End HTTP Proxy) and EventID 1309 (ASP.NET 4.0.30319.0). The latter clearly shows it has something to do with certificates:

It turned out that the Exchange Server Auth Certificate was expired, just a few days ago. You can see this when running the following command:

[PS] C:\>(Get-AuthConfig).CurrentCertificateThumbprint | Get-ExchangeCertificate | Format-List

AccessRules        : {System.Security.AccessControl.CryptoKeyAccessRule,
                     System.Security.AccessControl.CryptoKeyAccessRule,
                     System.Security.AccessControl.CryptoKeyAccessRule}
CertificateDomains : {}
HasPrivateKey      : True
IsSelfSigned       : True
Issuer             : CN=Microsoft Exchange Server Auth Certificate
NotAfter           : 11/28/2021 12:23:07 AM
NotBefore          : 12/24/2016 12:23:07 AM
PublicKeySize      : 2048
RootCAType         : Unknown
SerialNumber       : 268E6FB9B7312AB24AAA6BA76D06190D
Services           : SMTP
Status             : Invalid
Subject            : CN=Microsoft Exchange Server Auth Certificate
Thumbprint         : A7F9CAA2C9016DB2A80F1E2972E2ED0E2FAE089D

As shown in the following screenshot:

Use the New-ExchangeCertificate command to create a new self-signed certificate for authentication purposes:

[PS] C:\>New-ExchangeCertificate -KeySize 2048 -PrivateKeyExportable $true -SubjectName "cn=Microsoft Exchange Server Auth Certificate" -FriendlyName "Microsoft Exchange Server Auth Certificate" -DomainName @()

Confirm
Overwrite the existing default SMTP certificate?

Current certificate: 'D7081FC32B9BFBEF0C0581584976F690D5F86E74' (expires 11/30/2026 5:21:19 PM)
Replace it with certificate: '309263C8C5B2DA9612E8A6FA9FFFCDEBAC93335D' (expires 11/30/2026 9:00:56 PM)
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): n

Thumbprint                                Services   Subject
----------                                --------   -------
309263C8C5B2DA9612E8A6FA9FFFCDEBAC93335D  ....S..    CN=Microsoft Exchange Server Auth Certificate

As shown in the following screenshot:

When the certificate is created, the AuthConfig needs to be configured, it needs to be published and the old (and expired) certificated needs to be removed. Use the Set-AuthConfig command to achieve this:

[PS] C:\>Set-AuthConfig -NewCertificateThumbprint 309263C8C5B2DA9612E8A6FA9FFFCDEBAC93335D -NewCertificateEffectiveDate (Get-Date)

Confirm
The new certificate effective date is not at least "48" hours in the future and may not be deployed on all necessary
servers. Do you wish to continue?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
[PS] C:\>Set-AuthConfig -PublishCertificate
[PS] C:\>Set-AuthConfig -ClearPreviousCertificate

As shown in the following screenshot:

Restart the Microsoft Exchange Service Host service and perform an IISRESET. If you cannot run IISRESET you can also recycle both the OWA and ECP App pool:

[PS] C:\> Restart-WebAppPool MSExchangeOWAAppPool
[PS] C:\> Restart-WebAppPool MSExchangeECPAppPool

The certificate is stored in Active Directory in CN=Auth Configuration, CN=, CN=Microsoft Exchange, CN=Services,DC=, DC= as shown in the following screenshot:

Since it is published in Active Directory, the new certificate will be automatically available for all Exchange servers in your organization. It can take up to an hour before it is fully published and available, so don’t worry when it doesn’t work immediately.

Please be aware that you do this only on one Exchange server. If you (accidentally) do this on multiple Exchange servers, you will see multiple Auth Certificates appear on your Exchange server. But only the last one created will be active though.

Only two steps remain:

  • Remove the old Auth Certificate on all Exchange servers. You can do this using EAC or using PowerShell (Remove-ExchangeCertficate -Server -Thumbprint <old certificate).
  • Run the Hybrid Configuration Wizard again to update the new certificate in Azure Active Directory.

Exchange security updates November 2021

I have been away for a couple of days, but you already might have seen that Microsoft released a number of Security Updates for Exchange 2019, Exchange 2016 and Exchange 2013, but only for the last two Cumulative Updates (as always).

Security Updates are available for the following products:

Exchange versionDownloadKnowledge Base
Exchange 2019 CU11https://www.microsoft.com/en-us/download/details.aspx?id=103643KB5007409
Exchange 2019 CU10https://www.microsoft.com/en-us/download/details.aspx?id=103642KB5007409
Exchange 2016 CU22https://www.microsoft.com/en-us/download/details.aspx?id=103644KB5007409
Exchange 2016 CU21https://www.microsoft.com/en-us/download/details.aspx?id=103645KB5007409
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103646KB5007409

The following vulnerabilities are addressed in these updates:

Security Updates are CU specific and can only be applied to the specific Cumulative Update. When trying to install a Security Update for another CU, an error message will be returned.

Security Updates are also cumulative, so this Security Update contains all previous security updates for this specific CU. There’s no need to install previous Security Updates before this Security Update.

As always, after downloading a Security Update, start the Security Update from a command prompt with elevated privileges (‘Run as Administrator’) to prevent an erratic installation. This does not apply when installing a Security Update via Windows Update or WSUS.

Exchange Security Updates October 2021

On October 12, 2021 Microsoft released Security Updates for vulnerabilities found in Exchange server 2013 CU23, Exchange server 2016 (CU21/CU22) and Exchange server 2019 (CU10/CU11). Severity is marked as ‘important’.

If you are running one of these versions, it is recommended to apply these security updates. Please note that the security updates are CU specific, and these are not interchangeable. Security updates are also cumulative, so these security updates contain all previous security updates for the same cumulative update. If you are running an older version of Exchange, it is strongly recommended to upgrade to the latest Cumulative Update and apply the security updates. You can use the healthchecker script to inventory your environment.

Please use the Microsoft Security Update Guide for more specific information about the vulnerabilities.

As always, after downloading the security updates, start the installation from an elevated command prompt (‘run as administrator’). This does not apply when installing from Windows Update or WSUS. And of course, please the security updates in a test environment first before installing in production.

You can download the security updates for the following products here:

This computer is a member of a database availability group (DAG)

When uninstalling an Exchange server that is part of a Database Availability Group, you perform several steps to remove the server from the DAG and maybe even delete the DAG itself.

So, after decommissioning the DAG I wanted to uninstall Exchange, but the readiness check failed with the following error message:

This computer is a member of a database availability group (DAG). It must be removed from the DAG before you can uninstall Exchange. For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.CannotUninstallClusterNode.aspx

Unfortunately this link has not been updates since 2015, and does not contain any useful information.

I decommissioned the DAG earlier as you can see in the following screenshot, but it still failed the readiness check:

When using the Get-DatabaseAvailabilityGroup command nothing showed up, and also when checking Active Directory with ADSI Edit (CN=Organization, CN=Administrative Groups, CN=Exchange Administrative Group (FYDIBOHF23SPDLT), CN=Database Availability Groups) nothing is shown.

The DAG is using bits of Windows failover clustering, so the culprit must be there. When executing the Get-ClusterNode and Get-Cluster commands in PowerShell, some DAG leftovers (on this particular machine) showed up:

Something must have gone wrong during decommissioning of the DAG, although no error were shown on the console.

To remove the cluster node (and thus the cluster) from this machine, execute the following command in PowerShell:

PS C:\> Remove-ClusterNode EXCH12 -Force

After this step uninstalling Exchange succeeds.

Exchange Quarterly Updates: Exchange 2019 CU11 and Exchange 2016 CU22

On September 28, 2021 Microsoft released their quarterly updates for Exchange server, Exchange 2019 CU11 and Exchange 2016 CU22. Despite earlier communications a new CU for Exchange 2016 is released as well.

Besides normal fixes, a new feature is introduced in these CUs as well, the Exchange Emergency Mitigation Server or EEMS. EEMS is a new service that can mitigate new security breaches when they arise. EEMS connects to a Microsoft endpoint (https://officeclient.microsoft.com/getexchangemitigations) and when needed, downloads and installs available mitigations. It performs a check once an hour. If you don’t feel comfortable with this, it is possible to disable this on an organization level 😉

Also new in Exchange 2019 CU11 and Exchange 2016 CU22 is telemetry regarding the mitigation service. When configured, it will automatically upload mitigation related service to Microsoft. Again, this can be disabled as well using the license agreement (enabled by default).

When installing this update you will see change in the License Agreement:

The default is I accept the license agreement and will share diagnostics data with Microsoft (recommended), but you can select other as well of course.

When using the unattended install, a new switch is used for accepting the License Agreement.

  • /IAcceptExchangeServerLicenseTerms_DiagnosticDataON – when you allow to upload diagnostics data to Microsoft
  • /IAcceptExchangeServerLicenseTerms_DiagnosticDataOFF – when you do not allow to update diagnostics data to Microsoft.

There are also two new prerequisites when installing Exchange 2019 CU11 or Exchange 2016 CU22. Prerequisite software contains now the ‘IIS URL Rewrite Module’ which needs to be installed. The second one is connectivity to the internet for accessing the mitigation service endpoint.

The setup application will check for these prerequisites and will generate an error when they are not met:

Note. The internet connectivity is not shown in this screenshot.

The ‘IIS URL Rewrite Module’ can be downloaded from https://download.microsoft.com/download/1/2/8/128E2E22-C1B9-44A4-BE2A-5859ED1D4592/rewrite_amd64_en-US.msi

Using PowerShell you can download the module, store it in the C:\Install directory and install it unattended using the following commands:

Start-BitsTransfer -Source "https://download.microsoft.com/download/1/2/8/128E2E22-C1B9-44A4-BE2A-5859ED1D4592/rewrite_amd64_en-US.msi" -Destination C:\Install
Start-Process -FilePath "C:\Install\ rewrite_amd64_en-US.msi " -ArgumentList "/q" -Wait

Updating the Exchange server to this latest CU is not different compared to earlier versions (except for the license agreement switch):

Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms_DiagnosticDataON
Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms_DiagnosticDataON
Setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms_DiagnosticDataON

Setup.EXE /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataON

Note. There are no schema changes when upgrading from Exchange 2019 CU10 or Exchange 2016 CU21, but there are changes when upgrading from previous releases.

After installing the updates, you will see the new services when opening the services MMC snap-in:

Or when using the Get-Service MSExchange* PowerShell command:

To check the status in the Exchange organization, you can use the Get-OrganizationConfig | Select mitigations command:

To disable the mitigation service, execute the following command:

Set-OrganizationConfig -MitigationsEnabled:$False

By default, only one mitigation is installed, this is the EEMS heartbeat probe. You can check the installed mitigations by navigating to the Exchange scripts directory and execute the Get-Mitigations.ps1 script:

As with any Cumulative Update, please test this CU in your lab to see if all works well for your environment. Also have a look at the telemetry configuration (is that allowed in your organization?) and at the automatic configuration changes made by the EEMS (I can hear CISO starting to complain).

More information and downloads regarding the Cumulative Updates can be found here: