ADFS Web Application Proxy Configuration Wizard fails with trust certificate error

I was installing a new ADFS environment on Windows 2022 and the Web Application Proxy Configuration Wizard failed with the following error message:

Retrieval of proxy configuration data from the Federation Server using trust certificate with thumbprint <thumbprint> failed with status code ‘InternalServerError’

The certificate as mentioned the wizard is available on the WAP server. You can check this using the following command in PowerShell:

PS C:\> Dir CERT:\LocalMachine\My

For some reason, the WAP server is having difficulties contacting the internal ADFS server which is also running on Windows 2022. Name resolution works fine and credentials of the local administrator were ok.

One of the new features of Windows 2022 is support for TLS 1.3 and here’s the culprit. It seems like ADFS is not working correctly with TLS 1.3.

To disable TLS 1.3 on the WAP server, add the following registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
"DisabledByDefault"=dword:00000001
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Server]

As shown in the following screenshot:

After adding these registry keys, the WAP Proxy configuration wizard finished successfully.

January 2022 Exchange Security Updates

On january 11, 2022 Microsoft released new Security Updates for Exchange versions:

  • Exchange 2013 CU23
  • Exchange 2016 CU21, Exchange 2016 CU22
  • Exchange 2019 CU10, Exchange 2019 CU11

The following vulnerabilities have been addressed in these Security Updates:

No exploits have been found in the wild, but it is recommended to install these Security Updates as soon as possible.

These updates are targeted toward Exchange server on-premises, including Exchange servers used in a hybrid configuration.

Please note the following:

  • Run the Exchange Server Healthcheck script on your Exchange server to get an overview of all issues in your environment, including installed Security Updates and Cumulative Updates versions.
  • If running an old (and unsupported!) version of Exchange server, please update to the latest CU to get in a supported state and install these Security Updates.
  • When installing manually, start the update from a command prompt with elevated privileges. If you fail to do so, it will look like installation successfully finishes, but various issues will occur. This is not needed when installing using Windows Update or WSUS.
  • Security Updates are also cumulative, so this Security Updates contains all previous Security Updates for this specific Cumulative Update. There’s no need to install previous Security Updates before installing this Security Update.
  • The December 2021 Cumulative Update is postponed, check the link on the Microsoft site. Microsoft does not release Security Updates and Security Updates in the same month, so do not except a new Cumulative Update anytime soon.
  • This Security Update does not contain a fix for the Y2K22 problem that popped up on January 1, see the Email stuck in Exchange on-premises Transport Queues article which also contains the solution.
  • As always, download and deploy in your test environment to see if it all works well in your environment.
Exchange versionDownloadKnowledge base
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103857KB5008631
Exchange 2016 CU21https://www.microsoft.com/en-us/download/details.aspx?id=103856KB5008631
Exchange 2016 CU22https://www.microsoft.com/en-us/download/details.aspx?id=103855KB5008631
Exchange 2019 CU10https://www.microsoft.com/en-us/download/details.aspx?id=103853KB5008631
Exchange 2019 CU11https://www.microsoft.com/en-us/download/details.aspx?id=103854KB5008631

The FIP-FS Scan Process failed initialisation. Mail is queued on Exchange servers.

According to a post on Reddit it seems that Exchange 2016 and Exchange 2019 on-premises are queueing email messages starting januari 1, 2022 at midnight (UTC).

Besides mail queueing you will also see EventID 1106 in the application event log stating “The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error”

This might be a bug is how the date is handles inside the scan engine, causing it to fail after midnight UTC on Januari 1, 2022 (or is it December 31, 2021).

As there is no fix from Microsoft yet, the workaround is to disable anti-malware scanning on all your Exchange servers and restart the Transport service:

CD $ExScripts
.\Disable-AntiMalwareScanning.ps1
Restart-Service MSExchangeTransport

Update 1 on January 1 at 9PM GMT+1. Microsoft is aware of this issue and working on a solution. Check the Exchange team blog: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

Update 2 on January 2 at 10AM GMT+1. Microsoft has released a solution for this issue. A script can be downloaded from https://aka.ms/ResetScanEngineVersion. This will stop the services, delete the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft and the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata directories and download new scan engines. This can take a couple of minutes, but the script can be run in parallel on all your Exchange servers.

When the script has finished, check the eventlog and you should see EventID 6036 that all is well.

Although not documented, I had to reboot my Exchange servers. I’ve several other reports from people that they had to reboot too. Another thing, when you disabled the anti-malware as a workaround, you have to re-enable the anti-malware manually. You can check this using the Get-TransportAgent “malware agent” command.

It is also possible to manually update your Exchange servers, this is also documented in the Microsoft article https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

One warning though….. All Exchange 2016 and 2019 servers worldwide are suffering from this issue and they all will queue messages. Queues expire after 48 hours, so when not fixed the Exchange servers will generate NDR messages on Sunday night. Worst case scenario, millions of NDRs will be generated which in turn will result in tons of helpdesk calls. If you read this, most likely you have fixed your Exchange servers and it is the Exchange environment of the intended recipients.

Let’s hope it will be quiet again for some time now 🙂