All posts by jaapwesselius

Exchange

Office Online server – Sorry there was a problem and we can’t open this document

2 Comments

For a current project I am working with Exchange 2019 and for OWA we want to implement Office Online Server. I did this in the past and blogged about it (Install Office Online Server 2016) so I thought it should not be a big deal.

Installed Windows 2016, installed prerequisite software, configured an SSL certificate, installed Office Online Server and created a new Office Web Apps farm.

After testing the https://fqdn/hosting/discovery and configured the organization configuration everything must be good.

When opening an attachment in OWA I do see the OOS environment, it tries to open a document and then generates this error:

“Sorry, there was a problem and we can’t open this document. If this happens again, try opening the document in Microsoft Word.”

When opening an Excel attachment, I get the following error message:
“Unable to open the file. We couldn’t find the file you wanted. It’s possible the file was renamed, moved or deleted.”

I know Office Online Server is sensitive for SSL certificates, but this was a regular Digicert certificate. Name resolution was fine as well. But the check https://fqdn/op/generate.aspx failed as well with the following (pretty useless) error:

“Server Error. We’re sorry. An error has occurred. We’ve logged the error for the server administrator.”

Unfortunately, nothing useful in the eventlog, or in the ULS logging on the Office Web Apps server. Asked colleagues, but they had only experience with Exchange 2016 and OOS.

After two days of searching, fiddling with the server, checking .NET versions (Windows 2016 comes with a newer version of .NET then required by Office Online Server), rebuilding the Office Online Server several times I realized it might be a TLS 1.2 issue. Exchange 2019 is using TLS 1.2 only by default, whereas Exchange 2016 can use multiple versions of TLS.

So, on the Windows 2016 server with OOS, I enabled strong cryptography in .NET and disabled older versions of TLS on Windows to fix the issue.

To enable strong cryptography in the .NET Framework, add the following registry key:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

To disabled older versions or TLS, add the following registry keys:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0]
@="DefaultValue"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\PCT 1.0\Server]
@="DefaultValue"
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0]
@="DefaultValue"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 2.0\Server]
@="DefaultValue"
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0]
@="DefaultValue"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]
@="DefaultValue"
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0]
@="DefaultValue"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]
@="DefaultValue"
"Enabled"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1]
@="DefaultValue"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server]
@="DefaultValue"
"Enabled"=dword:00000000

After rebooting the Office Online Server, it worked as expected.

Exchange

Exchange server patching performance and windows defender

Leave a comment

Patching an Exchange server, whether it be Windows Update, a Cumulative Update or a Security Update always takes a long time. When looking at the task manager, it is always the Antimalware Service Executable (Windows Defender Antivirus Service) that is responsible for this. It just consumes a lot of processor cycles:

To overcome this and speed up the overall performance of patching the Exchange server you can temporarily disable Windows Defender.

For Exchange 2016 running on Windows 2016 follow these steps:

Start | Settings | Update and Security | Windows Defender

For Exchange 2019 running on Windows 2019 follow these steps:

Start | Settings | Update and Security | Windows Security | Open Windows Security I Virus & Threat protection I Manage Settings

And switch Real-time protection to off as shown in the following screenshot:

Much easier is using PowerShell, just execute this command:

Set-MpPreference -DisableRealtimeMonitoring $True

When patching the Exchange server you will notice how much faster it will be. When patched and rebooted, enable Windows Defender by executing the following PowerShell command:

Set-MpPreference -DisableRealtimeMonitoring $False

You can check the status of Windows defender using one of the following commands:

Get-MpPreference | select DisableRealtimeMonitoring
Get-MpComputerStatus

Check the output for RealTimeProtectionEnabled, this should be set to True. As a sidenote, there is a lot of other interesting information when executing Get-MpComputerStatus for anti-malware.

Exchange

May 2021 Exchange Server security updates

Leave a comment

On May 11, 2021 Microsoft released new Security Updates for the following Exchange server versions:

  • Exchange Server 2013 CU23
  • Exchange Server 2016 CU19 and CU20
  • Exchange Server 2019 CU8 and CU9

The following vulnerabilities have been addressed:

VulnerabilityCategorySeverity
CVE-2021-31209SpoofingImportant
CVE-2021-31207Security Feature BypassModerate
CVE-2021-31198Remote Code Execution Important
CVE-2021-31195 Remote Code Execution Important

Personally, I am happy to see no critical and zero-day issues have been found, no immediate action on Tuesday night this time 😊. However, these are still important security updates so you must install them as soon as possible.

These Security Updates are only available for the Exchange versions mentioned above. If you are on an older version of Exchange, you must first upgrade your Exchange servers to the latest CU and then deploy these Security Updates. Security Updates are cumulative, to a Security Update contains all previous fixes for this specific Cumulative Update.

A couple of remarks:

  • If you are running Exchange Hybrid, even if you have all your mailboxes in Exchange Online and use the on-premises Exchange server only for management purposes, you still must deploy these Security Updates on the Hybrid Server. If you have an Exchange management server (with only the management tools installed) you do not need to install the Security Updates.
  • Start the Security Update from a command prompt with elevated privileges. If you do not use elevated privileges, setup will fail and leave your Exchange server in an unknown state. Known problems here are with OWA and EAC. This does not apply when installing the Security Update using Windows Update or WSUS.
  • When the installation of the Security Update has finished it does not ask for a reboot although this is needed, so reboot the server when finished.

And the downloads:

Exchange

April 2021 Exchange Server Security Updates

2 Comments

There we go again…. Last week there has been some rumor going on about pwn2own 2021, some kind of security contest to find any security issues in software products and according to this statement taken from the pwn2own site, vulnerabilities were found in Exchange:

“SUCCESS – The DEVCORE team combined an authentication bypass and a local privilege escalation to complete take over the Exchange server. They earn $200,000 and 20 Master of Pwn points.”

Today Microsoft released security updates for Exchange 2013, Exchange 2016 and Exchange 2019 that addresses security vulnerability found recently. The following Remote Code Execution vulnerabilities are fixed with these updates:

You can find more information and the download links in the following table.

Exchange versionDownloadKB Article
Exchange 2019 CU9https://www.microsoft.com/en-us/download/details.aspx?id=103004KB5001779
Exchange 2019 CU8https://www.microsoft.com/en-us/download/details.aspx?id=103003 KB5001779
Exchange 2016 CU20https://www.microsoft.com/en-us/download/details.aspx?id=103002 KB5001779
Exchange 2016 CU19https://www.microsoft.com/en-us/download/details.aspx?id=103001 KB5001779
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103000 KB5001779

Notes:

  • At this moment no active exploits using these vulnerabilities are reported.
  • These vulnerabilities only concern Exchange 2013/2016/2019 on-premises. Exchange Online is not vulnerable because of its different architecture. Please remember that Exchange Online uses a different codebase.
  • Updates are specific for Cumulative Updates, an update for CU9 cannot be installed on CU8. The CU version is in the name of the update.
  • Updates are cumulative, so these updates also contain all previous updates for this CU versions.
  • If you are running Exchange hybrid you need to update the hybrid servers as well, even when all mailboxes are in Exchange Online.
  • Previous mitigation scripts like EOMT will not mitigate the April 2021 vulnerabilities.
  • Start the updates from a command prompt with elevated privileges. If you do not, the update can finish successfully (or report no errors) but under the hood stuff will break. When updating from Windows Update there’s no need to use elevated privileges.
  • Use the Exchange Server Health Checker script (available from Microsoft Github) for an inventory of your Exchange environment. The script will return if any servers are behind with Cumulative Updates and Security Updates.
  • More information can be found on the Microsoft Security Response Center (MSRC).

Exchange

Rebuild your Exchange Server (after HAFNIUM infection)

12 Comments

If you are unlucky and your Exchange server is infected because of the HAFNIUM zero-day vulnerability, you must nuke your Exchange server and rebuild it. And I already got the first questions on how to do this. It is not that difficult and does not take days of time if some prerequisites are met of course.

Note. This blog was written with the HAFNIUM infected machines in mind, but from a procedural point of view it can be used for every disaster recovery scenario of course. Also, I used Windows 2012 R2 and Exchange 2013 for this blog, but this procedure can also be used for Exchange 2016 and Exchange 2019.

Basically, what happens is that the old server is forcibly removed (shutdown, delete VM) but that the computer account in AD is not deleted but reset (shown in screenshot below). A new server is built, same OS, same patching, same configuration, same computername and joined to the domain. This way the SID will remain the same, which makes recovering a lot easier.

Rebuilding Windows server including all prerequisite software and Windows patches will take most of the time in this entire process.

You must use the same server version when rebuilding the server. So, if you old server was running Windows 2012 R2, then you MUST use Windows 2021 R2 again. If you try different, bad things will happen.

Exchange is a bit more flexible these days. You can use the /RecoverServer option with a newer Cumulative Update. For example, when your old server is running Exchange 2016 CU11, you can do a /RecoverServer with Exchange 2016 CU19. The only drawback here is that the CU11 version information is stored in Active Directory, and even when CU19 is used, it will show up as CU11 for the AdminDisplayVersion. But that’s cosmetic and will automatically be corrected the next time a new CU is installed.

What you CANNOT do is use /RecoverServer to upgrade to a newer version of Exchange, like upgrading from Exchange 2013 to Exchange 2016. Again, bad things will happen.

Storage with Mailbox database need of course to be kept. If configured on separate disks of VHD’s you must connect these to the new server. Use the same drive letters or mount points. In the screenshot below the old disk with the Mailbox database is mounted to the new VM with Windows 2012 R2. Exchange 2013 is not yet installed on this server.

Note. Instead of my approach (connecting the existing storage) it is also possible to recover the server and restore the mailbox database later from backup. This can range from any backup application to a file level backup of your .EDB file. The latter needs extensive knowledge about the Mailbox database internals and how to deal with that.

When Windows is up and running it is time to install the server. Use the same Cumulative Update for this and use the unattended setup of Exchange with the RecoverServer option. Open a command prompt with elevated privileges and execute the following command:

Z:> Setup.exe /mode:RecoverServer /IAcceptExchangeServerLicenseTerms

Z:\ in this case is the DVD drive letter, but you can use a different drive letter of course.

There’s one catch here. If Exchange is installed in a different location, for example on the D:\ drive, you must use the /TargetDir option to specify the location where the Exchange binaries must be installed.

The /RecoverServer will retrieve all configuration information from Active Directory (instead of using the default settings) and install Exchange using this configuration. For example, all Mailbox database information is stored in Active Directory, so when setup has finished, the ‘new’ server has all the correct Mailbox database information. Even better, after a reboot it will even automatically mount the Mailbox databases.

Some settings are not stored in Active Directory, especially when looking at server specific configuration that are stored in local .config files. OWA customization and SSL Certificates for example that are configured on the server are lost. So, existing documentation of the Exchange server is vital here to quickly rebuild the Exchange server.

After reinstallation, the server was rebooted, I had to mount the Mailbox database manually (but without issues). The following settings I had to configure manually:

  • Import the SSL certificate and bind it to the Exchange services.
  • Set the Exchange Virtual Directories (I did not expect this one).
  • Relocate the SMTP Queue database.

To my surprise the SMTP relay connector that was configured on the Exchange server was available instantly and working correctly.

Summary

If you have to rebuild your Exchange server, whether it be it crashed or got infected or something, you can use the Setup.exe /RecoverServer option of Exchange (Exchange 2013 and up). This will retrieve a lot of the information from Active Directory, and if you have the Mailbox databases available you can use these directly without any restore from backup activities.

There are some settings that are still manually configured after the server is recovered, so proper documentation is important to have to make life much easier when recovering.

Last updated: March 10, 2021