On August 9, 2022 Microsoft has released important Security Updates for Exchange 2013, Exchange 2016 and Exchange 2019 that are rated ‘critical’ (Elevation of Privileges) and ‘important’ (Information Disclosure).
This security update rollup resolves vulnerabilities found in Microsoft Exchange Server. To learn more about these vulnerabilities, see the following Common Vulnerabilities and Exposures (CVE):
CVE-2022-21979 – Microsoft Exchange Information Disclosure Vulnerability
CVE-2022-21980 – Microsoft Exchange Server Elevation of Privilege Vulnerability
CVE-2022-24477 – Microsoft Exchange Server Elevation of Privilege Vulnerability
CVE-2022-24516 – Microsoft Exchange Server Elevation of Privilege Vulnerability
CVE-2022-30134 – Microsoft Exchange Server Elevation of Privilege Vulnerability
This Security Update introduces support for Extended Protection. Extended protection enhances authentication to mitigate ‘man in the middle’ attacks. Extended protection is supported on the latest version of Exchange 2016 and Exchange 2019 (2022H1) and the August 2022 Security Update (this one) so it is vital to bring your Exchange servers up-to-date.
Be aware of the following limitations:
Extended protection is only supported on the current and previous versions of Exchange (i.e. Exchange 2016 CU21/CU21 and Exchange 2019 CU12/CU11) and Exchange 2013 CU23 with the August 2022 SU installed
Extended protection is not supported on hybrid servers with the hybrid agent.
Extended protection is not supported with SSL Offloading. SSL Re-encrypt (also knows as SSL Bridging) is supported, as long as the SSL certificate on the load balancer is identical to the SSL certificate on the Exchange servers.
If you still have Exchange 2013 in your environment and you are using Public Folders, make sure your Public Folders are hosted on Exchange 2016 or Exchange 2019.
Note. Make sure you have your Exchange server properly configured with all related security settings. Use the latest HealthChecker.ps1 script to find any anomalies in your Exchange configuration. If you fail to do so, the script to enable Extended Protection will fail with numerous error messages.
Enable Extended Protection
First off, make sure you have the latest Cumulative Update installed on all your Exchange servers and install the August 2022 Security Updates on all your servers, including the Exchange 2013 servers.
Another important thing is that you must make sure that TLS settings across all Exchange servers are identical. You can use the healthchecker.ps1 script to figure out if this is the case. Personally, it took me quite some time to get this right.
The easiest way to configure Extended Protection is by using the ExchangeExtendedProtectionManagement.ps1 script (which can be found on github). This script can enable Extended Protection on all Exchange servers in your organization, but by using the -SkipExchangeServerNames option you can exclude certain Exchange servers (for example, Exchange 2013 servers or servers running the hybrid agent). There’s also the -ExchangeServerNames option which lets you specify which servers to enable the Extended Protection on.
Installing and configuring MCDB in Exchange 2019 has been on my bucket list for a long time, but like most organizations my Exchange servers have been running on either Hyper-V or VMware. I have seen posts on the forums where people were able to publish SSD disks on VMware to VMs, but MCDB has been targeted towards bare metal deployments. But the good news is that I managed to get two big HP boxes to play around with 🙂
MCDB was introduced in Exchange 2019 to speed up access to information in mailboxes that is used frequently. According to the Exchange Preferred Architecture, Mailbox database are stored on (relatively) slow SATA disks. When using MCDB, frequently accessed mailbox information is also stored on SSD disks instead of spinning disks and as such it is much faster to access this information. It is a cache mechanism, so the information on SSD is a copy of information on the spinning disks. If the SSD disk is lost, performance will drop but no information will be lost.
MCDB is useful when running Outlook clients in online mode, for example in a Citrix environment. OWA can also benefit from the improved performance.
MCDB is built on top of Database Availability Groups, so it is not available on single servers (I assume you don’t have a DAG with only one server). It also depends on the AutoReseed feature, so you must deploy this first before configuring MCDB.
My Servers have eight disks installed:
2 SSD disks for boot and system.
4 SAS disks (10 krpm) for Mailbox databases (I preferred SATA disks, but the servers came with these disks).
1 SSD disk for MCDB.
1 (large) SATA disk for storing other information (IIS logs, Queue database, ISO image etc.).
Three of the SAS disks contain two Mailbox databases each, the remaining SAS disk is used as a hot spare for AutoReseed.
In short, AutoReseed is using the ‘multiple mount points per volume’ option in Windows. For example, the first disk is mounted in C:\ExchVols\Vol1, but this disk is also mounted in C:\ExchDbs\MDB11 as shown in the following image (where only 2 disks are shown):
To achieve this, add additional mount points using the disk management MMC snap-in. In the screenshot below, Disk1 is mounted in C:\ExchVols\Vol1, and C:\ExchDbs\MDB11 and C:\ExchDbs\MDB12 are additional mount points on this disk:
Mailbox database locations are strict when using AutoReseed. For example, MDB11 mailbox database and logfiles must be created in the following directories:
C:\ExchDbs\MDB11\MDB11.db
C:\ExchDbs\MDB11\MDB11.log
The locations of the mount points are properties of the Database Availability Group:
AutoDagDatabaseRootFolderPath
AutoDagVolumesRootFolderPath
AutoDagDatabaseCopiesPerVolume
You can check the correct values using the Get-DatabaseAvailabilityGroup command in Exchange PowerShell:
Create all Mailbox databases, mount them and create Mailbox database copies in the correct locations with the correct names. That’s all it takes to configure AutoReseed. When one of the disks containing Mailbox databases fail, the repair workflow will kick in and in about an hour the spare disk is configured, and a reseed will start. All steps of the repair workflow will be logged in the event log. The last entry with a successful reseed is shown in the following screenshot:
Now that we have AutoReseed up-and-running, we can continue with configuring the MetaCache Database.
Configuring MetaCache Database
One of the prerequisites of the MetaCache Database is a fully functional AutoReseed configuration as outlined in the previous steps. Of course, you also need one or more SSD disks, depending on your disk layout.
The official recommendation for the SSD disks is one SSD disk on three spinning disks. Also, the SSD disk should be a raw disk (not formatted) and it must be exposed as MediaType SSD in Windows. To check this, use the Get-PhysicalDisk | sort DeviceID command in PowerShell as shown in the following screenshot:
As for sizing, 5% or 6% SSD capacity of the total HDD capacity is sufficient. So, if you have 8TB of storage for databases, your SSD capacity can be approx. 400 GB.
Important to note is that the disks must be symmetrical amongst all Exchange 2019 servers in the DAGs.
The first step is to configure the DAG for use with MCDB. MCDB uses the following properties of the DAG for its configuration:
AutoDagTotalNumberOfDatabases. The number of Mailbox databases in the DAG.
AutoDagDatabaseCopiesPerDatabase. The total number of copies (active and passive) of each individual Mailbox database.
AutoDagTotalNumberOfServers. The number of Exchange 2019 Mailbox servers in your DAG.
In my lab, there are two Exchange 2019 Servers in a DAG, 3 spinning disks (plus 1 hot spare), 6 Mailbox databases and 2 copies (one active, one passive) per Mailbox database.
Use the following command to set these properties:
The second step is to configure the MCDB prerequisites using the Manage-MCDB command. This command takes the DagName, SSDSizeInBytes and SSDCountPerServer options.
Notes:
The Manage-MDCB command is default not available in PowerShell. You must import the Manage-MetaCacheDatabase.ps1 script (found in $Exscripts) first using the following command:
CD $ExScripts
Import-Module .\Manage-MetaCacheDatabase.ps1
This step is not in the Microsoft documentation, it took me quite some time to figure it out 😦
The SSDSizeInBytes can be found using the following command:
Get-PhysicalDisk -DeviceNumber x | Select Size
The command for the MCDB prerequisites will be something like this:
The third step is to allow (or disallow) an Exchange 2019 server to use MCDB using the ServerAllowMCDB parameter of the Manage-MCDB command. To do this, execute the following Exchange PowerShell command on each DAG member:
The fourth step is to actually configure MCDB on each Exchange 2019 server. In this step, the raw (unformatted) SSD disk is formatted and the mount points for the MCDB instances are created. To do this, execute the following Exchange PowerShell command, again on each DAG member:
This is all it takes to configure MCDB on a DAG and it is now ready to create the MCDB instances and populate it with cached data. The creation and population (en thus enable acceleration) is initiated by a fail-over. You can use the following MCDB command to initiate this fail-over:
When it comes to monitoring there’s not much to see. You can use the Get-MailboxDatabase command to retrieve configuration properties of MCDB, and you can use the Get-MailboxDatabaseCopyStatus command to see ‘some’ health information regarding MCDB as shown in the following two screenshots:
Unfortunately, Unfortunately that’s it, no more options for monitoring, not even counters in performance monitoring.
So how do you know it works?
Besides the Get-MailboxDatabaseCopyStatus command, you can check the SSD disk which is visible in the Explorer. When configured, the SSD disk is mounted in C:\ExchangeMetaCacheDbs and C:\ExchangeMCDBVolumes. You will find a special (small) MCDB version of the Mailbox database as shown in the following screenshot:
Since this is a regular physical disk you can find it in perfmon, but there are no MCDB specific counters here.
The most interesting thing to test is just login to a Mailbox in one of these Mailbox databases. The look and feel is seriously better than without MCDB. When opening a mailbox in Outlook online mode, or in OWA is just much faster. I have also tried opening a Mailbox remotely via a 20 Mbit line (fiber, so low latency) and it also works better than Exchange without MCDB.
Summary
Exchange 2019 came with this new feature called MetaCache Database, where mailbox data is stored on SSD disks. In the Preferred Architecture, mailbox databases are stored on large SATA disks, but to improve performance frequently accessed data is stored on SSD disk.
The tricky part in configuring MCDB is the configuration of AutoReseed which I find more complex. The lack of proper monitoring is disappointing, but when configured it works very well and you will experience an improved performance. Like most of us, I have worked a lot with properly designed virtualized Exchange environments, but never seen an Exchange environment working as fast as a bare metal Exchange environment with Exchange 2019 and MCDB.
I have been away for a couple of days, but you already might have seen that Microsoft released a number of Security Updates for Exchange 2019, Exchange 2016 and Exchange 2013, but only for the last two Cumulative Updates (as always).
Security Updates are available for the following products:
Security Updates are CU specific and can only be applied to the specific Cumulative Update. When trying to install a Security Update for another CU, an error message will be returned.
Security Updates are also cumulative, so this Security Update contains all previous security updates for this specific CU. There’s no need to install previous Security Updates before this Security Update.
As always, after downloading a Security Update, start the Security Update from a command prompt with elevated privileges (‘Run as Administrator’) to prevent an erratic installation. This does not apply when installing a Security Update via Windows Update or WSUS.
On September 28, 2021 Microsoft released their quarterly updates for Exchange server, Exchange 2019 CU11 and Exchange 2016 CU22. Despite earlier communications a new CU for Exchange 2016 is released as well.
Besides normal fixes, a new feature is introduced in these CUs as well, the Exchange Emergency Mitigation Server or EEMS. EEMS is a new service that can mitigate new security breaches when they arise. EEMS connects to a Microsoft endpoint (https://officeclient.microsoft.com/getexchangemitigations) and when needed, downloads and installs available mitigations. It performs a check once an hour. If you don’t feel comfortable with this, it is possible to disable this on an organization level 😉
Also new in Exchange 2019 CU11 and Exchange 2016 CU22 is telemetry regarding the mitigation service. When configured, it will automatically upload mitigation related service to Microsoft. Again, this can be disabled as well using the license agreement (enabled by default).
When installing this update you will see change in the License Agreement:
The default is I accept the license agreement and will share diagnostics data with Microsoft (recommended), but you can select other as well of course.
When using the unattended install, a new switch is used for accepting the License Agreement.
/IAcceptExchangeServerLicenseTerms_DiagnosticDataON – when you allow to upload diagnostics data to Microsoft
/IAcceptExchangeServerLicenseTerms_DiagnosticDataOFF – when you do not allow to update diagnostics data to Microsoft.
There are also two new prerequisites when installing Exchange 2019 CU11 or Exchange 2016 CU22. Prerequisite software contains now the ‘IIS URL Rewrite Module’ which needs to be installed. The second one is connectivity to the internet for accessing the mitigation service endpoint.
The setup application will check for these prerequisites and will generate an error when they are not met:
Note. The internet connectivity is not shown in this screenshot.
Note. There are no schema changes when upgrading from Exchange 2019 CU10 or Exchange 2016 CU21, but there are changes when upgrading from previous releases.
After installing the updates, you will see the new services when opening the services MMC snap-in:
Or when using the Get-Service MSExchange* PowerShell command:
To check the status in the Exchange organization, you can use the Get-OrganizationConfig | Select mitigations command:
To disable the mitigation service, execute the following command:
Set-OrganizationConfig -MitigationsEnabled:$False
By default, only one mitigation is installed, this is the EEMS heartbeat probe. You can check the installed mitigations by navigating to the Exchange scripts directory and execute the Get-Mitigations.ps1 script:
As with any Cumulative Update, please test this CU in your lab to see if all works well for your environment. Also have a look at the telemetry configuration (is that allowed in your organization?) and at the automatic configuration changes made by the EEMS (I can hear CISO starting to complain).
More information and downloads regarding the Cumulative Updates can be found here:
When you want to check which updates are installed on an Exchange server you can navigate to Control Panel | Programs | View Installed Updates and you will see a list of installed updates, including the Exchange Security Updates.
When running Exchange 2019 on Windows 2019 Server Core there is no Control Panel and you can view the registry to check which updates are installed. Use the following command to view all installed Updates:
You must be logged in to post a comment.