Tag Archives: Exchange 2019

Installing and Configuring MetaCache Database (MCDB)

Installing and configuring MCDB in Exchange 2019 has been on my bucket list for a long time, but like most organizations my Exchange servers have been running on either Hyper-V or VMware. I have seen posts on the forums where people were able to publish SSD disks on VMware to VMs, but MCDB has been targeted towards bare metal deployments. But the good news is that I managed to get two big HP boxes to play around with 🙂

MCDB was introduced in Exchange 2019 to speed up access to information in mailboxes that is used frequently. According to the Exchange Preferred Architecture, Mailbox database are stored on (relatively) slow SATA disks. When using MCDB, frequently accessed mailbox information is also stored on SSD disks instead of spinning disks and as such it is much faster to access this information. It is a cache mechanism, so the information on SSD is a copy of information on the spinning disks. If the SSD disk is lost, performance will drop but no information will be lost.

MCDB is useful when running Outlook clients in online mode, for example in a Citrix environment. OWA can also benefit from the improved performance.

MCDB is built on top of Database Availability Groups, so it is not available on single servers (I assume you don’t have a DAG with only one server). It also depends on the AutoReseed feature, so you must deploy this first before configuring MCDB.

My Servers have eight disks installed:

  • 2 SSD disks for boot and system.
  • 4 SAS disks (10 krpm) for Mailbox databases (I preferred SATA disks, but the servers came with these disks).
  • 1 SSD disk for MCDB.
  • 1 (large) SATA disk for storing other information (IIS logs, Queue database, ISO image etc.).

Three of the SAS disks contain two Mailbox databases each, the remaining SAS disk is used as a hot spare for AutoReseed.

I have blogged auto AutoReseed in Exchange 2013 a long time ago (https://jaapwesselius.com/2014/08/09/implementing-and-configuring-autoreseed/ and it hasn’t changed much) but I will repeat the most important parts of it.

AutoReseed

In short, AutoReseed is using the ‘multiple mount points per volume’ option in Windows. For example, the first disk is mounted in C:\ExchVols\Vol1, but this disk is also mounted in C:\ExchDbs\MDB11 as shown in the following image (where only 2 disks are shown):

To achieve this, add additional mount points using the disk management MMC snap-in. In the screenshot below, Disk1 is mounted in C:\ExchVols\Vol1, and C:\ExchDbs\MDB11 and C:\ExchDbs\MDB12 are additional mount points on this disk:

Mailbox database locations are strict when using AutoReseed. For example, MDB11 mailbox database and logfiles must be created in the following directories:

C:\ExchDbs\MDB11\MDB11.db

C:\ExchDbs\MDB11\MDB11.log

The locations of the mount points are properties of the Database Availability Group:

  • AutoDagDatabaseRootFolderPath
  • AutoDagVolumesRootFolderPath
  • AutoDagDatabaseCopiesPerVolume

You can check the correct values using the Get-DatabaseAvailabilityGroup command in Exchange PowerShell:

[PS] C:\>Get-DatabaseAvailabilityGroup -Identity DAG11 | select AutoDag*

AutoDagSchemaVersion             : 1.0
AutoDagDatabaseCopiesPerDatabase : 2
AutoDagDatabaseCopiesPerVolume   : 2
AutoDagTotalNumberOfDatabases    : 6
AutoDagTotalNumberOfServers      : 2
AutoDagDatabasesRootFolderPath   : C:\ExchDbs
AutoDagVolumesRootFolderPath     : C:\ExchVols
AutoDagAllServersInstalled       : False
AutoDagAutoReseedEnabled         : True
AutoDagDiskReclaimerEnabled      : True
AutoDagBitlockerEnabled          : False
AutoDagFIPSCompliant             : False
AutoDagAutoRedistributeEnabled   : True
AutoDagSIPEnabled                : False

Create all Mailbox databases, mount them and create Mailbox database copies in the correct locations with the correct names. That’s all it takes to configure AutoReseed. When one of the disks containing Mailbox databases fail, the repair workflow will kick in and in about an hour the spare disk is configured, and a reseed will start. All steps of the repair workflow will be logged in the event log. The last entry with a successful reseed is shown in the following screenshot:

Now that we have AutoReseed up-and-running, we can continue with configuring the MetaCache Database.

Configuring MetaCache Database

One of the prerequisites of the MetaCache Database is a fully functional AutoReseed configuration as outlined in the previous steps. Of course, you also need one or more SSD disks, depending on your disk layout.

The official recommendation for the SSD disks is one SSD disk on three spinning disks. Also, the SSD disk should be a raw disk (not formatted) and it must be exposed as MediaType SSD in Windows. To check this, use the Get-PhysicalDisk | sort DeviceID command in PowerShell as shown in the following screenshot:

As for sizing, 5% or 6% SSD capacity of the total HDD capacity is sufficient. So, if you have 8TB of storage for databases, your SSD capacity can be approx. 400 GB.

Important to note is that the disks must be symmetrical amongst all Exchange 2019 servers in the DAGs.

The first step is to configure the DAG for use with MCDB. MCDB uses the following properties of the DAG for its configuration:

  • AutoDagTotalNumberOfDatabases. The number of Mailbox databases in the DAG.
  • AutoDagDatabaseCopiesPerDatabase. The total number of copies (active and passive) of each individual Mailbox database.
  • AutoDagTotalNumberOfServers. The number of Exchange 2019 Mailbox servers in your DAG.

In my lab, there are two Exchange 2019 Servers in a DAG, 3 spinning disks (plus 1 hot spare), 6 Mailbox databases and 2 copies (one active, one passive) per Mailbox database.

Use the following command to set these properties:

[PS] C:\> Set-DatabaseAvailabilityGroup DAG11 -AutoDagTotalNumberOfDatabases 6 -AutoDagDatabaseCopiesPerDatabase 2 -AutoDagTotalNumberOfServers 2

The second step is to configure the MCDB prerequisites using the Manage-MCDB command. This command takes the DagName, SSDSizeInBytes and SSDCountPerServer options.

Notes:

The Manage-MDCB command is default not available in PowerShell. You must import the Manage-MetaCacheDatabase.ps1 script (found in $Exscripts) first using the following command:

CD $ExScripts
Import-Module .\Manage-MetaCacheDatabase.ps1

This step is not in the Microsoft documentation, it took me quite some time to figure it out 😦

The SSDSizeInBytes can be found using the following command:

Get-PhysicalDisk -DeviceNumber x | Select Size

The command for the MCDB prerequisites will be something like this:

Manage-MCDB -DagName DAG11 -ConfigureMCDBPrerequisite -SSDSizeInBytes 119998218240 -SSDCountPerServer 1

The third step is to allow (or disallow) an Exchange 2019 server to use MCDB using the ServerAllowMCDB parameter of the Manage-MCDB command. To do this, execute the following Exchange PowerShell command on each DAG member:

[PS] C:\> Manage-MCDB  -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH11

This is shown in the following screenshot:

The fourth step is to actually configure MCDB on each Exchange 2019 server. In this step, the raw (unformatted) SSD disk is formatted and the mount points for the MCDB instances are created. To do this, execute the following Exchange PowerShell command, again on each DAG member:

[PS] C:\> .\Manage-MCDB -DagName DAG11 -ConfigureMCDBOnServer -ServerName EXCH11 -SSDSizeInBytes 119998218240

As shown in the following screenshot:

This is all it takes to configure MCDB on a DAG and it is now ready to create the MCDB instances and populate it with cached data. The creation and population (en thus enable acceleration) is initiated by a fail-over. You can use the following MCDB command to initiate this fail-over:

[PS] C:\> .\Manage-MCDB.ps1 -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH11 -ForceFailover $true

And fail-over back to the previous state:

[PS] C:\> .\Manage-MetacacheDatabase.ps1 -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH12 -ForceFailover $true

When it comes to monitoring there’s not much to see. You can use the Get-MailboxDatabase command to retrieve configuration properties of MCDB, and you can use the Get-MailboxDatabaseCopyStatus command to see ‘some’ health information regarding MCDB as shown in the following two screenshots:

Get-MailboxDatabase -Identity MDB11 | fl *metacache*
[PS] C:\> Get-MailboxDatabaseCopyStatus | Select Identity,MetaCacheDatabaseStatus

Unfortunately, Unfortunately that’s it, no more options for monitoring, not even counters in performance monitoring.

So how do you know it works?

Besides the Get-MailboxDatabaseCopyStatus command, you can check the SSD disk which is visible in the Explorer. When configured, the SSD disk is mounted in C:\ExchangeMetaCacheDbs and C:\ExchangeMCDBVolumes. You will find a special (small) MCDB version of the Mailbox database as shown in the following screenshot:

Since this is a regular physical disk you can find it in perfmon, but there are no MCDB specific counters here.

The most interesting thing to test is just login to a Mailbox in one of these Mailbox databases. The look and feel is seriously better than without MCDB. When opening a mailbox in Outlook online mode, or in OWA is just much faster. I have also tried opening a Mailbox remotely via a 20 Mbit line (fiber, so low latency) and it also works better than Exchange without MCDB.

Summary

Exchange 2019 came with this new feature called MetaCache Database, where mailbox data is stored on SSD disks. In the Preferred Architecture, mailbox databases are stored on large SATA disks, but to improve performance frequently accessed data is stored on SSD disk.

The tricky part in configuring MCDB is the configuration of AutoReseed which I find more complex. The lack of proper monitoring is disappointing, but when configured it works very well and you will experience an improved performance. Like most of us, I have worked a lot with properly designed virtualized Exchange environments, but never seen an Exchange environment working as fast as a bare metal Exchange environment with Exchange 2019 and MCDB.

Exchange security updates November 2021

I have been away for a couple of days, but you already might have seen that Microsoft released a number of Security Updates for Exchange 2019, Exchange 2016 and Exchange 2013, but only for the last two Cumulative Updates (as always).

Security Updates are available for the following products:

Exchange versionDownloadKnowledge Base
Exchange 2019 CU11https://www.microsoft.com/en-us/download/details.aspx?id=103643KB5007409
Exchange 2019 CU10https://www.microsoft.com/en-us/download/details.aspx?id=103642KB5007409
Exchange 2016 CU22https://www.microsoft.com/en-us/download/details.aspx?id=103644KB5007409
Exchange 2016 CU21https://www.microsoft.com/en-us/download/details.aspx?id=103645KB5007409
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103646KB5007409

The following vulnerabilities are addressed in these updates:

Security Updates are CU specific and can only be applied to the specific Cumulative Update. When trying to install a Security Update for another CU, an error message will be returned.

Security Updates are also cumulative, so this Security Update contains all previous security updates for this specific CU. There’s no need to install previous Security Updates before this Security Update.

As always, after downloading a Security Update, start the Security Update from a command prompt with elevated privileges (‘Run as Administrator’) to prevent an erratic installation. This does not apply when installing a Security Update via Windows Update or WSUS.

Exchange Quarterly Updates: Exchange 2019 CU11 and Exchange 2016 CU22

On September 28, 2021 Microsoft released their quarterly updates for Exchange server, Exchange 2019 CU11 and Exchange 2016 CU22. Despite earlier communications a new CU for Exchange 2016 is released as well.

Besides normal fixes, a new feature is introduced in these CUs as well, the Exchange Emergency Mitigation Server or EEMS. EEMS is a new service that can mitigate new security breaches when they arise. EEMS connects to a Microsoft endpoint (https://officeclient.microsoft.com/getexchangemitigations) and when needed, downloads and installs available mitigations. It performs a check once an hour. If you don’t feel comfortable with this, it is possible to disable this on an organization level 😉

Also new in Exchange 2019 CU11 and Exchange 2016 CU22 is telemetry regarding the mitigation service. When configured, it will automatically upload mitigation related service to Microsoft. Again, this can be disabled as well using the license agreement (enabled by default).

When installing this update you will see change in the License Agreement:

The default is I accept the license agreement and will share diagnostics data with Microsoft (recommended), but you can select other as well of course.

When using the unattended install, a new switch is used for accepting the License Agreement.

  • /IAcceptExchangeServerLicenseTerms_DiagnosticDataON – when you allow to upload diagnostics data to Microsoft
  • /IAcceptExchangeServerLicenseTerms_DiagnosticDataOFF – when you do not allow to update diagnostics data to Microsoft.

There are also two new prerequisites when installing Exchange 2019 CU11 or Exchange 2016 CU22. Prerequisite software contains now the ‘IIS URL Rewrite Module’ which needs to be installed. The second one is connectivity to the internet for accessing the mitigation service endpoint.

The setup application will check for these prerequisites and will generate an error when they are not met:

Note. The internet connectivity is not shown in this screenshot.

The ‘IIS URL Rewrite Module’ can be downloaded from https://download.microsoft.com/download/1/2/8/128E2E22-C1B9-44A4-BE2A-5859ED1D4592/rewrite_amd64_en-US.msi

Using PowerShell you can download the module, store it in the C:\Install directory and install it unattended using the following commands:

Start-BitsTransfer -Source "https://download.microsoft.com/download/1/2/8/128E2E22-C1B9-44A4-BE2A-5859ED1D4592/rewrite_amd64_en-US.msi" -Destination C:\Install
Start-Process -FilePath "C:\Install\ rewrite_amd64_en-US.msi " -ArgumentList "/q" -Wait

Updating the Exchange server to this latest CU is not different compared to earlier versions (except for the license agreement switch):

Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms_DiagnosticDataON
Setup.exe /PrepareAD /IAcceptExchangeServerLicenseTerms_DiagnosticDataON
Setup.exe /PrepareDomain /IAcceptExchangeServerLicenseTerms_DiagnosticDataON

Setup.EXE /Mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataON

Note. There are no schema changes when upgrading from Exchange 2019 CU10 or Exchange 2016 CU21, but there are changes when upgrading from previous releases.

After installing the updates, you will see the new services when opening the services MMC snap-in:

Or when using the Get-Service MSExchange* PowerShell command:

To check the status in the Exchange organization, you can use the Get-OrganizationConfig | Select mitigations command:

To disable the mitigation service, execute the following command:

Set-OrganizationConfig -MitigationsEnabled:$False

By default, only one mitigation is installed, this is the EEMS heartbeat probe. You can check the installed mitigations by navigating to the Exchange scripts directory and execute the Get-Mitigations.ps1 script:

As with any Cumulative Update, please test this CU in your lab to see if all works well for your environment. Also have a look at the telemetry configuration (is that allowed in your organization?) and at the automatic configuration changes made by the EEMS (I can hear CISO starting to complain).

More information and downloads regarding the Cumulative Updates can be found here:

Check Server Core for Exchange 2019 Security Updates

When you want to check which updates are installed on an Exchange server you can navigate to Control Panel | Programs | View Installed Updates and you will see a list of installed updates, including the Exchange Security Updates.

When running Exchange 2019 on Windows 2019 Server Core there is no Control Panel and you can view the registry to check which updates are installed. Use the following command to view all installed Updates:

Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*| select-object displayname

Or more specifically for Exchange Server:

Get-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\*| ?{$_.DisplayName -like "*Exchange Server*"} | Select DisplayName

This will show all software and updates for Exchange as shown in the following screenshot:

Get-DatabaseAvailabilityGroupNetwork Misconfigured

When installing a Database Availability Group in Exchange 2019 running on Windows 2019 Server Core, I got IP address configuration errors when creating the DAG. It did succeed, but when running the Get-DatabaseAvailabilityGroupNetwork command in EMS is get a ‘Misconfigured’ warning:

Or in plain text:

Get-DatabaseAvailabilityGroupNetwork
Identity ReplicationEnabled Subnets
-------- ------------------ -------
DAG01\MapiDagNetwork True {{10.38.96.0/24,Misconfigured}, {192.168.0.0/24,Misconfigured}}

This will be a DAG with two networks. One Mapi network (10.38.96.0/24) for clients, and one replication network (192.168.0.0/24).

The requirements for an additional replication network in a DAG are:

  • No default gateway configured
  • No DNS servers configured
  • IP address must not be registered in DNS

When using Windows 2019 Server Core, the first two can be configured using the SCONFIG utility, but the last one is not that simple (but in Windows 2019 Desktop Experience it is just a matter of deselection the “Register this connection’s address in DNS” option when configuring the network interface).

When using Windows Server Core, you can use the Set-DnsClient command to configure this setting, combined with the Get-NetAdapter command.

Execute the Get-NetAdapter command to retrieve the network interfaces. In the following screenshot “Ethernet” is the regular Mapi network interface, “Ethernet 2” is the replication network interface.

To configure the DNS registration option, execute the following command:

Get-NetAdapter -Name "Ethernet 2" | Set-DnsClient -RegisterThisConnectionsAddress $False

When you run the Get-DatabaseAvailabilityGroupNetwork command again, both DAG networks will show up correctly.