Tag Archives: Database Availability Group

Installing and Configuring MetaCache Database (MCDB)

Installing and configuring MCDB in Exchange 2019 has been on my bucket list for a long time, but like most organizations my Exchange servers have been running on either Hyper-V or VMware. I have seen posts on the forums where people were able to publish SSD disks on VMware to VMs, but MCDB has been targeted towards bare metal deployments. But the good news is that I managed to get two big HP boxes to play around with 🙂

MCDB was introduced in Exchange 2019 to speed up access to information in mailboxes that is used frequently. According to the Exchange Preferred Architecture, Mailbox database are stored on (relatively) slow SATA disks. When using MCDB, frequently accessed mailbox information is also stored on SSD disks instead of spinning disks and as such it is much faster to access this information. It is a cache mechanism, so the information on SSD is a copy of information on the spinning disks. If the SSD disk is lost, performance will drop but no information will be lost.

MCDB is useful when running Outlook clients in online mode, for example in a Citrix environment. OWA can also benefit from the improved performance.

MCDB is built on top of Database Availability Groups, so it is not available on single servers (I assume you don’t have a DAG with only one server). It also depends on the AutoReseed feature, so you must deploy this first before configuring MCDB.

My Servers have eight disks installed:

  • 2 SSD disks for boot and system.
  • 4 SAS disks (10 krpm) for Mailbox databases (I preferred SATA disks, but the servers came with these disks).
  • 1 SSD disk for MCDB.
  • 1 (large) SATA disk for storing other information (IIS logs, Queue database, ISO image etc.).

Three of the SAS disks contain two Mailbox databases each, the remaining SAS disk is used as a hot spare for AutoReseed.

I have blogged auto AutoReseed in Exchange 2013 a long time ago (https://jaapwesselius.com/2014/08/09/implementing-and-configuring-autoreseed/ and it hasn’t changed much) but I will repeat the most important parts of it.

AutoReseed

In short, AutoReseed is using the ‘multiple mount points per volume’ option in Windows. For example, the first disk is mounted in C:\ExchVols\Vol1, but this disk is also mounted in C:\ExchDbs\MDB11 as shown in the following image (where only 2 disks are shown):

To achieve this, add additional mount points using the disk management MMC snap-in. In the screenshot below, Disk1 is mounted in C:\ExchVols\Vol1, and C:\ExchDbs\MDB11 and C:\ExchDbs\MDB12 are additional mount points on this disk:

Mailbox database locations are strict when using AutoReseed. For example, MDB11 mailbox database and logfiles must be created in the following directories:

C:\ExchDbs\MDB11\MDB11.db

C:\ExchDbs\MDB11\MDB11.log

The locations of the mount points are properties of the Database Availability Group:

  • AutoDagDatabaseRootFolderPath
  • AutoDagVolumesRootFolderPath
  • AutoDagDatabaseCopiesPerVolume

You can check the correct values using the Get-DatabaseAvailabilityGroup command in Exchange PowerShell:

[PS] C:\>Get-DatabaseAvailabilityGroup -Identity DAG11 | select AutoDag*

AutoDagSchemaVersion             : 1.0
AutoDagDatabaseCopiesPerDatabase : 2
AutoDagDatabaseCopiesPerVolume   : 2
AutoDagTotalNumberOfDatabases    : 6
AutoDagTotalNumberOfServers      : 2
AutoDagDatabasesRootFolderPath   : C:\ExchDbs
AutoDagVolumesRootFolderPath     : C:\ExchVols
AutoDagAllServersInstalled       : False
AutoDagAutoReseedEnabled         : True
AutoDagDiskReclaimerEnabled      : True
AutoDagBitlockerEnabled          : False
AutoDagFIPSCompliant             : False
AutoDagAutoRedistributeEnabled   : True
AutoDagSIPEnabled                : False

Create all Mailbox databases, mount them and create Mailbox database copies in the correct locations with the correct names. That’s all it takes to configure AutoReseed. When one of the disks containing Mailbox databases fail, the repair workflow will kick in and in about an hour the spare disk is configured, and a reseed will start. All steps of the repair workflow will be logged in the event log. The last entry with a successful reseed is shown in the following screenshot:

Now that we have AutoReseed up-and-running, we can continue with configuring the MetaCache Database.

Configuring MetaCache Database

One of the prerequisites of the MetaCache Database is a fully functional AutoReseed configuration as outlined in the previous steps. Of course, you also need one or more SSD disks, depending on your disk layout.

The official recommendation for the SSD disks is one SSD disk on three spinning disks. Also, the SSD disk should be a raw disk (not formatted) and it must be exposed as MediaType SSD in Windows. To check this, use the Get-PhysicalDisk | sort DeviceID command in PowerShell as shown in the following screenshot:

As for sizing, 5% or 6% SSD capacity of the total HDD capacity is sufficient. So, if you have 8TB of storage for databases, your SSD capacity can be approx. 400 GB.

Important to note is that the disks must be symmetrical amongst all Exchange 2019 servers in the DAGs.

The first step is to configure the DAG for use with MCDB. MCDB uses the following properties of the DAG for its configuration:

  • AutoDagTotalNumberOfDatabases. The number of Mailbox databases in the DAG.
  • AutoDagDatabaseCopiesPerDatabase. The total number of copies (active and passive) of each individual Mailbox database.
  • AutoDagTotalNumberOfServers. The number of Exchange 2019 Mailbox servers in your DAG.

In my lab, there are two Exchange 2019 Servers in a DAG, 3 spinning disks (plus 1 hot spare), 6 Mailbox databases and 2 copies (one active, one passive) per Mailbox database.

Use the following command to set these properties:

[PS] C:\> Set-DatabaseAvailabilityGroup DAG11 -AutoDagTotalNumberOfDatabases 6 -AutoDagDatabaseCopiesPerDatabase 2 -AutoDagTotalNumberOfServers 2

The second step is to configure the MCDB prerequisites using the Manage-MCDB command. This command takes the DagName, SSDSizeInBytes and SSDCountPerServer options.

Notes:

The Manage-MDCB command is default not available in PowerShell. You must import the Manage-MetaCacheDatabase.ps1 script (found in $Exscripts) first using the following command:

CD $ExScripts
Import-Module .\Manage-MetaCacheDatabase.ps1

This step is not in the Microsoft documentation, it took me quite some time to figure it out 😦

The SSDSizeInBytes can be found using the following command:

Get-PhysicalDisk -DeviceNumber x | Select Size

The command for the MCDB prerequisites will be something like this:

Manage-MCDB -DagName DAG11 -ConfigureMCDBPrerequisite -SSDSizeInBytes 119998218240 -SSDCountPerServer 1

The third step is to allow (or disallow) an Exchange 2019 server to use MCDB using the ServerAllowMCDB parameter of the Manage-MCDB command. To do this, execute the following Exchange PowerShell command on each DAG member:

[PS] C:\> Manage-MCDB  -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH11

This is shown in the following screenshot:

The fourth step is to actually configure MCDB on each Exchange 2019 server. In this step, the raw (unformatted) SSD disk is formatted and the mount points for the MCDB instances are created. To do this, execute the following Exchange PowerShell command, again on each DAG member:

[PS] C:\> .\Manage-MCDB -DagName DAG11 -ConfigureMCDBOnServer -ServerName EXCH11 -SSDSizeInBytes 119998218240

As shown in the following screenshot:

This is all it takes to configure MCDB on a DAG and it is now ready to create the MCDB instances and populate it with cached data. The creation and population (en thus enable acceleration) is initiated by a fail-over. You can use the following MCDB command to initiate this fail-over:

[PS] C:\> .\Manage-MCDB.ps1 -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH11 -ForceFailover $true

And fail-over back to the previous state:

[PS] C:\> .\Manage-MetacacheDatabase.ps1 -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH12 -ForceFailover $true

When it comes to monitoring there’s not much to see. You can use the Get-MailboxDatabase command to retrieve configuration properties of MCDB, and you can use the Get-MailboxDatabaseCopyStatus command to see ‘some’ health information regarding MCDB as shown in the following two screenshots:

Get-MailboxDatabase -Identity MDB11 | fl *metacache*
[PS] C:\> Get-MailboxDatabaseCopyStatus | Select Identity,MetaCacheDatabaseStatus

Unfortunately, Unfortunately that’s it, no more options for monitoring, not even counters in performance monitoring.

So how do you know it works?

Besides the Get-MailboxDatabaseCopyStatus command, you can check the SSD disk which is visible in the Explorer. When configured, the SSD disk is mounted in C:\ExchangeMetaCacheDbs and C:\ExchangeMCDBVolumes. You will find a special (small) MCDB version of the Mailbox database as shown in the following screenshot:

Since this is a regular physical disk you can find it in perfmon, but there are no MCDB specific counters here.

The most interesting thing to test is just login to a Mailbox in one of these Mailbox databases. The look and feel is seriously better than without MCDB. When opening a mailbox in Outlook online mode, or in OWA is just much faster. I have also tried opening a Mailbox remotely via a 20 Mbit line (fiber, so low latency) and it also works better than Exchange without MCDB.

Summary

Exchange 2019 came with this new feature called MetaCache Database, where mailbox data is stored on SSD disks. In the Preferred Architecture, mailbox databases are stored on large SATA disks, but to improve performance frequently accessed data is stored on SSD disk.

The tricky part in configuring MCDB is the configuration of AutoReseed which I find more complex. The lack of proper monitoring is disappointing, but when configured it works very well and you will experience an improved performance. Like most of us, I have worked a lot with properly designed virtualized Exchange environments, but never seen an Exchange environment working as fast as a bare metal Exchange environment with Exchange 2019 and MCDB.

This computer is a member of a database availability group (DAG)

When uninstalling an Exchange server that is part of a Database Availability Group, you perform several steps to remove the server from the DAG and maybe even delete the DAG itself.

So, after decommissioning the DAG I wanted to uninstall Exchange, but the readiness check failed with the following error message:

This computer is a member of a database availability group (DAG). It must be removed from the DAG before you can uninstall Exchange. For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.CannotUninstallClusterNode.aspx

Unfortunately this link has not been updates since 2015, and does not contain any useful information.

I decommissioned the DAG earlier as you can see in the following screenshot, but it still failed the readiness check:

When using the Get-DatabaseAvailabilityGroup command nothing showed up, and also when checking Active Directory with ADSI Edit (CN=Organization, CN=Administrative Groups, CN=Exchange Administrative Group (FYDIBOHF23SPDLT), CN=Database Availability Groups) nothing is shown.

The DAG is using bits of Windows failover clustering, so the culprit must be there. When executing the Get-ClusterNode and Get-Cluster commands in PowerShell, some DAG leftovers (on this particular machine) showed up:

Something must have gone wrong during decommissioning of the DAG, although no error were shown on the console.

To remove the cluster node (and thus the cluster) from this machine, execute the following command in PowerShell:

PS C:\> Remove-ClusterNode EXCH12 -Force

After this step uninstalling Exchange succeeds.

Get-DatabaseAvailabilityGroupNetwork Misconfigured

When installing a Database Availability Group in Exchange 2019 running on Windows 2019 Server Core, I got IP address configuration errors when creating the DAG. It did succeed, but when running the Get-DatabaseAvailabilityGroupNetwork command in EMS is get a ‘Misconfigured’ warning:

Or in plain text:

Get-DatabaseAvailabilityGroupNetwork
Identity ReplicationEnabled Subnets
-------- ------------------ -------
DAG01\MapiDagNetwork True {{10.38.96.0/24,Misconfigured}, {192.168.0.0/24,Misconfigured}}

This will be a DAG with two networks. One Mapi network (10.38.96.0/24) for clients, and one replication network (192.168.0.0/24).

The requirements for an additional replication network in a DAG are:

  • No default gateway configured
  • No DNS servers configured
  • IP address must not be registered in DNS

When using Windows 2019 Server Core, the first two can be configured using the SCONFIG utility, but the last one is not that simple (but in Windows 2019 Desktop Experience it is just a matter of deselection the “Register this connection’s address in DNS” option when configuring the network interface).

When using Windows Server Core, you can use the Set-DnsClient command to configure this setting, combined with the Get-NetAdapter command.

Execute the Get-NetAdapter command to retrieve the network interfaces. In the following screenshot “Ethernet” is the regular Mapi network interface, “Ethernet 2” is the replication network interface.

To configure the DNS registration option, execute the following command:

Get-NetAdapter -Name "Ethernet 2" | Set-DnsClient -RegisterThisConnectionsAddress $False

When you run the Get-DatabaseAvailabilityGroupNetwork command again, both DAG networks will show up correctly.

Exchange 2016 Database Availability Group and Cloud Witness

When implementing a Database Availability Group (in Exchange 2010 and higher) you need a File Share Witness (FSW). This FSW is located on a Witness Server which can be any domain joined server in your internal network, as long as it is running a supported Operating System. It can be another Exchange server, as long as the Witness Server is not a member of the DAG you are deploying.

A long time ago (I don’t recall exactly, but it could well be around Exchange 2013 SP1) Microsoft started to support using Azure for hosting the Witness server. In this scenario you would host a Virtual Machine in Azure. This VM is a domain joined VM, for which you most likely also host a Domain Controller in Azure, and for connectivity you would need a site-2-site VPN connection to Azure. Not only from your primary datacenter, but also from your secondary datacenter, i.e. a multi-site VPN Connection, as shown in the following picture:

image

While this is possible and fully supported, it is costly adventure, and personally I haven’t seen any of my customers deploy it yet (although my customers are still interested).

Windows 2016 Cloud Witness

In Windows 2016 the concept of ‘Cloud Witness’ was introduced. The Cloud Witness concept is the same as the Witness server, but instead of using a file share it is using Azure Blob Storage for read/write purposes, which is used as an arbitration point in case of a split-brain situation.

The advantages are obvious:

  • No need for a 3rd datacenter hosting your Witness server.
  • No need for an expensive VM in Azure hosting you Witness server.
  • Using standard Azure Blob Storage (thus cheap).
  • Same Azure Storage Account can be used for multiple clusters.
  • Built-in Cloud Witness resource type (in Windows 2016 of course).

Looking at all this it seems like a good idea to use the Cloud Witness when deploying Windows 2016 failover clusters, or when deploying a Database Availability Group when running Exchange 2016 on Windows 2016.

Unfortunately, this is not a supported scenario at this point. All information you find on the Internet is most likely not officially published by the Microsoft Exchange team. If at one point the Cloud Witness becomes a supported solution for Exchange 2016, you can find it on the Exchange blog. When this happens, I’ll update this page as well.

More information

Using a Microsoft Azure VM as a DAG witness server – https://technet.microsoft.com/en-us/library/dn903504(v=exchg.160).aspx

Balance Mailbox databases in a DAG

If you have a DAG with multiple Mailbox servers and a lot of Mailbox databases it’s a good practice to regularly have a look at the distribution of the Mailbox database.

When you reboot a Mailbox server for example all active copies of Mailbox databases are moved to other Mailbox server but they are never moved back to their original location.

Continue reading Balance Mailbox databases in a DAG