Category Archives: Exchange

Installing and Configuring MetaCache Database (MCDB)

Installing and configuring MCDB in Exchange 2019 has been on my bucket list for a long time, but like most organizations my Exchange servers have been running on either Hyper-V or VMware. I have seen posts on the forums where people were able to publish SSD disks on VMware to VMs, but MCDB has been targeted towards bare metal deployments. But the good news is that I managed to get two big HP boxes to play around with 🙂

MCDB was introduced in Exchange 2019 to speed up access to information in mailboxes that is used frequently. According to the Exchange Preferred Architecture, Mailbox database are stored on (relatively) slow SATA disks. When using MCDB, frequently accessed mailbox information is also stored on SSD disks instead of spinning disks and as such it is much faster to access this information. It is a cache mechanism, so the information on SSD is a copy of information on the spinning disks. If the SSD disk is lost, performance will drop but no information will be lost.

MCDB is useful when running Outlook clients in online mode, for example in a Citrix environment. OWA can also benefit from the improved performance.

MCDB is built on top of Database Availability Groups, so it is not available on single servers (I assume you don’t have a DAG with only one server). It also depends on the AutoReseed feature, so you must deploy this first before configuring MCDB.

My Servers have eight disks installed:

  • 2 SSD disks for boot and system.
  • 4 SAS disks (10 krpm) for Mailbox databases (I preferred SATA disks, but the servers came with these disks).
  • 1 SSD disk for MCDB.
  • 1 (large) SATA disk for storing other information (IIS logs, Queue database, ISO image etc.).

Three of the SAS disks contain two Mailbox databases each, the remaining SAS disk is used as a hot spare for AutoReseed.

I have blogged auto AutoReseed in Exchange 2013 a long time ago (https://jaapwesselius.com/2014/08/09/implementing-and-configuring-autoreseed/ and it hasn’t changed much) but I will repeat the most important parts of it.

AutoReseed

In short, AutoReseed is using the ‘multiple mount points per volume’ option in Windows. For example, the first disk is mounted in C:\ExchVols\Vol1, but this disk is also mounted in C:\ExchDbs\MDB11 as shown in the following image (where only 2 disks are shown):

To achieve this, add additional mount points using the disk management MMC snap-in. In the screenshot below, Disk1 is mounted in C:\ExchVols\Vol1, and C:\ExchDbs\MDB11 and C:\ExchDbs\MDB12 are additional mount points on this disk:

Mailbox database locations are strict when using AutoReseed. For example, MDB11 mailbox database and logfiles must be created in the following directories:

C:\ExchDbs\MDB11\MDB11.db

C:\ExchDbs\MDB11\MDB11.log

The locations of the mount points are properties of the Database Availability Group:

  • AutoDagDatabaseRootFolderPath
  • AutoDagVolumesRootFolderPath
  • AutoDagDatabaseCopiesPerVolume

You can check the correct values using the Get-DatabaseAvailabilityGroup command in Exchange PowerShell:

[PS] C:\>Get-DatabaseAvailabilityGroup -Identity DAG11 | select AutoDag*

AutoDagSchemaVersion             : 1.0
AutoDagDatabaseCopiesPerDatabase : 2
AutoDagDatabaseCopiesPerVolume   : 2
AutoDagTotalNumberOfDatabases    : 6
AutoDagTotalNumberOfServers      : 2
AutoDagDatabasesRootFolderPath   : C:\ExchDbs
AutoDagVolumesRootFolderPath     : C:\ExchVols
AutoDagAllServersInstalled       : False
AutoDagAutoReseedEnabled         : True
AutoDagDiskReclaimerEnabled      : True
AutoDagBitlockerEnabled          : False
AutoDagFIPSCompliant             : False
AutoDagAutoRedistributeEnabled   : True
AutoDagSIPEnabled                : False

Create all Mailbox databases, mount them and create Mailbox database copies in the correct locations with the correct names. That’s all it takes to configure AutoReseed. When one of the disks containing Mailbox databases fail, the repair workflow will kick in and in about an hour the spare disk is configured, and a reseed will start. All steps of the repair workflow will be logged in the event log. The last entry with a successful reseed is shown in the following screenshot:

Now that we have AutoReseed up-and-running, we can continue with configuring the MetaCache Database.

Configuring MetaCache Database

One of the prerequisites of the MetaCache Database is a fully functional AutoReseed configuration as outlined in the previous steps. Of course, you also need one or more SSD disks, depending on your disk layout.

The official recommendation for the SSD disks is one SSD disk on three spinning disks. Also, the SSD disk should be a raw disk (not formatted) and it must be exposed as MediaType SSD in Windows. To check this, use the Get-PhysicalDisk | sort DeviceID command in PowerShell as shown in the following screenshot:

As for sizing, 5% or 6% SSD capacity of the total HDD capacity is sufficient. So, if you have 8TB of storage for databases, your SSD capacity can be approx. 400 GB.

Important to note is that the disks must be symmetrical amongst all Exchange 2019 servers in the DAGs.

The first step is to configure the DAG for use with MCDB. MCDB uses the following properties of the DAG for its configuration:

  • AutoDagTotalNumberOfDatabases. The number of Mailbox databases in the DAG.
  • AutoDagDatabaseCopiesPerDatabase. The total number of copies (active and passive) of each individual Mailbox database.
  • AutoDagTotalNumberOfServers. The number of Exchange 2019 Mailbox servers in your DAG.

In my lab, there are two Exchange 2019 Servers in a DAG, 3 spinning disks (plus 1 hot spare), 6 Mailbox databases and 2 copies (one active, one passive) per Mailbox database.

Use the following command to set these properties:

[PS] C:\> Set-DatabaseAvailabilityGroup DAG11 -AutoDagTotalNumberOfDatabases 6 -AutoDagDatabaseCopiesPerDatabase 2 -AutoDagTotalNumberOfServers 2

The second step is to configure the MCDB prerequisites using the Manage-MCDB command. This command takes the DagName, SSDSizeInBytes and SSDCountPerServer options.

Notes:

The Manage-MDCB command is default not available in PowerShell. You must import the Manage-MetaCacheDatabase.ps1 script (found in $Exscripts) first using the following command:

CD $ExScripts
Import-Module .\Manage-MetaCacheDatabase.ps1

This step is not in the Microsoft documentation, it took me quite some time to figure it out 😦

The SSDSizeInBytes can be found using the following command:

Get-PhysicalDisk -DeviceNumber x | Select Size

The command for the MCDB prerequisites will be something like this:

Manage-MCDB -DagName DAG11 -ConfigureMCDBPrerequisite -SSDSizeInBytes 119998218240 -SSDCountPerServer 1

The third step is to allow (or disallow) an Exchange 2019 server to use MCDB using the ServerAllowMCDB parameter of the Manage-MCDB command. To do this, execute the following Exchange PowerShell command on each DAG member:

[PS] C:\> Manage-MCDB  -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH11

This is shown in the following screenshot:

The fourth step is to actually configure MCDB on each Exchange 2019 server. In this step, the raw (unformatted) SSD disk is formatted and the mount points for the MCDB instances are created. To do this, execute the following Exchange PowerShell command, again on each DAG member:

[PS] C:\> .\Manage-MCDB -DagName DAG11 -ConfigureMCDBOnServer -ServerName EXCH11 -SSDSizeInBytes 119998218240

As shown in the following screenshot:

This is all it takes to configure MCDB on a DAG and it is now ready to create the MCDB instances and populate it with cached data. The creation and population (en thus enable acceleration) is initiated by a fail-over. You can use the following MCDB command to initiate this fail-over:

[PS] C:\> .\Manage-MCDB.ps1 -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH11 -ForceFailover $true

And fail-over back to the previous state:

[PS] C:\> .\Manage-MetacacheDatabase.ps1 -DagName DAG11 -ServerAllowMCDB:$True -ServerName EXCH12 -ForceFailover $true

When it comes to monitoring there’s not much to see. You can use the Get-MailboxDatabase command to retrieve configuration properties of MCDB, and you can use the Get-MailboxDatabaseCopyStatus command to see ‘some’ health information regarding MCDB as shown in the following two screenshots:

Get-MailboxDatabase -Identity MDB11 | fl *metacache*
[PS] C:\> Get-MailboxDatabaseCopyStatus | Select Identity,MetaCacheDatabaseStatus

Unfortunately, Unfortunately that’s it, no more options for monitoring, not even counters in performance monitoring.

So how do you know it works?

Besides the Get-MailboxDatabaseCopyStatus command, you can check the SSD disk which is visible in the Explorer. When configured, the SSD disk is mounted in C:\ExchangeMetaCacheDbs and C:\ExchangeMCDBVolumes. You will find a special (small) MCDB version of the Mailbox database as shown in the following screenshot:

Since this is a regular physical disk you can find it in perfmon, but there are no MCDB specific counters here.

The most interesting thing to test is just login to a Mailbox in one of these Mailbox databases. The look and feel is seriously better than without MCDB. When opening a mailbox in Outlook online mode, or in OWA is just much faster. I have also tried opening a Mailbox remotely via a 20 Mbit line (fiber, so low latency) and it also works better than Exchange without MCDB.

Summary

Exchange 2019 came with this new feature called MetaCache Database, where mailbox data is stored on SSD disks. In the Preferred Architecture, mailbox databases are stored on large SATA disks, but to improve performance frequently accessed data is stored on SSD disk.

The tricky part in configuring MCDB is the configuration of AutoReseed which I find more complex. The lack of proper monitoring is disappointing, but when configured it works very well and you will experience an improved performance. Like most of us, I have worked a lot with properly designed virtualized Exchange environments, but never seen an Exchange environment working as fast as a bare metal Exchange environment with Exchange 2019 and MCDB.

January 2022 Exchange Security Updates

On january 11, 2022 Microsoft released new Security Updates for Exchange versions:

  • Exchange 2013 CU23
  • Exchange 2016 CU21, Exchange 2016 CU22
  • Exchange 2019 CU10, Exchange 2019 CU11

The following vulnerabilities have been addressed in these Security Updates:

No exploits have been found in the wild, but it is recommended to install these Security Updates as soon as possible.

These updates are targeted toward Exchange server on-premises, including Exchange servers used in a hybrid configuration.

Please note the following:

  • Run the Exchange Server Healthcheck script on your Exchange server to get an overview of all issues in your environment, including installed Security Updates and Cumulative Updates versions.
  • If running an old (and unsupported!) version of Exchange server, please update to the latest CU to get in a supported state and install these Security Updates.
  • When installing manually, start the update from a command prompt with elevated privileges. If you fail to do so, it will look like installation successfully finishes, but various issues will occur. This is not needed when installing using Windows Update or WSUS.
  • Security Updates are also cumulative, so this Security Updates contains all previous Security Updates for this specific Cumulative Update. There’s no need to install previous Security Updates before installing this Security Update.
  • The December 2021 Cumulative Update is postponed, check the link on the Microsoft site. Microsoft does not release Security Updates and Security Updates in the same month, so do not except a new Cumulative Update anytime soon.
  • This Security Update does not contain a fix for the Y2K22 problem that popped up on January 1, see the Email stuck in Exchange on-premises Transport Queues article which also contains the solution.
  • As always, download and deploy in your test environment to see if it all works well in your environment.
Exchange versionDownloadKnowledge base
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103857KB5008631
Exchange 2016 CU21https://www.microsoft.com/en-us/download/details.aspx?id=103856KB5008631
Exchange 2016 CU22https://www.microsoft.com/en-us/download/details.aspx?id=103855KB5008631
Exchange 2019 CU10https://www.microsoft.com/en-us/download/details.aspx?id=103853KB5008631
Exchange 2019 CU11https://www.microsoft.com/en-us/download/details.aspx?id=103854KB5008631

The FIP-FS Scan Process failed initialisation. Mail is queued on Exchange servers.

According to a post on Reddit it seems that Exchange 2016 and Exchange 2019 on-premises are queueing email messages starting januari 1, 2022 at midnight (UTC).

Besides mail queueing you will also see EventID 1106 in the application event log stating “The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error”

This might be a bug is how the date is handles inside the scan engine, causing it to fail after midnight UTC on Januari 1, 2022 (or is it December 31, 2021).

As there is no fix from Microsoft yet, the workaround is to disable anti-malware scanning on all your Exchange servers and restart the Transport service:

CD $ExScripts
.\Disable-AntiMalwareScanning.ps1
Restart-Service MSExchangeTransport

Update 1 on January 1 at 9PM GMT+1. Microsoft is aware of this issue and working on a solution. Check the Exchange team blog: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

Update 2 on January 2 at 10AM GMT+1. Microsoft has released a solution for this issue. A script can be downloaded from https://aka.ms/ResetScanEngineVersion. This will stop the services, delete the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft and the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata directories and download new scan engines. This can take a couple of minutes, but the script can be run in parallel on all your Exchange servers.

When the script has finished, check the eventlog and you should see EventID 6036 that all is well.

Although not documented, I had to reboot my Exchange servers. I’ve several other reports from people that they had to reboot too. Another thing, when you disabled the anti-malware as a workaround, you have to re-enable the anti-malware manually. You can check this using the Get-TransportAgent “malware agent” command.

It is also possible to manually update your Exchange servers, this is also documented in the Microsoft article https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

One warning though….. All Exchange 2016 and 2019 servers worldwide are suffering from this issue and they all will queue messages. Queues expire after 48 hours, so when not fixed the Exchange servers will generate NDR messages on Sunday night. Worst case scenario, millions of NDRs will be generated which in turn will result in tons of helpdesk calls. If you read this, most likely you have fixed your Exchange servers and it is the Exchange environment of the intended recipients.

Let’s hope it will be quiet again for some time now 🙂

Exchange Server OWA and ECP not working

Since a couple of days my OWA and ECP are not working anymore. This happens on both Exchange 2019 CU11 and Exchange 2016 CU22, in two AD sites both with external access. I didn’t notice before since Outlook Mobile and Outlook on the desktop continue to work. After logging in, the Something went wrong message appears, in the navigation bar you can see it is an Error 500 message.

I think (but I’m not sure) that this started after applying the latest November 2021 Security Updates and this is usually caused by starting the Security Update without elevated privileges. There’s an Microsoft article about this: OWA or ECP stops working after you install a security update – Exchange | Microsoft Docs.

I am pretty sure that I installed the Security Update with elevated privileges, but also after reinstalling the Security Update (with elevated privileges!) the error continues. The Microsoft article also mentions settings in IIS Manager (Application settings > BinsearchFolder), but that was not the issue (settings were ok).

When authentication fails, two entries are written in the Application Eventlog, EventID 1003 (MSExchange Front End HTTP Proxy) and EventID 1309 (ASP.NET 4.0.30319.0). The latter clearly shows it has something to do with certificates:

It turned out that the Exchange Server Auth Certificate was expired, just a few days ago. You can see this when running the following command:

[PS] C:\>(Get-AuthConfig).CurrentCertificateThumbprint | Get-ExchangeCertificate | Format-List

AccessRules        : {System.Security.AccessControl.CryptoKeyAccessRule,
                     System.Security.AccessControl.CryptoKeyAccessRule,
                     System.Security.AccessControl.CryptoKeyAccessRule}
CertificateDomains : {}
HasPrivateKey      : True
IsSelfSigned       : True
Issuer             : CN=Microsoft Exchange Server Auth Certificate
NotAfter           : 11/28/2021 12:23:07 AM
NotBefore          : 12/24/2016 12:23:07 AM
PublicKeySize      : 2048
RootCAType         : Unknown
SerialNumber       : 268E6FB9B7312AB24AAA6BA76D06190D
Services           : SMTP
Status             : Invalid
Subject            : CN=Microsoft Exchange Server Auth Certificate
Thumbprint         : A7F9CAA2C9016DB2A80F1E2972E2ED0E2FAE089D

As shown in the following screenshot:

Use the New-ExchangeCertificate command to create a new self-signed certificate for authentication purposes:

[PS] C:\>New-ExchangeCertificate -KeySize 2048 -PrivateKeyExportable $true -SubjectName "cn=Microsoft Exchange Server Auth Certificate" -FriendlyName "Microsoft Exchange Server Auth Certificate" -DomainName @()

Confirm
Overwrite the existing default SMTP certificate?

Current certificate: 'D7081FC32B9BFBEF0C0581584976F690D5F86E74' (expires 11/30/2026 5:21:19 PM)
Replace it with certificate: '309263C8C5B2DA9612E8A6FA9FFFCDEBAC93335D' (expires 11/30/2026 9:00:56 PM)
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): n

Thumbprint                                Services   Subject
----------                                --------   -------
309263C8C5B2DA9612E8A6FA9FFFCDEBAC93335D  ....S..    CN=Microsoft Exchange Server Auth Certificate

As shown in the following screenshot:

When the certificate is created, the AuthConfig needs to be configured, it needs to be published and the old (and expired) certificated needs to be removed. Use the Set-AuthConfig command to achieve this:

[PS] C:\>Set-AuthConfig -NewCertificateThumbprint 309263C8C5B2DA9612E8A6FA9FFFCDEBAC93335D -NewCertificateEffectiveDate (Get-Date)

Confirm
The new certificate effective date is not at least "48" hours in the future and may not be deployed on all necessary
servers. Do you wish to continue?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
[PS] C:\>Set-AuthConfig -PublishCertificate
[PS] C:\>Set-AuthConfig -ClearPreviousCertificate

As shown in the following screenshot:

Restart the Microsoft Exchange Service Host service and perform an IISRESET. If you cannot run IISRESET you can also recycle both the OWA and ECP App pool:

[PS] C:\> Restart-WebAppPool MSExchangeOWAAppPool
[PS] C:\> Restart-WebAppPool MSExchangeECPAppPool

The certificate is stored in Active Directory in CN=Auth Configuration, CN=, CN=Microsoft Exchange, CN=Services,DC=, DC= as shown in the following screenshot:

Since it is published in Active Directory, the new certificate will be automatically available for all Exchange servers in your organization. It can take up to an hour before it is fully published and available, so don’t worry when it doesn’t work immediately.

Please be aware that you do this only on one Exchange server. If you (accidentally) do this on multiple Exchange servers, you will see multiple Auth Certificates appear on your Exchange server. But only the last one created will be active though.

Only two steps remain:

  • Remove the old Auth Certificate on all Exchange servers. You can do this using EAC or using PowerShell (Remove-ExchangeCertficate -Server -Thumbprint <old certificate).
  • Run the Hybrid Configuration Wizard again to update the new certificate in Azure Active Directory.

Exchange security updates November 2021

I have been away for a couple of days, but you already might have seen that Microsoft released a number of Security Updates for Exchange 2019, Exchange 2016 and Exchange 2013, but only for the last two Cumulative Updates (as always).

Security Updates are available for the following products:

Exchange versionDownloadKnowledge Base
Exchange 2019 CU11https://www.microsoft.com/en-us/download/details.aspx?id=103643KB5007409
Exchange 2019 CU10https://www.microsoft.com/en-us/download/details.aspx?id=103642KB5007409
Exchange 2016 CU22https://www.microsoft.com/en-us/download/details.aspx?id=103644KB5007409
Exchange 2016 CU21https://www.microsoft.com/en-us/download/details.aspx?id=103645KB5007409
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103646KB5007409

The following vulnerabilities are addressed in these updates:

Security Updates are CU specific and can only be applied to the specific Cumulative Update. When trying to install a Security Update for another CU, an error message will be returned.

Security Updates are also cumulative, so this Security Update contains all previous security updates for this specific CU. There’s no need to install previous Security Updates before this Security Update.

As always, after downloading a Security Update, start the Security Update from a command prompt with elevated privileges (‘Run as Administrator’) to prevent an erratic installation. This does not apply when installing a Security Update via Windows Update or WSUS.