Tag Archives: disaster recovery

Exchange 2016 Setup RecoverServer fails with internal transport certificate warning

I am currently working with a customer on their Exchange 2016 design, implementation and disaster recovery process. While writing a new Exchange 2016 disaster recovery document I ran into this issue in my lab environment while running “Setup.exe /Mode:RecoverServer /IAcceptExchangeServerLicenseTerms”.

image

For search engine options this is a part of the actual error message.

Mailbox role: Transport service FAILED

The following error was generated when “$error.Clear();

Install-ExchangeCertificate -DomainController

$RoleDomainController -Services SMTP

” was run: “System.InvalidOperationException: The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

The solution looks simple since it says “the problem has been fixed”. However, running the setup application again results in the next error message.

image

Again, for search engine possibilities:

Performing Microsoft Exchange Server Prerequisite Check

Configuring Prerequisites COMPLETED

Prerequisite Analysis FAILED

A Setup failure previously occurred while installing the HubTransportRole role. Either run Setup again for just this role, or remove the role using Control Panel.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.InstallWatermark.aspx

The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the

:\ExchangeSetupLogs folder.

Z:\>

To remove the watermark, start the registry editor on the Exchange 2016 server and go to HKLM\Software\Microsoft\ExchangeServer\v15\HubTransportRole and delete the Watermark and Action entries.

image

Rerunning the setup application unfortunately results in the 1st error, despite the “the problem has been fixed” and the removal of the watermark entries.

It turns out that I have two Edge Transport servers in my environment, with an Edge Subscription. This Edge subscription is using the self-signed certificate for encryption purposes, and since this self-signed certificate on the new Exchange 2016 server differs from the original (before the crash) self-signed certificate the encryption possibilities fail.

To resolve this, using ADSI Edit to find the msExchEdgeSyncCredential on the Exchange 2016 server you are recovering, and delete all credential entries.

image

When running the Setup application with the /RecoverServer option again (for the third time ) it will succeed and successfully recover the Exchange 2016 server.

Exchange 2013 Mailbox database Disaster Recovery

In Exchange 2010 Microsoft introduced the Database Availability Group to implement redundancy on mailbox server level and mailbox database level. If a mailbox database (or a server) fails, another one can take over. This concept is carried forward into Exchange 2013 and Exchange 2016 and has improved ever since.

There are still customers that do not use a Database Availability Group and rely on a single server and a solid backup software solution. A backup of the mailbox database is created every night and this continue to run for years. You hope. Until disaster strikes…..

image

I got a call earlier today from a customer. He has been patching his Hyper-V host, and after a reboot his Exchange 2013 server didn’t come up properly. Well, after questioning it turned out that the Exchange server booted correctly, but that only one of three Mailbox databases mounted properly. So, two Mailbox databases (approx. 250 GB in size) seem to be corrupt and this is where the pain begins.

To ‘resolve’ the issue the customer tried to reboot the box again, tried to restore the databases from backup, tried a ‘soft repair’ and tried a ‘hard repair’. No idea what the latter are by the way, but that was according to the customer. But if you know anything about Mailbox databases in Exchange, then you also know that most destruction happens in the first 15 minutes!

If you rely on a single server and a backup solution for restoring services or a disaster recovery scenario you have to know the basics of Exchange database technology. Know what a mailbox database is (except for a large .edb file), know what transactional logging is and how the mailbox database, the transaction log file and the checkpoint file relate to each other. And related to this, it is of utmost importance that you know how to replay transaction log files into a Mailbox database.

Although old, these are good starting points:

Furthermore, you have to know how your backup solution works, and how to restore mailbox database into a production environment. There are streaming backups, but these are rare these days and VSS snapshot backups. You can find more detailed information in the following articles:

Besides the technical knowledge about the Mailbox database technology you have to perform regular ‘fire drills’. Restore a Mailbox database into production, restore using a recovery database, perform replaying of transaction log files, get your hands on ESEUTIL and see what the /G, /K, /P and /R are doing. It will save you a considerable amount of time when you know the technology and the tools, it will reduce risk of data loss and you are able to give a proper planning to your users/manager/customer when the mailboxes are available again.

If you don’t know this you’re playing with fire, and it will backfire to you, believe me!