/RecoverServer | Jaap Wesselius

I am currently working with a customer on their Exchange 2016 design, implementation and disaster recovery process. While writing a new Exchange 2016 disaster recovery document I ran into this issue in my lab environment while running “Setup.exe /Mode:RecoverServer /IAcceptExchangeServerLicenseTerms”.

For search engine options this is a part of the actual error message.

Mailbox role: Transport service FAILED

The following error was generated when “$error.Clear(); Install-ExchangeCertificate -DomainController $RoleDomainController -Services SMTP” was run: “System.InvalidOperationException: The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

Edit: I ran into the same issue with Exchange 2019 running on Windows 2019 Server Core.

The solution looks simple since it says “the problem has been fixed”. However, running the setup application again results in the next error message.

Again, for search engine possibilities:

Performing Microsoft Exchange Server Prerequisite Check

Configuring Prerequisites COMPLETED

Prerequisite Analysis FAILED

A Setup failure previously occurred while installing the HubTransportRole role. Either run Setup again for just this role, or remove the role using Control Panel.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.InstallWatermark.aspx

The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the :\ExchangeSetupLogs folder.

To remove the watermark, start the registry editor on the Exchange 2016 server and go to HKLM\Software\Microsoft\ExchangeServer\v15\HubTransportRole and delete the Watermark and Action entries.

Rerunning the setup application unfortunately results in the 1^st error, despite the “the problem has been fixed” and the removal of the watermark entries.

It turns out that I have two Edge Transport servers in my environment, with an Edge Subscription. This Edge subscription is using the self-signed certificate for encryption purposes, and since this self-signed certificate on the new Exchange 2016 server differs from the original (before the crash) self-signed certificate the encryption possibilities fail.

To resolve this, using ADSI Edit to find the msExchEdgeSyncCredential on the Exchange 2016 server you are recovering, and delete all credential entries.

When running the Setup application with the /RecoverServer option again (for the third time ) it will succeed and successfully recover the Exchange 2016 server.

Recently I ran into an issue with a customer where one mailbox server in a DAG was accidentally deleted. According to Microsoft it is more or less easy to recover an Exchange server since all configuration data is stored in Active Directory. While this is true it was not that easy as I expected since the DAG (and therefore the cluster service) come into play….

Suppose we have a 2 node DAG, both nodes have three server roles (Hub, CAS and Mailbox) and the File Share Witness is on another Hub Transport Server. Node 2 is called EXCH02 is ‘accidentally’ deleted and won’t come back ever. EXCH01 will now hold the active copy of the Mailbox Database and EXCH02 needs to be recovered. The following steps can be used to recover this server:

EXCH02 needs to be removed from the DAG, but before doing this the passive copy of the Mailbox Database should be removed from EXCH02. Open the Exchange Management Shell on EXCH01 and enter the following command:
Remove-MailboxDatabaseCopy –Identity “Mailbox Database1027716905\EXCH02”
This references the passive copy of the database and will remove it. Since EXCH02 is no longer available the command will throw several warning messages, but the passive copy will be delete (actually the info will be deleted from Active Directory):
Remove EXCH02 from the DAG. Again, use the Exchange Management Shell on EXCH01 to achieve this and use the following command:
Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EXCH02 –ConfigurationOnly

The –ConfigurationOnly switch is needed to actually remove the info from Active Directory. If this switch is omitted the command will try to contact EXCH02 to remove the cluster service, but since the server is not available anymore this won’t work. The server EXCH02 should now be fully removed from the DAG, you can check this in the Management Console;
It can happen that the DAG is not fully cleaned up, you can see this in the Management Console:

Also when opening the Failover Cluster Management snap-in you’ll see that there are some leftovers from the old DAG member.

Right click the old DAG member (i.e. EXCH02) and select “Evict”. The server is now really completely removed from the cluster (=DAG).
Now, if you don’t check the actual DAG status and there are some leftover you’ll notice later on that it’s not possible to add the newly created server to the DAG;
Reset the Computer Account in Active Directory;
Build a new server and configure it exactly like the server (EXCH02) that was lost. Use the same Windows Service Packs and hotfixes and use the same disk configuration etc. After installing, give it the same name as the original name (EXCH02) and join it to the domain. Since the computer account was reset this should give no issues at all;
Install all prerequisite software for the Exchange server;
Install Exchange 2010 from the command prompt (using setup.com, the graphical setup application setup.exe is not capable of doing this) using the /mode:RecoverServer option. This will start the setup of Exchange 2010, but instead of using input from the administrator all input is taken from Active Directory. The setup application will automatically pickup the right configuration:
After installation reboot the server and bring it up to date with the latest Exchange Update Rollup fixes (when needed);
After rebooting the new server can be added to the DAG. Open an Exchange Management Shell and enter the following command:
Add-DatabaseAvailabilityGroupServer –Identity <<dagname>> -MailboxServer EXCH02
When the DAG is complete a new copy of the mailbox database can be created, a copy of the active mailbox database (running on EXCH01) will be created on the new server and log replication will be initiated between the DAG nodes. Use the following Management Shell to achieve this:
Add-MailboxDatabaseCopy –Identity “Mailbox Database1027716905” –MailboxServer EXCH02 –ActivationPreference:2
When done, use the Exchange Management Shell to check if replication is healthy