Recently I ran into an issue with a customer where one mailbox server in a DAG was accidentally deleted. According to Microsoft it is more or less easy to recover an Exchange server since all configuration data is stored in Active Directory. While this is true it was not that easy as I expected since the DAG (and therefore the cluster service) come into play….

Suppose we have a 2 node DAG, both nodes have three server roles (Hub, CAS and Mailbox) and the File Share Witness is on another Hub Transport Server. Node 2 is called EXCH02 is ‘accidentally’ deleted and won’t come back ever. EXCH01 will now hold the active copy of the Mailbox Database and EXCH02 needs to be recovered. The following steps can be used to recover this server:

1. EXCH02 needs to be removed from the DAG, but before doing this the passive copy of the Mailbox Database should be removed from EXCH02. Open the Exchange Management Shell on EXCH01 and enter the following command:
   Remove-MailboxDatabaseCopy –Identity “Mailbox Database1027716905\EXCH02”
   This references the passive copy of the database and will remove it. Since EXCH02 is no longer available the command will throw several warning messages, but the passive copy will be delete (actually the info will be deleted from Active Directory):
   

   

2. Remove EXCH02 from the DAG. Again, use the Exchange Management Shell on EXCH01 to achieve this and use the following command:

   Remove-DatabaseAvailabilityGroupServer -Identity DAG -MailboxServer EXCH02 –ConfigurationOnly

   The –ConfigurationOnly switch is needed to actually remove the info from Active Directory. If this switch is omitted the command will try to contact EXCH02 to remove the cluster service, but since the server is not available anymore this won’t work. The server EXCH02 should now be fully removed from the DAG, you can check this in the Management Console;

3. It can happen that the DAG is not fully cleaned up, you can see this in the Management Console:

   

   Also when opening the Failover Cluster Management snap-in you’ll see that there are some leftovers from the old DAG member.

   

   Right click the old DAG member (i.e. EXCH02) and select “Evict”. The server is now really completely removed from the cluster (=DAG).
   Now, if you don’t check the actual DAG status and there are some leftover you’ll notice later on that it’s not possible to add the newly created server to the DAG;

4. Reset the Computer Account in Active Directory;
5. Build a new server and configure it exactly like the server (EXCH02) that was lost. Use the same Windows Service Packs and hotfixes and use the same disk configuration etc. After installing, give it the same name as the original name (EXCH02) and join it to the domain. Since the computer account was reset this should give no issues at all;
6. Install all prerequisite software for the Exchange server;
7. Install Exchange 2010 from the command prompt (using setup.com, the graphical setup application setup.exe is not capable of doing this) using the /mode:RecoverServer option. This will start the setup of Exchange 2010, but instead of using input from the administrator all input is taken from Active Directory. The setup application will automatically pickup the right configuration:

   

8. After installation reboot the server and bring it up to date with the latest Exchange Update Rollup fixes (when needed);
9. After rebooting the new server can be added to the DAG. Open an Exchange Management Shell and enter the following command:

   Add-DatabaseAvailabilityGroupServer –Identity <<dagname>>  -MailboxServer EXCH02

   

10. When the DAG is complete a new copy of the mailbox database can be created, a copy of the active mailbox database (running on EXCH01) will be created on the new server and log replication will be initiated between the DAG nodes. Use the following Management Shell to achieve this:

    Add-MailboxDatabaseCopy –Identity “Mailbox Database1027716905” –MailboxServer EXCH02 –ActivationPreference:2

    

11. When done, use the Exchange Management Shell to check if replication is healthy

    

If all is well the mailbox server is completely recovered and again part of the DAG