If you are unlucky and your Exchange server is infected because of the HAFNIUM zero-day vulnerability, you must nuke your Exchange server and rebuild it. And I already got the first questions on how to do this. It is not that difficult and does not take days of time if some prerequisites are met of course.
Note. This blog was written with the HAFNIUM infected machines in mind, but from a procedural point of view it can be used for every disaster recovery scenario of course. Also, I used Windows 2012 R2 and Exchange 2013 for this blog, but this procedure can also be used for Exchange 2016 and Exchange 2019.
Basically, what happens is that the old server is forcibly removed (shutdown, delete VM) but that the computer account in AD is not deleted but reset (shown in screenshot below). A new server is built, same OS, same patching, same configuration, same computername and joined to the domain. This way the SID will remain the same, which makes recovering a lot easier.
Rebuilding Windows server including all prerequisite software and Windows patches will take most of the time in this entire process.
You must use the same server version when rebuilding the server. So, if you old server was running Windows 2012 R2, then you MUST use Windows 2021 R2 again. If you try different, bad things will happen.
Exchange is a bit more flexible these days. You can use the /RecoverServer option with a newer Cumulative Update. For example, when your old server is running Exchange 2016 CU11, you can do a /RecoverServer with Exchange 2016 CU19. The only drawback here is that the CU11 version information is stored in Active Directory, and even when CU19 is used, it will show up as CU11 for the AdminDisplayVersion. But that’s cosmetic and will automatically be corrected the next time a new CU is installed.
What you CANNOT do is use /RecoverServer to upgrade to a newer version of Exchange, like upgrading from Exchange 2013 to Exchange 2016. Again, bad things will happen.
Storage with Mailbox database need of course to be kept. If configured on separate disks of VHD’s you must connect these to the new server. Use the same drive letters or mount points. In the screenshot below the old disk with the Mailbox database is mounted to the new VM with Windows 2012 R2. Exchange 2013 is not yet installed on this server.
Note. Instead of my approach (connecting the existing storage) it is also possible to recover the server and restore the mailbox database later from backup. This can range from any backup application to a file level backup of your .EDB file. The latter needs extensive knowledge about the Mailbox database internals and how to deal with that.
When Windows is up and running it is time to install the server. Use the same Cumulative Update for this and use the unattended setup of Exchange with the RecoverServer option. Open a command prompt with elevated privileges and execute the following command:
Z:> Setup.exe /mode:RecoverServer /IAcceptExchangeServerLicenseTerms
Z:\ in this case is the DVD drive letter, but you can use a different drive letter of course.
There’s one catch here. If Exchange is installed in a different location, for example on the D:\ drive, you must use the /TargetDir option to specify the location where the Exchange binaries must be installed.
The /RecoverServer will retrieve all configuration information from Active Directory (instead of using the default settings) and install Exchange using this configuration. For example, all Mailbox database information is stored in Active Directory, so when setup has finished, the ‘new’ server has all the correct Mailbox database information. Even better, after a reboot it will even automatically mount the Mailbox databases.
Some settings are not stored in Active Directory, especially when looking at server specific configuration that are stored in local .config files. OWA customization and SSL Certificates for example that are configured on the server are lost. So, existing documentation of the Exchange server is vital here to quickly rebuild the Exchange server.
After reinstallation, the server was rebooted, I had to mount the Mailbox database manually (but without issues). The following settings I had to configure manually:
- Import the SSL certificate and bind it to the Exchange services.
- Set the Exchange Virtual Directories (I did not expect this one).
- Relocate the SMTP Queue database.
To my surprise the SMTP relay connector that was configured on the Exchange server was available instantly and working correctly.
Summary
If you have to rebuild your Exchange server, whether it be it crashed or got infected or something, you can use the Setup.exe /RecoverServer option of Exchange (Exchange 2013 and up). This will retrieve a lot of the information from Active Directory, and if you have the Mailbox databases available you can use these directly without any restore from backup activities.
There are some settings that are still manually configured after the server is recovered, so proper documentation is important to have to make life much easier when recovering.
Last updated: March 10, 2021
Jaap Wesselius 
Hi Jaap,
Small question about rebuiding an exchange (2016) exchange server with the -recoverserver option. What about licensing/activation of Windows and Exchange? Does it go automatically? Or do I reactivate? Can I use same license keys as original server?
Thx!
LikeLike
yes, you have to reactivate Windows. As for the licenses, you can use the original ones, no problem
LikeLike
Hi Jaap,
This is just what I needed. Our client is running an Exchange 2013 CU19 DAG cluster. We scanned our servers with the MSERT tool and the Powershell scripts on Github. The first server returned negative. No suspicious activity found and upgraded to CU23 + security patch without problems. The second Exchange server did get compromised, so I immediately put it in maintenance mode in the DAG and disconnected it from the network.
I have a brand new 2012 R2 VM ready that is fully updated. Before I run the 2013 CU23 setup with the /m:recoverserver switch, are there any special requirements for reinstalling a DAG member node?
My first Exchange server holds all the mailbox databases. If I reinstall Exchange on the second “new” server and add it back into the DAG, are all Exchange settings and all mailbox databases automatically synced back to the second DAG node from node 1?
LikeLike
Sorry forgot to mention that I also have an “Application Aware” Veeam backup of this Exchange server with restore points going back as far as mid-December. However, even though its an Application Aware backup, I still hear many people say to never restore an Exchange server from backup, but rather start from scratch with the /m:recoverserver switch.
What’s the point of backing up your Exchange VMs anyway if you can’t ever restore them from backups? What is your verdict on this?
LikeLike
Well, there is the possibility to restore an older VM that is already compromised, so I can understand that. That risk is not there when installing a brand new Exchange server of course.
Personally (not a verdict) I don’t care about VM backups, as long as the data is safe and the config is well documented and stored I’m perfectly happy. But, other mileage may vary though
LikeLike
Hi,
recovering a server in a DAG is a bit more work. Remember that all information is stored in Active Directory and that also includes mailbox database copies. As soon as you recover the server and reboot it, it knows it must have copies on it. You have to work with that.
I have an old blog (almost 10 years old) on Exchange 2010 DAG recovery here. Haven’t checked it, but it might server as a guideline: https://jaapwesselius.com/2011/08/01/recover-a-mailbox-server-in-a-dag/
LikeLike
Hi Jaap, love your articles, always informative and I learn a lot from them.
I have a customer who’s insurance company now forced them to rebuild their whole environment, even AD from scratch. Only the DB can be kept from the Exchange 2016 server.
Do you have any suggestions as to how about how to go about this? We quoted 120 hours for the job, but if we have to export 800gb of user mailboxes to pst’s and then back into the new system, that will be taking a long time.
Your thoughts?
Thanks
Gelbert
LikeLike
Good lord, there are so many options in this scenario. Is it an option to clone Active Directory? That would help tremendously. If not, use the same domain name, Exchange organisation name etc. which will help you recover. Use the old database and see if you can use that as a recovery database. Or worst case, create a PST file for every mailbox in the database. And if you have to rebuild AD (without a clone), export all AD properties. And use PowerShell to get all additional mailbox information, delegates etc.
Pfff… now I think about it, you can spend 120 alone on all options, good luck!
LikeLike
Yeah, they got hit by a full system wide ransomware attack about 4 months ago, and we tasked to rebuild their environment. While planning the rebuild they got hit with the Hafnium breach. We are currently waiting on the CDO/Security team to advise if we can do a ADMT export, which are doubting, or a complete ground up build. Seems there is lingering questions about migrating AD objects, especially GUID/SID’s as they might have been compromised.
Hence me asking what route will be easier/less time consuming.
LikeLike
I can understand the concerns. You can use any tool to export users, groups, contacts etc to a CSV file, so at least you have all that information.
LikeLike
Yes, don’t try to recover Exchange on a newer OS that what it was previously running on. Also previous server was CU20 build so i downloaded the CU20 build of exhange install. My Exchange was on C drive and database was on the E drive. Setup the new server with the same exact structure. Installed OS and all prereqs. Ran the recover command and it was smooth. For some reason I needed to dismount the database and then remount in exchange shell. Once that was done email started flowing. Only minor issue was that imap and pop services were set to manual and not running, but i just set them to automatic and started them.
LikeLike
Yesterday I recovered an Exchange 2019 on Windows 2019 Server Core, the process was the same. It took me a couple of minutes to figure out how to assign the drive letters and mount points. In my experience the mailbox databases do not mount at the first startup, so mount them manually.
LikeLike