Hafnium and Exchange mitigation and remediation

Last week Microsoft discovered a zero-day vulnerability for Exchange (which was initially detected by security companies last January) and an urgent patch was released. Unfortunately this patch is only available for recent versions of Exchange 2019 and Exchange 2016 and the last version of Exchange 2013. If you have an older version of Exchange running you have to bring it to the latest Cumulative Update first and then deploy the Security Update.

There are some mitigation rules available though:

  • Exchange server that are not available on the Internet are much less vulnerable (ok, this is an open door, I know). I have two customers that have their Exchange servers available only via a VPN connection. This works well from a security perspective.
  • Similar to the previous bullet, Exchange hybrid servers should not be publicly available, only Exchange Online must be able to access the Exchange on-premises servers. URL’s and IP ranges can be found on https://docs.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide.
  • Microsoft also posted a number of mitigation rules on the Microsoft Security Response Center Blog. These mitigation rules are temporary though and should only be used until the Exchange servers are fully patched. Mitigation rules are an IIS Re-Write Rule, disabling UM services and disable OAB en ECP Application Pool.

Microsoft has published a script (Test-ProxyLogon.ps1) on GitHub that can be used to check your Exchange servers if they are compromised. This script can be found on CSS-Exchange/Security at main · microsoft/CSS-Exchange · GitHub.

When you run the script it will show in seconds if something is found:

Too bad that this is a production Exchange 2016 server that was compromised.

At this moment I would recommend to turn off and remove the Exchange server and rebuild it using the /Mode:RecoverServer option of the Exchange setup application. This is documented in my next blog Rebuild your server (after HAFNIUM infection. When other (better or easier) recommendations are published I’ll update this blog.

Last updated: March 10, 2021

Zero Day Vulnerabilities Discovered in all Versions of Microsoft Exchange Server

Microsoft has detected multiple 0-day exploits being used to attack on-premises versions of Exchange Server in limited and targeted attacks. In the campaigns observed, threat actors used this vulnerability to access on-premises Exchange servers, which enabled access to email accounts, and install additional malware to facilitate long-term access to victim environments.  Microsoft Threat Intelligence Center (MSTIC) attributes this campaign with high confidence to HAFNIUM, a group assessed to be state-sponsored and operating out of China, based on observed victimology, tactics, and procedures.

Because Microsoft is aware of active exploits of related vulnerabilities in the wild (limited targeted attacks), Microsoft released security updates for four different Exchange Server vulnerabilities. Microsoft strongly urge customers to update on-premises Exchange servers immediately to protect against these exploits and prevent future abuse across the ecosystem.  Even though Microsoft has worked quickly to deploy an update for the HAFNIUM exploits, it is known that many nation-state actors and criminal groups will move quickly to take advantage of any unpatched systems for years to come. Promptly applying the security updates is the best protection against this attack.

To stress the importance of this issue, Microsoft conducted a series of webcasts earlier today, at least in the APAC and EMEA regions. Hand-out of the webcast is available at https://aka.ms/ExOOB.

A few remarks:

  • All Exchange server versions are affected and the exploit has been detected on Exchange 2013, Exchange 2016 and Exchange 2019.
  • If you have restricted your firewall to Microsoft only (when running Exchange hybrid) you are less vulnerable, but the risk is not reduced to zero.
  • Updates are available for the current CU and the CU before. If you are on an older version of Exchange, please upgrade to the current CU before you can install this Security Update. If you run a really old version of Exchange you will run into .NET Framework update issues. Please use the overview of fellow MVP Michel de Rooij: Upgrade Paths for CU’s & .NET | EighTwOne (821)

Updates are available for :

Please note that the security updates are CU specific. Also, Security Updates are cumulative, so this security update contains previous security updates as well.

Installation is straightforward, download the update from the Microsoft website (but I already saw it appear in our WSUS environment this morning) and install it. Personally I start the update from an elevated command prompt. Installation is the same on Windows 2019 server core as well as Windows 2019 Desktop Experience.

If you have Exchange servers in a DAG, don’t forget to put them in maintenance mode first. After installation the Exchange servers need to be rebooted.

More information, including CVE information can be found on:

Released: March 2021 Exchange Server Security Updates – Microsoft Tech Community

Multiple Security Updates Released for Exchange Server – Microsoft Security Response Center

Description of the security update for Microsoft Exchange Server 2019, 2016, and 2013: March 2, 2021 (KB5000871)

HAFNIUM targeting Exchange Servers with 0-day exploits – Microsoft Security

Transfer FSMO Role Win32 error returned is 0x20af

For a client I had to test some domain controller scenarios and issues. There are two domain controllers (DC01 and DC02) and a new one (NewDC) is added. FSMO roles are moved to NewDC and DC01 must be decommissioned. No big deal you would say….

Domain FSMO roles (RID Master, PDC and Infrastructure master) moved in an instant, but moving the forest FSMO roles (Schema master and Domain Naming master) failed with the following error, both in the GUI as well as NTDSUTIL:

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
Depending on the error code this may indicate a connection, ldap, or role transfer error.

Using Netdom query FSMO I could see the roles.

This happened when logged on the AD01 as well as NewDC, connectivity was not an issue and I did not notice any strange issues with Exchange and Outlook or OWA clients.

Using Google I found several other similar issues, and the overall recommendation was to seize the FSMO roles. Seizing FSMO roles is an option when the old Domain Controller is no longer available, but in my scenario I had to keep the old Domain Controller for testing.

But instead of using Google I had to look in the eventlog in the first place. The first entry I found was a confirmation that the roles could not be transferred, but Additional Data Error value: 4 was not very helpful:

An attempt to transfer the operations master role represented by the following object failed.

Current operations master role:
CN=NTDS Settings,CN=DC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=msexchangebook,DC=com
Proposed operations master role:
CN=NTDS Settings,CN=NIEUWEDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=msexchangebook,DC=com

Additional Data
Error value:

Subsequent entries were more helpful, Event 2019, ActiveDirectory_DomainService indicated replication issues:

Or in tekst for Search Optimization:

This server is the owner of the following FSMO role, but does not consider it valid. For the partition which contains the FSMO, this server has not replicated successfully with any of its partners since this server has been restarted. Replication errors are preventing validation of this role.

Operations which require contacting a FSMO operation master will fail until this condition is corrected.

FSMO Role: CN=Partitions,CN=Configuration,DC=msexchangebook,DC=com

User Action:

  1. Initial synchronization is the first early replications done by a system as it is starting. A failure to initially synchronize may explain why a FSMO role cannot be validated. This process is explained in KB article 305476.
  2. This server has one or more replication partners, and replication is failing for all of these partners. Use the command repadmin /showrepl to display the replication errors. Correct the error in question. For example there maybe problems with IP connectivity, DNS name resolution, or security authentication that are preventing successful replication.
  3. In the rare event that all replication partners are expected to be offline (for example, because of maintenance or disaster recovery), you can force the role to be validated. This can be done by using NTDSUTIL.EXE to seize the role to the same server. This may be done using the steps provided in KB articles 255504 and 324801 on http://support.microsoft.com.

The following operations may be impacted:
Schema: You will no longer be able to modify the schema for this forest.
Domain Naming: You will no longer be able to add or remove domains from this forest.
PDC: You will no longer be able to perform primary domain controller operations, such as Group Policy updates and password resets for non-Active Directory Domain Services accounts.
RID: You will not be able to allocation new security identifiers for new user accounts, computer accounts or security groups.
Infrastructure: Cross-domain name references, such as universal group memberships, will not be updated properly if their target object is moved or renamed.

In the end it turned out that DNS was not configured correctly, hence the replication issues. Fixed the DNS settings on the Domain Controllers and after a short while the Domain Controllers resumed replication and I was able to move the Forest FSMO roles to the new Domain Controller.

So in short, always check eventlog, replication and DNS when troubleshooting Domain Controller issues 😉

DKIM record in WordPress DNS

So, today I found out that outbound mail from my jaapwesselius.com did not have a DKIM signature (after mail was blocked by prodigy.net). I have my jaapwesselius.com running on WordPress.com. To do this, WordPress requires to have DNS hosted with them. No problem, but adding a DKIM record in WordPress DNS is not possible, it fails with a TXT records may not exceed 255 characters error message as shown below:

The solution is relatively simple. You can add a CNAME record for the original DKIM record. For example, have safemail._domainkey.jaapwesselius.com point to something like safemailhop.exchangelabs.nl (I own that domain too, and DNS is hosted at my provider Argeweb).

CNAME: safemail._domainkey.jaapwesselius.com safemailhop.exchangelabs.nl

Create a new TXT record safemailhop.exchangelabs.nl and add the original DKIM record (from my jaapwesselius.com domain) to it et voila, that’s it.

Check with https://mxtoolbox.com/dkim.aspx reveals that it works:

And some header information:

Note. Yes I know, p=NONE in the DMARC record could (should/must) be changed to quarantine or even REJECT, but I’m still in development 😊

A reboot from a previous installation is pending. Please restart the system and then rerun Setup

While installing the Exchange 2019 Management Tools (only the Management Tools) on a server, I ran into the error message “A reboot from a previous installation is pending. Please restart the system and then rerun Setup”

Normally a reboot fixes this problem, but unfortunately this time it did not fix it.

The option to reboot is also logged in the registry of the server. There is a key called PendingFileRenameOperations located in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager which in this case has a certain data that was not cleaned up previously:

When you check the data you can even see which process did not clean up. Remove the data from the key (or remove the entire key) and continue with the installation.

Microsoft UC Specialist