After decommissioning the Resource Forest I still have an Exchange 2016 environment on-premises, but all my mailboxes are in Office 365. Users are provisioned in Active Directory, Remote Mailboxes are provisioned in Exchange 2016 and everything is synchronized to Office 365 using Azure AD Connect.
Do I still need an Exchange Hybrid Configuration? Unless there are plans to move resources back to Exchange on-premises there’s no need for a Hybrid Configuration. To stay in a supported configuration, an Exchange server on-premises is still needed for management purposes, but only Azure AD Connect is needed and not a full hybrid configuration.
Note. If you want to use the on-premises Exchange server for SMTP relay purposes you don’t need the Hybrid configuration either. Just make sure you have a SMTP Send Connector that points to Exchange Online Protection and you’re good.
Removing the Hybrid configuration consists of the following steps:
Disable Autodiscover SCP in Exchange
Remove the Hybrid Configuration from Active Directory
Remove Connectors in Exchange Online
Remove the Organization Sharing from Exchange Online
Disable OAuth
Disable Autodiscover SCP in Exchange
When all Exchange resources are in Exchange Online you no longer need the on-premises Service Connection Points (SCP) for Autodiscover. But make sure you have the correct CNAME records for Autodiscover that point to Autodiscover.outlook.com.
To disable the SCP records in Active Directory, execute the following command in Exchange Management Shell:
Remove the Hybrid Configuration from Active Directory
Removing the Hybrid Configuration from Active Directory is just one PowerShell command in Exchange Management Shell:
Remove-HybridConfiguration -Confirm:$false
There’s one pitfall here, this will also remove the outbound to Office 365 Send Connector from Exchange. If you want to keep SMTP relay from on-premises to your mailboxes in Exchange Online you have to manually recreate this connector (use yourdomain-com.mail.protection.outlook.com as a smarthost for this)
Remove Connectors in Exchange Online
In the Exchange Online Admin Center, remove the outbound SMTP connectors that point from Exchange Online to your on-premises Exchange organization. If you want to keep SMTP routing, keep the inbound SMTP connector, otherwise you can remove this as well.
Remove the Organization Sharing from Exchange Online To remove the Hybrid Organization Sharing from Exchange Online navigate to Organization | Sharing in the Exchange Admin Center and remove the organization sharing.
Disable OAuth on-premises
When used before you can disable the OAuth configuration as well from Exchange on-premises and Exchange Online.
In Exchange on-premises Management Shell, execute the following command:
These are the steps needed to remove the Hybrid Configuration from your Exchange environment.
Note. Microsoft recommends to leave the Exchange Hybrid option in Azure AD Connect.
Summary
In this blogpost I explained how to remove the Hybrid Configuration from your Exchange environment after you have moved all resources to Exchange Online.
The on-premises Exchange server is still needed for management purposes. After removing the Hybrid Configuration you can still manage your recipient Exchange Online using the on-premises Exchange server, all changes are replicated through Azure Active Directory.
Is that last Exchange server on-premises still needed? Yes, you need it for managing your recipients in Exchange Online. When you have Azure AD Connect running in your environment, the objects are managed in on-premises Active Directory. The source of authority is Active Directory. As long as Microsoft hasn’t fixed the source of authority problem, an Exchange server on-premises is still needed.
Two years ago I wrote a number of blogpost regarding an Exchange Resource Forest model and Office 365 starting at Exchange Resource Forest and Office 365 part I. There are four articles, at the end of each article you’ll find a link to the next article.
Over the years I have received several requests about how to decommission the Exchange Resource Forest when all mailboxes have been migrated to Office 365. This is a valid question, why keeping an empty forest with only an Exchange server, including the complex configuration with provisioning remote mailboxes. It is not too difficult to achieve, it’s just a matter of moving the Exchange server from the resource forest to the account forest. Unfortunately, not a word about supportability of such a migration, but it is a method that Microsoft uses for collapsing forests or Active Directory migrations.
To decommission the resource forest, the following steps need to be executed:
I will go through these steps in this last article of the series.
Step 0. The existing situation
This series started with Exchange 2010 in a resource forest and was configured in a hybrid environment. Exchange 2010 was upgrade to Exchange 2016 and now all mailboxes are moved to Office 365. The existing situations is shown in the image below:
Exchange 2016 is configured in a hybrid environment, there’s an Exchange 2016 Edge Transport server and the MX record points to this environment. Outbound email is sent directly from Exchange Online, but in my environment MX has never changed (because of Public Folders have existed for a long time in Exchange 2016). Azure AD Connect is running in the account forest and is combining information from the account and the resource forest, and sends this to Azure AD.
Step 1. Install Exchange in the account forest
The first step is to install Exchange 2016 in the account forest. This will make changes to the Active Directory Schema, Configuration and Domain partition, but existing user accounts will not be touched at this point. Because of this, Azure AD Connect won’t notice anything and no changes will be sent to Azure Active Directory.
Warning! Service Connection Point! When installing Exchange 2016 in the account forest you have to be aware that a Service Connection Point (SCP) is created during installation. Depending of the outlook clients being used, they can pick up this SCP in AD within minutes and when not configured correctly Outlook clients can end up with a corrupted profile (been there, done that unfortunately). To work around this, you can use Tony Murrays script as mentioned in the SSL Certificate warning during or after Exchange server setup blog. Make sure you use the same AutodiscoverURL as used in the resource forest (or set the well-known GUID of the SCP to NULL of course).
Other than this I won’t go into detail about installing the Exchange 2016 server, and I assume the Virtual Directories, SSL certificates, internal- and externalURL are identical. It is a new and most likely greenfield deployment, so you have to configuring everything, ranging from the Offline Address Book to the OWA and ActiveSync policies and everything between in between. Oh, and don’t forget stuff like the Accepted Domains and Email Address Policies to set the correct email addresses when creating new remote mailboxes. Did I also mention the routing infrastructure and SMTP relay options? It looks simple, but there’s quite a lot of work involved before even starting to move resources….
Step 2. Move Exchange resources to the account forest
The second step is to move resources from the resource forest to the account forest, to begin with the Distribution Groups followed by the mailboxes.
Moving Distribution Groups
You can use ADMT to clone Distribution Groups but exporting Distribution Groups using PowerShell works fine as well, for example with the following commands.
Import the CSV file in Exchange 2016 in the accounts forest and you’re good. Optionally you can export and import options like HiddenFromAddressListEnabled, MemberJoinRestriction, MemberDepartRestriction and ManagedBy when used of course. As long as the PrimarySmtpAddress property of the new Distribution Group is identical to the Distribution Group in the resource forest Azure AD won’t complain and everything will continue to work in Office 365 as expected.
Similar to this you can use the Get-ADGroupMember PowerShell command to export members of Distribution Groups and use Add-ADGroupMember to import the members in Distribution Groups in the accounts forest.
Moving Mailboxes
Moving mailboxes from the resource forest to the account forest is not a big thing. Remember that Azure AD Connect combines the user account in the account forest and the mailbox account in the resources forest and sends this to Azure AD. So, you can safely add properties to the user account in the account forest without breaking things.
You can use the Enable-RemoteMailbox command on the user account in the account forest without breaking things in Azure AD Connect. This also means that during testing you can use the Disable-RemoteMailbox command in the accounts forest, and Azure AD Connect won’t complain either.
To move mailboxes from the resource forest to the account forest I used PowerShell to export a list of mailboxes and their properties to a CSV file containing the following properties:
PrimarySmtpAddress.
RemoteRoutingAddress.
ExchangeGuid.
Alias.
DisplayName.
EmailAddresses (beware, this is a multi-value attribute).
HiddenFromAddressListsEnabled.
To export these properties, you can use the following commands on an Exchange server in the resource forest to get all information in a CSV file:
In my hybrid resource forest I had a few room mailboxes. These had a disabled account in the resource forest, but no account in the accounts forest. These had to be converted to linked mailboxes. To do this and to avoid disruption in Office 365 I moved the room mailbox back to Exchange 2016 and converted this room mailbox to a linked mailbox using the following commands:
And then moved the room mailbox back to Exchange Online to continue with the process of moving mailboxes to the account forest.
Quad Erat Demonstrandum, after enabling the remote mailboxes in the account forest and importing all additional properties Azure AD Connect continued to run as expected, no errors were logged in Azure AD and nothing strange happened in Exchange Online.
Step 3. Reconfigure Azure AD Connect
When all mailboxes and Distribution Groups are moved to the accounts forest, Azure AD Connect can be reconfigured. This is not a big deal. In the resource forest environment, Azure AD Connect takes the user account details from the accounts forest and combines this with the mailbox information from the resource forest based on the MasterAccountSID. But now all data can be read from the account forest, the resource forest can just be removed from Azure AD Connect.
When you start Azure AD Connect on the Azure AD Connect server, click Configure and select Customize Synchronization Options from the Additional tasks menu you’ll find there’s only a way to add additional forests and not the option to remove it.
To remove the resources forest from Azure AD connect the connectors to this forest should be removed from the configuration. This can be achieved using the MIISClient on the Azure AD Connect server (can be found in C:\Program Files\Microsoft Azure AD Sync\UIShell\miisclient.exe). I typically have a shortcut to this useful application on my desktop.
Before starting stop the synchronization using the following PowerShell command:
Set-ADSyncScheduler -SyncCycleEnabled $False
Open the Synchronization Service Manager (MIISClient.exe) and click on Connectors. Select the connector to the resource forest, right click and select delete.
In the pop-up select the Delete Connector and Connector Space option, this will remove the connector and all configuration data. Click Yes to confirm and the forest is removed from Azure AD Connect.
The second step in reconfiguring Azure AD Connect is to refresh the schema so that Azure AD Connect ‘knows’ all properties are now coming from the account forest. In Azure AD Connect click on Refresh directory schema, enter the account forest credentials and that’s all.
Enable the synchronization again and start a full sync using the following PowerShell commands:
Set-ADSyncScheduler -SyncCycleEnabled $True
Start-ADSyncSyncCycle -PolicyType Initial
At this point we’ve successfully moved all Exchange resources from the resource forest to the account forest and removed the resource forest from Azure AD Connect. Exchange 2016 in the resource forest is no longer used and can be decommissioned at all. Before doing this, makes sure you copy the contents from the old administrator mailbox (in the resource forest) to the new administrator mailbox (in the account forest).
Cool: since there still is an Edge Subscription between the Edge Transport server and the old Exchange 2016 server (which at this point still has the remote mailboxes) mail flow is unaffected and is still delivered to the correct mailbox in Exchange Online.
Rerouting inbound mailflow is easy in this scenario. Create a new Edge Subscription between the existing Edge Transport server and the new Exchange 2016 server and you’re good. Make sure there’s a Send Connector from Exchange 2016 to Exchange Online Protection as well.
Note. Don’t forget to reconfigure internal mail flow like SMTP relay, applications, (multi-functional) devices etc.
Step 4. Reconfigure the Hybrid Configuration
Do you need to reconfigure the Hybrid Configuration? Theoretically not, Active Directory and Exchange information is sent to Azure Active Directory using Azure AD Connect and there’s no need to have a hybrid configuration. But, I also use the hybrid configuration for SMTP relay (I know, there are different solutions) and I’d like to have the option to move mailboxes back to Exchange on-premises, just in case.
Running the Hybrid Configuration Wizard here is not different than any other HCW deployment so I won’t go into detail about that.
Step 5. Decommission the Resource Forest
At this point we have a fully operations account forest with Exchange 2016 installed and Azure AD Connect up-and-running. The ‘old’ resource forest is not operational anymore and can be decommissioned. Remove the forest trust between the account forest and the resource forest and remove it. This can be as simple as turning off the VM’s and destroying them (never thought I would write that one down here 😊).
Summary
The Exchange resource forest has been used for quite a time, but with the move to Office 365 I have several customers decommissioning their resource forest. Although maintaining a resource forest can be quite complex, decommissioning it is not.
Install Exchange 2016 in the account forest, move Exchange resources to the account forest and reconfigure Azure AD Connect. These are the most important steps, although provisioning, SMTP relay and corresponding infrastructure are important too, and will take the most time I’m afraid.
But, when the resource forest is finally decommissioned it’s not anymore than a regular single forest, single domain Exchange hybrid configuration.
As a consultant in messaging and collaboration I have created dozens of Exchange hybrid configurations that last years, ranging from Exchange 2010 to Exchange 2019.
Today we’ve run into an issue I have not seen before, the HCW8001 – Unable to determine the Tenant Routing Domain error:
Azure AD Connect was running fine, I checked the configuration (optional features) and found nothing strange and no errors were logged.
When checking the Microsoft Portal I could see Directory Synchronization was running:
The tenant routing domain is typically something like <tenantname>.mail.onmicrosoft.com and this is set in Office 365 when installing Azure AD Connect. But when checking the Accepted Domains (in Exchange Online) this domain is not available:
There’s no way you can add this <tenantname>.mail.onmicrosoft.com domain manually (also not via the Microsoft Online Portal) so you are out of luck (I tried the Azure AD Connect server multiple times, but it didn’t work).
You can open a ticket with Microsoft Support or see if you can create a new tenant and start over again. Since this happens rarely I would be surprised you run into this again with a new tenant.
One of my clients is running Exchange 2010 in hybrid mode, and they have Outlook 2010 and Outlook 365 ProPlus client. For testing purposes, I have two VMs, one with Windows 7 and Office 2010 and one with Windows 10 and Office 365 ProPlus. And every Monday morning I run the Windows 7 VM for an hour or so to see if everything is working fine 😊
This morning my Outlook 2010 was working offline, and it didn’t want to go online (OWA and Outlook 365 ProPlus were working fine). Remove the Outlook profile but creating a new Outlook profile didn’t work. After a minute the dreaded an encrypted connection to your mail server is not available error message appeared:
Mostly this is caused by Autodiscover that goes wrong somewhere, the Remote Connectivity Analyzer shows that Autodiscover to the on-premises Exchange 2010 goes well, but that the redirect to Exchange Online goes wrong and it generates the following error message:
An HTTP 456 Unauthorized response was received from the remote Unknown server. This indicates that the user may not have logged on for the first time, or the account may be locked. To logon, go to http://portal.microsoftonline.com.
The AppPasswordRequired explains more. Last week I changed the MFA settings (see previous authenticator app for Office 365 blogpost). This works fine for OWA and Office 365 ProPlus, but not for Outlook 2010. Since Outlook 2010 does not work with Office 365 MFA, especially not in a hybrid environment (not even with an App Password).
The only workaround here was to temporarily disable MFA for my user account, create a new Outlook profile (which worked fine without MFA) and re-enable MFA. Again, Outlook 2010 does not recognize the MFA and still works with Exchange Online using basic authentication, but all other Office 365 services work fine with Office 365 MFA (both SMS and Authenticator authentication).
Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.
When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.
But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:
It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).
Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.
This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).
The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:
When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.
But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.
When opening the certificate store using PowerShell using the following commands:
CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem
All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:
That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:
After restarting the Transport service cross-premises mail flow works again.
The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊
