Tag Archives: Exchange Hybrid

Remove Exchange Hybrid Configuration

After decommissioning the Resource Forest I still have an Exchange 2016 environment on-premises, but all my mailboxes are in Office 365. Users are provisioned in Active Directory, Remote Mailboxes are provisioned in Exchange 2016 and everything is synchronized to Office 365 using Azure AD Connect.

Do I still need an Exchange Hybrid Configuration? Unless there are plans to move resources back to Exchange on-premises there’s no need for a Hybrid Configuration. To stay in a supported configuration, an Exchange server on-premises is still needed for management purposes, but only Azure AD Connect is needed and not a full hybrid configuration.

Note. If you want to use the on-premises Exchange server for SMTP relay purposes you don’t need the Hybrid configuration either. Just make sure you have a SMTP Send Connector that points to Exchange Online Protection and you’re good.

Removing the Hybrid configuration consists of the following steps:

  • Disable Autodiscover SCP in Exchange
  • Remove the Hybrid Configuration from Active Directory
  • Remove Connectors in Exchange Online
  • Remove the Organization Sharing from Exchange Online
  • Disable OAuth

Disable Autodiscover SCP in Exchange

When all Exchange resources are in Exchange Online you no longer need the on-premises Service Connection Points (SCP) for Autodiscover. But make sure you have the correct CNAME records for Autodiscover that point to Autodiscover.outlook.com.

To disable the SCP records in Active Directory, execute the following command in Exchange Management Shell:

Get-ClientAccessServer | Set-ClientAccessServer -AutoDiscoverServiceInternalUri $Null

Remove the Hybrid Configuration from Active Directory

Removing the Hybrid Configuration from Active Directory is just one PowerShell command in Exchange Management Shell:

Remove-HybridConfiguration -Confirm:$false

There’s one pitfall here, this will also remove the outbound to Office 365 Send Connector from Exchange. If you want to keep SMTP relay from on-premises to your mailboxes in Exchange Online you have to manually recreate this connector (use yourdomain-com.mail.protection.outlook.com as a smarthost for this)

Remove Connectors in Exchange Online

In the Exchange Online Admin Center, remove the outbound SMTP connectors that point from Exchange Online to your on-premises Exchange organization. If you want to keep SMTP routing, keep the inbound SMTP connector, otherwise you can remove this as well.

Remove the Organization Sharing from Exchange Online
To remove the Hybrid Organization Sharing from Exchange Online navigate to Organization | Sharing in the Exchange Admin Center and remove the organization sharing.

Disable OAuth on-premises

When used before you can disable the OAuth configuration as well from Exchange on-premises and Exchange Online.

In Exchange on-premises Management Shell, execute the following command:

Get-intraOrganizationConnector | Set-IntraOrganizationConnector -Enabled $False

And to do this in Exchange Online Management Shell, execute the following (same) command:

Get-IntraorganizationConnector | Set-IntraOrganizationConnector -Enabled $False

These are the steps needed to remove the Hybrid Configuration from your Exchange environment.

Note. Microsoft recommends to leave the Exchange Hybrid option in Azure AD Connect.

Summary

In this blogpost I explained how to remove the Hybrid Configuration from your Exchange environment after you have moved all resources to Exchange Online.

The on-premises Exchange server is still needed for management purposes. After removing the Hybrid Configuration you can still manage your recipient Exchange Online using the on-premises Exchange server, all changes are replicated through Azure Active Directory.

Is that last Exchange server on-premises still needed? Yes, you need it for managing your recipients in Exchange Online. When you have Azure AD Connect running in your environment, the objects are managed in on-premises Active Directory. The source of authority is Active Directory. As long as Microsoft hasn’t fixed the source of authority problem, an Exchange server on-premises is still needed.

HCW8001 – Unable to determine the tenant routing domain

As a consultant in messaging and collaboration I have created dozens of Exchange hybrid configurations that last years, ranging from Exchange 2010 to Exchange 2019.
Today we’ve run into an issue I have not seen before, the HCW8001 – Unable to determine the Tenant Routing Domain error:

Unable to determine the tenant routing domain

Note. In one of my earlier blogs Moving from Exchange 2010 to Office 365 somebody mentioned this specific error in the comments in September 2018.

When you click Learn more you are redirected to the following Microsoft page:
https://support.microsoft.com/en-us/help/3068010/unable-to-determine-the-routing-domain-for-the-cloud-organization-erro which states you have to enable Directory Synchronization using the following command:

Set-MsolDirSyncEnabled -EnableDirsync $true

Which of course we did, but no success.

Azure AD Connect was running fine, I checked the configuration (optional features) and found nothing strange and no errors were logged.

Optional Features

When checking the Microsoft Portal I could see Directory Synchronization was running:

Azure AD Connect

The tenant routing domain is typically something like <tenantname>.mail.onmicrosoft.com and this is set in Office 365 when installing Azure AD Connect. But when checking the Accepted Domains (in Exchange Online) this domain is not available:

Accepted Domains

There’s no way you can add this <tenantname>.mail.onmicrosoft.com domain manually (also not via the Microsoft Online Portal) so you are out of luck (I tried the Azure AD Connect server multiple times, but it didn’t work).

You can open a ticket with Microsoft Support or see if you can create a new tenant and start over again. Since this happens rarely I would be surprised you run into this again with a new tenant.

 

Outlook 2010 stays offline with Exchange Online

One of my clients is running Exchange 2010 in hybrid mode, and they have Outlook 2010 and Outlook 365 ProPlus client. For testing purposes, I have two VMs, one with Windows 7 and Office 2010 and one with Windows 10 and Office 365 ProPlus. And every Monday morning I run the Windows 7 VM for an hour or so to see if everything is working fine 😊

This morning my Outlook 2010 was working offline, and it didn’t want to go online (OWA and Outlook 365 ProPlus were working fine). Remove the Outlook profile but creating a new Outlook profile didn’t work. After a minute the dreaded an encrypted connection to your mail server is not available error message appeared:

An encrypted connection to your mail server is not available

Mostly this is caused by Autodiscover that goes wrong somewhere, the Remote Connectivity Analyzer shows that Autodiscover to the on-premises Exchange 2010 goes well, but that the redirect to Exchange Online goes wrong and it generates the following error message:

An HTTP 456 Unauthorized response was received from the remote Unknown server. This indicates that the user may not have logged on for the first time, or the account may be locked. To logon, go to http://portal.microsoftonline.com.

And further down more details are revealed:

X-AutoDiscovery-Error: LiveIdBasicAuth:AppPasswordRequired:<RequestId=8a51c25b-9213-4873-aff8-ebc1da40544f>;

An HTTP 456 Unauthorized response was received from the remote Unknown server

The AppPasswordRequired explains more. Last week I changed the MFA settings (see previous authenticator app for Office 365 blogpost). This works fine for OWA and Office 365 ProPlus, but not for Outlook 2010. Since Outlook 2010 does not work with Office 365 MFA, especially not in a hybrid environment (not even with an App Password).

The only workaround here was to temporarily disable MFA for my user account, create a new Outlook profile (which worked fine without MFA) and re-enable MFA. Again, Outlook 2010 does not recognize the MFA and still works with Exchange Online using basic authentication, but all other Office 365 services work fine with Office 365 MFA (both SMS and Authenticator authentication).

Exchange Hybrid TLS negotiation failed with error NoCredentials

Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.

When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.

But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:

LastError : [{LED=451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out};{MSG=};{FQDN=exchangelabsnl-mail-onmicrosoft-com.mail.protection.outlook.com}; {IP=104.47.10.36};{LRT=5/2/2019 6:32:14 AM}]

421 4.4.1 connection timed out

It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).

Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.

TLS Negotiation failed with error NoCredentials

This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).

The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:

Has Private Key

When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.

Certificate Thumbprint

But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.

When opening the certificate store using PowerShell using the following commands:

CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem

All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:

Check private key

That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:

Get-ChildItem | ?{$_.Thumbprint -like “B79*”} | Remove-Item

After restarting the Transport service cross-premises mail flow works again.

The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊