Tag Archives: Exchange Hybrid

HCW8001 – Unable to determine the tenant routing domain

As a consultant in messaging and collaboration I have created dozens of Exchange hybrid configurations that last years, ranging from Exchange 2010 to Exchange 2019.
Today we’ve run into an issue I have not seen before, the HCW8001 – Unable to determine the Tenant Routing Domain error:

Unable to determine the tenant routing domain

Note. In one of my earlier blogs Moving from Exchange 2010 to Office 365 somebody mentioned this specific error in the comments in September 2018.

When you click Learn more you are redirected to the following Microsoft page:
https://support.microsoft.com/en-us/help/3068010/unable-to-determine-the-routing-domain-for-the-cloud-organization-erro which states you have to enable Directory Synchronization using the following command:

Set-MsolDirSyncEnabled -EnableDirsync $true

Which of course we did, but no success.

Azure AD Connect was running fine, I checked the configuration (optional features) and found nothing strange and no errors were logged.

Optional Features

When checking the Microsoft Portal I could see Directory Synchronization was running:

Azure AD Connect

The tenant routing domain is typically something like <tenantname>.mail.onmicrosoft.com and this is set in Office 365 when installing Azure AD Connect. But when checking the Accepted Domains (in Exchange Online) this domain is not available:

Accepted Domains

There’s no way you can add this <tenantname>.mail.onmicrosoft.com domain manually (also not via the Microsoft Online Portal) so you are out of luck (I tried the Azure AD Connect server multiple times, but it didn’t work).

You can open a ticket with Microsoft Support or see if you can create a new tenant and start over again. Since this happens rarely I would be surprised you run into this again with a new tenant.

 

Outlook 2010 stays offline with Exchange Online

One of my clients is running Exchange 2010 in hybrid mode, and they have Outlook 2010 and Outlook 365 ProPlus client. For testing purposes, I have two VMs, one with Windows 7 and Office 2010 and one with Windows 10 and Office 365 ProPlus. And every Monday morning I run the Windows 7 VM for an hour or so to see if everything is working fine 😊

This morning my Outlook 2010 was working offline, and it didn’t want to go online (OWA and Outlook 365 ProPlus were working fine). Remove the Outlook profile but creating a new Outlook profile didn’t work. After a minute the dreaded an encrypted connection to your mail server is not available error message appeared:

An encrypted connection to your mail server is not available

Mostly this is caused by Autodiscover that goes wrong somewhere, the Remote Connectivity Analyzer shows that Autodiscover to the on-premises Exchange 2010 goes well, but that the redirect to Exchange Online goes wrong and it generates the following error message:

An HTTP 456 Unauthorized response was received from the remote Unknown server. This indicates that the user may not have logged on for the first time, or the account may be locked. To logon, go to http://portal.microsoftonline.com.

And further down more details are revealed:

X-AutoDiscovery-Error: LiveIdBasicAuth:AppPasswordRequired:<RequestId=8a51c25b-9213-4873-aff8-ebc1da40544f>;

An HTTP 456 Unauthorized response was received from the remote Unknown server

The AppPasswordRequired explains more. Last week I changed the MFA settings (see previous authenticator app for Office 365 blogpost). This works fine for OWA and Office 365 ProPlus, but not for Outlook 2010. Since Outlook 2010 does not work with Office 365 MFA, especially not in a hybrid environment (not even with an App Password).

The only workaround here was to temporarily disable MFA for my user account, create a new Outlook profile (which worked fine without MFA) and re-enable MFA. Again, Outlook 2010 does not recognize the MFA and still works with Exchange Online using basic authentication, but all other Office 365 services work fine with Office 365 MFA (both SMS and Authenticator authentication).

Exchange Hybrid TLS negotiation failed with error NoCredentials

Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.

When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.

But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:

LastError : [{LED=451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out};{MSG=};{FQDN=exchangelabsnl-mail-onmicrosoft-com.mail.protection.outlook.com}; {IP=104.47.10.36};{LRT=5/2/2019 6:32:14 AM}]

421 4.4.1 connection timed out

It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).

Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.

TLS Negotiation failed with error NoCredentials

This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).

The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:

Has Private Key

When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.

Certificate Thumbprint

But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.

When opening the certificate store using PowerShell using the following commands:

CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem

All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:

Check private key

That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:

Get-ChildItem | ?{$_.Thumbprint -like “B79*”} | Remove-Item

After restarting the Transport service cross-premises mail flow works again.

The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊

 

Moving from Exchange 2010 to Office 365 Part III

Exchange 2010 hybrid and Edge Transport Server

In two previous blog posts I explained how to setup an Exchange 2010 hybrid environment. In these blog posts I used the Exchange 2010 (multi-role) server for the hybrid configuration, so both the Exchange Web Services (used for free/busy, Mailbox Replication Service, OOF, mail tips) and the SMTP connection between Exchange Online and Exchange 2010.

Now that Exchange 2010 end of support is getting closer (less than a year!) I get more questions regarding the move from Exchange 2010 to Exchange Online. And several questions include the use of an Exchange 2010 Edge Transport server in front of the Exchange 2010 multi-role server.

This configuration will look something like this:

exchange 2010 hybrid edge transport

Inbound mail from Internet is getting through Exchange Online Protection and when the mailbox is still on Exchange 2010 it is routed via the Edge Transport Server to the internal Exchange organization. Outbound mail is leaving the organization via the Edge Transport server or via Exchange Online Protection, depending of the location of the mailbox.

The challenge when configuring this in Exchange 2010 is shown in the following screenshot:

missing edge transport server

Compared to running the HCW on Exchange 2013 or Exchange 2016 there’s no option to configure the Edge Transport server for secure mail transport in Exchange 2010!!
The only option right now is to run the Hybrid Configuration Wizard, configure it using the Client Access and Mailbox servers option, but use the data for the Edge Transport where needed.

So, run the Hybrid Configuration Wizard, and when you need to enter the public IP address of the transport servers, enter the Public IP address of the Edge Transport server and not the public IP address of the load balancer VIP pointing to the Exchange 2010 internal servers). In my environment, the webmail.inframan.nl points to 176.62.196.253 and the Edge Transport Server smtphost.inframan.nl points to 176.62.196.245. This is the IP address I am going to use as shown in the following screenshot:

hcw public ip address

The next step is to select the certificate that’s used on the Edge Transport Server. By default, the HCW will only look at the internal Exchange 2010 server, so it won’t find any certificate installed on the Edge Transport server. To overcome this, I have imported the certificate of the Edge Transport Server in the certificate store of the internal Exchange 2010 server used by the HCW. In the HWC, click on the drop down box and select the certificate of the Edge Transport server, in my environment the smtphost.inframan.nl certificate:

hcw loading certificates

The last step is where the FQDN of the organization needs to be entered. I have a lot of discussion here because most admins want to enter something like ‘hybrid.domain.com’, but the FQDN of the transport server needs to be entered here, so in my environment this is the FQDN of the Edge Transport Server, i.e. smtphost.inframan.nl. This FQDN is used (together with the certificate information) to create a Send Connector from Exchange Online to the Edge Transport server.

hcw organization fqdn

Finish the Hybrid Configuration Wizard. It will be configured in Exchange 2010 and in Exchange Online and after a short time you can close the HCW:

hcw configure organization relationship

Now, when looking at the Exchange Management Console you can see the Send Connector from Exchange 2010 to Exchange Online. It is configured with the FQDN smtphost.inframan.nl as expected, but the source server of the Send Connector is still the internal Hub Transport server as shown in the following screen shot:

outbound to office 365 connector

Remove the Hub Transport server entry and add the Edge Transport server instead. If you have an Edge Synchronization in place you will see it immediately when you click the Add button.

In Exchange Online, the Receive Connector that’s created will check for any certificate with a wildcard like name, so the smtphost.inframan.nl certificate will automatically be accepted. The Send Connector is also created correctly. The FQDN of the Edge Transport server is used as the server to route message to, and the CN of the certificate that was selected in the HCW is also configured as shown in the following screenshot:

office 365 send connector certificate

We’re almost there. The only thing that needs to be done is to configure the Receive Connector on the Edge Transport Server for TLS from Exchange Online. You should have already configured the Edge Transport Server with the correct 3rd party certificate and when setting up an inbound connection it should use the 3rd party certificate. You can test this using the https://checktls.com tool online.

On the Edge Transport server, execute the following command in the Exchange Management Shell:

Get-ReceiveConnector "smtphost\Default internal receive connector SMTPHOST" | Set-ReceiveConnector -Fqdn smtphost.inframan.nl -TlsDomainCapabilities mail.protection.outlook.com:AcceptOorgProtocol

This will make sure the cross-premises email will be treated as internal email (SCL=-1). If you omit this step, there’s always the risk the email will be treated as external (I’ve seen SCL=5 in my environment) and will end up in the user’s Junk Email Folder.

Summary

When configuring an Exchange 2010 hybrid configuration it is not possible to configure an Edge Transport Server in the Hybrid Configuration Wizard. It is possible to configure this in the HCW for Exchange 2013 and Exchange 2016, but for Exchange 2010 this needs some manual changes.

In this blogpost I showed you the steps needed to configure an Edge Transport Server for secure messaging between Exchange Online and Exchange 2010. When configured this way cross-premises email will be seen as internal email and thus treated accordingly.

 

Ignite 2018 – Exchange sessions, the good and a bit ugly

My first blog about Ignite 2018 was more about the keynotes on the first day, Microsoft’s strategy around the cloud and how various application integrate with each other and take advantage of the cloud. But I’m a technical consultant and more interested in the technical stuff. And as a MVP in ‘Office Apps and Service’  (used to be ‘Office Servers and Services’, and before that just ‘Exchange MVP’) my heart is still with Exchange. Although I also attended lots of other sessions, there are better blogs available for these technologies instead of mine 😊

Welcome to Exchange 2019

The first break-out session I attended was BRK “Welcome to Exchange 2019” by Greg Taylor and Brent Alinger. Lots of information was already available since Microsoft released the preview version of Exchange 2019, but some other interesting points were mentioned as well:

  • Exchange 2019 runs on Windows 2019 only. There are so much security features in Windows 2019 that Exchange is using, features that are not available in Windows 2016. The preview version was running on Windows 2016, but the final version won’t.
  • Windows 2019 Server Core is the recommended platform because of the lower footprint and attack surface.
  • Required Forest Functional Level is Windows 2012 R2, which may cause issues with customers I guess.
  • Minimum recommended RAM is 128 GB. Be aware, this is the recommended amount of RAM, not the required amount of RAM. This amount comes from .NET usage in Windows 2019 that does much better performance with lots amount of memory. If an Exchange 2019 server does not have 100GB or more, it won’t take advantage of a lot of these improvements. There’s also a correlation with the amount of processors in the Exchange 2019 server, and this 128 GB is related to 48 processors. If you are using less processors, the memory usage decreases as well.
  • Exchange 2019 will (at least for now) only be available via Volume Licensing. Discussions are going on whether this will lead to piracy via download sites. Microsoft is aware of this, but at this moment it’s only Volume Licensing.
  • A very cool new feature is the MCDB, or metacache database. This is a cache stored on SSD drives where metadata from the Mailbox databases is stored, like search information, folder structure etc. This will improve performance for Outlook clients running in Online Mode, not only search information, but also logon to the Mailbox is dramatically improved (I’m starting to sound like a marketing guy 😊
  • Related to the previous bullet, Search is improved (or rewritten actually) using Bing technology. The indexing information is no longer stored in a separate directory on the disk, but it is stored in the Mailbox, and thus inside the Database. This means that passive copies have the same information, and the same search indexes. A failover will now never fail because search is not healthy in replication, speeding up fail-over times.
  • And Microsoft also released the documentation for Exchange 2019, which can be found on https://docs.microsoft.com/en-us/Exchange/exchange-server?view=exchserver-2019.

Unfortunately this session is not available online yet. It is recorded, but somehow not yet available.

As a side note, Microsoft has organized a side meeting with the Exchange and Outlook Product Groups and a couple of (Exchange) MVPs. In this 2 hour session we could have a decent discussion with all Program Managers in the room, and we were able to express our deepest concerns regarding the announcements and presentations at Ignite. Some people prefer to do this on Twitter, but we think it’s better to discuss with the Program Managers directly. They gave us additional background information, but at the same time were impressed by the feedback we gave them, and will take it back to HQ. They won’t change the product of course, but the marking messeag and documentation (background information) around Exchange 2019 will certainly change.

Securing Exchange Online from modern threats

Another interesting session was BRK3148 “Securing Exchange Online from modern threats”. It all about security, and what steps the bad guys usually take to attach a platform like Exchange Online. And it’s incredibly easy. Every heard of ‘password spray’? This a brute force attack, but the other way around. The bad guy has a list of usernames (UPN = Email address to make their life easy) and standard passwords like Summer2018 or Spring2018. But with a spray attack they take a password, and try this against all users. If not successful then try the next password against all users in the list. Incredibly simple, and unlike a regular brute force attack this does not result in locked out accounts. And we all know it, it’s a matter of little time before a simple password is compromised. In the demo the presenter shows how easy it is, and once logged on how to continue with elevated privileges.

The good news is, this presentation is available on YouTube: https://youtu.be/jnUUioUU_eY

Hybrid Exchange: making it easier and faster to move to the cloud

Exchange hybrid is also a hot topic. Last year Jeff Kizner did a session on hybrid, and announced Microsoft was working on removing the last Exchange server when all Mailboxes are moved to the cloud. Expectations were high when attending this years session BRK3143 “Hybrid Exchange: making it easier and faster to move to the cloud”.

The first part of this presentation is about the work done on the Organization Configuration Transfer in the Hybrid Configuration Wizard. Still not finished, but a lot of the configuration information cannot be copied over to Exchange Online. It is copying to Exchange Online, there’s no synchronization. So when making changes in Exchange on-premises, they are not transferred automatically to Exchange Online. You have to either make the changes in Exchange Online, or run the Hybrid Configuration Wizard again.

Completely new (and not available yet) is the Hybrid Agent that runs on-premises. The hybrid agent works with an endpoint in Microsoft Azure, and is outbound HTTPS traffic only. Exchange Online is configured with the HCW to use the same endpoint in Microsoft Azure. This way only outbound connections are used, and it is no longer needed to make all kinds of firewall changes when configuring Exchange Hybrid. Even better, when Autodiscover and Exchange on-premises are not published this still works, since only Outbound connections are used, and configuration information is stored safely in the endpoint in Microsoft Azure.

Exchange Hybrid Agent

TargetSharingEpr

At this moment it is only going to work with Free/Busy and Mailbox moves using the MRS, but it’s a good start. Next versions can include more features, and using this technique everything is possible, imaging Microsoft Search that’s searching your on-premises Mailbox servers…. I’m not sure if that’s a good idea, and I have an idea how the average security officer will react to a solution like this. Some people will refer to this as a man in the middle that has full access to your Exchange environment (something with Exchange Trusted Subsystem). Also, the Hybrid Agent only supports auto-update, and I’m not sure if I want that to happen on my Exchange servers. The good news, you can run the Agent on dedicated servers instead of the Exchange servers, as long as these servers have a decent Internet connection.

The Exchange Product Group released a blogpost on both topics, which can be found on https://blogs.technet.microsoft.com/exchange/2018/09/26/announced-improvements-to-hybrid-publishing-and-organization-configuration-transfer/.

Also, the presentation itself is available on YouTube: https://youtu.be/QhOh5RCcLu8

Unfortunately not a single word about that last Exchange server on-premises, so at this point this will need to be available for some time I’m afraid.

Email search in a flash! Accelerating Exchange 2019 with SSDs

As already mentioned in the first section of this blog, Microsoft introduced Metacache Database (MCDB) in Exchange 2019. Exchange 2019 will now work in the regular JBOD solutions, but now added are SSD disks in the servers. These SSD disks are used as an additional cache (special data from the mailbox database is stored additionally on the SSD disks) to speed up performance. Think about the message table, mailbox information, message metadata information, all kinds of information that’s regularly needed by Outlook clients, and can now be retrieved by the Exchange 2019 at a much faster pace.

Personally I think this is a very impressive technology, but I don’t see it appear at customers anytime soon. It is build on top of a DAG (should be no issue), but is also using AutoReseed as outlined in the Preferred Architecture. The SSD versus spinning disks ration is 1:3, so for a 12 disk Exchange server, three SSD disks are needed. Now, I don’t see a lot of customers deploying Exchange 2019 this way, at least not the smaller organizations, but maybe these customers are better off with Exchange Online, but that’s a different discussion.

metacache database

metacache guidance

The technology is impressive, and I’m looking forward to test this is a lab. Unfortunately this feature is not yet available in the preview version of Exchange, so you have to wait until the official release of Exchange 2019 later this year.

The presentation is available on YouTube: https://youtu.be/VHrScskhCQk

Stay tuned for more information…