Category Archives: Exchange Hybrid

Outlook 2010 stays offline with Exchange Online

One of my clients is running Exchange 2010 in hybrid mode, and they have Outlook 2010 and Outlook 365 ProPlus client. For testing purposes, I have two VMs, one with Windows 7 and Office 2010 and one with Windows 10 and Office 365 ProPlus. And every Monday morning I run the Windows 7 VM for an hour or so to see if everything is working fine 😊

This morning my Outlook 2010 was working offline, and it didn’t want to go online (OWA and Outlook 365 ProPlus were working fine). Remove the Outlook profile but creating a new Outlook profile didn’t work. After a minute the dreaded an encrypted connection to your mail server is not available error message appeared:

An encrypted connection to your mail server is not available

Mostly this is caused by Autodiscover that goes wrong somewhere, the Remote Connectivity Analyzer shows that Autodiscover to the on-premises Exchange 2010 goes well, but that the redirect to Exchange Online goes wrong and it generates the following error message:

An HTTP 456 Unauthorized response was received from the remote Unknown server. This indicates that the user may not have logged on for the first time, or the account may be locked. To logon, go to http://portal.microsoftonline.com.

And further down more details are revealed:

X-AutoDiscovery-Error: LiveIdBasicAuth:AppPasswordRequired:<RequestId=8a51c25b-9213-4873-aff8-ebc1da40544f>;

An HTTP 456 Unauthorized response was received from the remote Unknown server

The AppPasswordRequired explains more. Last week I changed the MFA settings (see previous authenticator app for Office 365 blogpost). This works fine for OWA and Office 365 ProPlus, but not for Outlook 2010. Since Outlook 2010 does not work with Office 365 MFA, especially not in a hybrid environment (not even with an App Password).

The only workaround here was to temporarily disable MFA for my user account, create a new Outlook profile (which worked fine without MFA) and re-enable MFA. Again, Outlook 2010 does not recognize the MFA and still works with Exchange Online using basic authentication, but all other Office 365 services work fine with Office 365 MFA (both SMS and Authenticator authentication).

Free/busy not working from Exchange Online to Exchange on-premises

Recently users started to complain that free/busy information was not available, more specifically users that had their mailbox in Exchange Online were not able to retrieve availability information from their colleagues or meeting rooms that were still in Exchange 2010 on-premises.

Complaints came from multiple users from multiple countries, there are multiple sites with multiple Exchange 2010 servers with multiple breakout points to the Internet, so the issue was consistent and not really related to only one Exchange 2010 server. And it was only happening cross-premises, and only from Exchange Online to Exchange 2010. From Exchange 2010 to Exchange Online it was working flawlessly, so was between mailboxes in Exchange 2010.

You see this happen often right after configuring an Exchange hybrid configuration with the HCW, but in my case it had been working fine for quite some time, so it had to be related to something we had changed recently. We had WLAN changes (new provider), Windows Update, Exchange 2010 rollup updates, SSL certificates, new Send and Receive Connectors, but nothing that immediately pointed in the right direction. To make things worse, the Remote Connectivity Analyzer (your first stop when troubleshooting) didn’t see any issues, everything worked well. Autodiscover returned the correct information from mailboxes in Exchange Online and Exchange 2010, and the free/busy test in RCA worked well.

Note. A lot of people don’t know this feature, but in the Remote Connectivity Analyzer you can check free/busy in a hybrid environment. Just select the Free/Busy radio button under the Office 365 tab as shown in the following screenshot:

Remote Connectivity Analyzer FreeBusy

My mailbox was in Exchange Online and I also did experience the issue, but at the same time I was able to open the calendar cross-premises. Ok, this is Outlook Anywhere and not Exchange Web Services (which is used for free/busy) but at least it ruled out a firewall issue.

When running the Get-OrganizationRelationship command, I could verify the TargetAutodiscoverEpr property, which was set to the correct Autodiscover URL. Using the TargetSharingEpr property instead of the TargetAutodiscoverEpr property didn’t help.

Get-FederationInformation command with the -DomainName switch… all looks good.
Security on the Virtual Directory? Running these commands often solve unexpected issues:

Get-AutodiscoverVirtualDirectory | Set-AutodiscoverVirtualDirectory -WSSecurity $true
Get-WebServicesVirtualDirectory -Server WPGREXC01 | Set-WebServicesVirtualDirectory -WSSecurity $True

Again, no success… Time for deeper troubleshooting. At this moment Microsoft support was already involved as well….

When testing I could see the request entering the Exchange 2010 server in the IIS logs, but the servers returned a 500 Error, so something in the request was causing issues on the Exchange server:


2019-10-14 06:45:06 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.155.157 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:33 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.140.165 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/Freebusy – 500 0 0 15


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.158.5 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

The next step to try was to recycle the AutodiscoverAppPool and the ExchangeServicesAppPool in the IIS Manager, but unfortunately this didn’t help.
After looking out of the window what might be the issue I had another look at the eventlog on the Exchange server, and found the following certificate warning:

EventID 403 Certificate is expired


Log Name: Application
Source: MSExchange Common
Date: 10/16/2019 9:28:59 AM
Event ID: 403
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: Exchange.contoso.com
Description:
The certificate named ‘791BC6AD9893AA570DF03452B4F8069C8A743C29’ in the Federation Trust ‘Microsoft Federation Gateway’ is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

This rings a bell. The certificate with the thumbprint mentioned in the error message is not on the Exchange server, but it’s in the Microsoft Federation Gateway. I didn’t see this earlier, but when checking the federation with Get-FederationTrust | FL you can see certificate information, and one certificate expired some time ago. July 2019 to be precise.

You can also run the Test-FederationTrust on the Exchange server. If you ran into this issue, you should see an error message like Failed to validate delegation token in the TokenValidation section.

Fixing this is easy, just run the following command:

Get-FederationTrust | Set-FederationTrust -RefreshMetadata

After running this command it works like a charm.

This only happens in Exchange 2010 (in Exchange 2013 and higher it is fixed automatically), and looking at the support chances are you are not running Exchange 2010 anymore when other certificates are renewed. You can running the command mentioned before on a regular basis, or you can use a scheduled task to perform this automatically on a regular basis.

Schtasks /create /sc Daily /tn FedRefresh /tr “C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
-version 2.0 -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010;
$fedTrust = Get-FederationTrust;Set-FederationTrust -Identity $fedTrust.Name -RefreshMetadata” /ru System

So why didn’t we notice this in the first place? The certificate change was announced by Microsoft and other community blogs, but this was summer holiday time. Also lack of resources in IT Staff didn’t help either. Too bad, but in the end it was fixed (with help of Microsoft support 😊)

More information

 

A hybrid deployment with Office 365 has been detected

There are lots of organizations running Exchange 2010 hybrid, but with the upcoming end-of-life of Exchange 2010 it’s time to move on. Sometimes it’s not always possible to move everything to Exchange Online, so then you must migrate from Exchange 2010 hybrid to Exchange 2016 hybrid.

When preparing Active Directory for Exchange 2016 using the /PrepareSchema swith the following error is (sometimes) raised:

hybrid deployment has been detected

A hybrid deployment with Office 365 has been detected. Please ensure that you are running setup with the /TenantOrganizationConfig switch. To use the TenantOrganizationConfig switch you must first connect to your Exchange Online tenant via PowerShell and execute the following command: “Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML”. Once the XML file has been generated, run setup with the TenantOrganizationConfig switch as follows “/TenantOrganizationConfig MyTenantOrganizationConfig.XML”.

If you continue to see this this message then it indicates that either the XML file specified is corrupt, or you are attempting to upgrade your on-premises Exchange installation to a build that isn’t compatible with the Exchange version of your Office 365 tenant. Your Office 365 tenant must be upgraded to a compatible version of Exchange before upgrading your on-premises Exchange installation. For more information, see: http://go.microsoft.com/fwlink/?LinkId=262888

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.DidTenantSettingCreatedAnException.aspx

(unfortunately the last link doesn’t show any useful information, and it hasn’t been updated since the early Exchange 2013 days).

Note. I’m not sure when and why this message pops up. Right now, it looks like it only happens in older tenants where Exchange 2010 hybrid is already running for some time. If you have more information about this, please feel free to leave a comment.

As stated in the error message you must run the following command in an Exchange Online PowerShell window:

Get-OrganizationConfig | Export-Clixml -Path C:\Install\MyTenantOrganizationConfig.XML

This retrieves the organization configuration from Exchange Online and exports it to an XML file. When opening the XML you can read the organization configuration as shown in the following screenshot:

TenantOrganizationConfig XML

There’s a pitfall when continuing, the /TenantOrganzationConfig switch that’s in the message is a switch that can only be combined with the /PrepareAD switch, it cannot be used with the /PrepareSchema switch. If you do, it will raise a “The parameter ‘tenantorganizationconfig’ is not valid for current operation ‘PrepareSchema’” Error message.

So, you must continue with the following command:

Setup.exe /PrepareAD /TenantOrganizationConfig C:\Install\TenantOrganizationConfig.XML /IAcceptExchangeServerLicenseTerms

Note. Running the /PrepareAD switch will automatically trigger the/PrepareSchema switch.

Now you can continue with the /PrepareDomain switch, followed by the installation of your first Exchange 2016 server in the existing Exchange 2010 environment.

Webinar: Top 5 Exchange hybrid considerations

This Thursday (May 16th) I’ll be doing a webinar on the Top 5 Exchange Hybrid Considerations with Jeff Guillet, MVP and MCM and well known for this ExPTA blogs.

The webinar will be hosted by Nicole Silva from Enow Software and will take approx 35 minutes, there is Q&A at the end and also the possibility to ask questions using a chat window during the call.

Topics are:

  • Identities.
  • Synchronization.
  • Authentication.
  • En two more 🙂

There are still a few seats left, you can register on the Enow website: https://enow.software/2WbwIQJ

Exchange Hybrid TLS negotiation failed with error NoCredentials

Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.

When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.

But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:

LastError : [{LED=451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out};{MSG=};{FQDN=exchangelabsnl-mail-onmicrosoft-com.mail.protection.outlook.com}; {IP=104.47.10.36};{LRT=5/2/2019 6:32:14 AM}]

421 4.4.1 connection timed out

It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).

Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.

TLS Negotiation failed with error NoCredentials

This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).

The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:

Has Private Key

When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.

Certificate Thumbprint

But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.

When opening the certificate store using PowerShell using the following commands:

CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem

All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:

Check private key

That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:

Get-ChildItem | ?{$_.Thumbprint -like “B79*”} | Remove-Item

After restarting the Transport service cross-premises mail flow works again.

The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊