Category Archives: Exchange Hybrid

Exchange Hybrid

Free/busy not working from Exchange Online to Exchange on-premises

Leave a comment

Recently users started to complain that free/busy information was not available, more specifically users that had their mailbox in Exchange Online were not able to retrieve availability information from their colleagues or meeting rooms that were still in Exchange 2010 on-premises.

Complaints came from multiple users from multiple countries, there are multiple sites with multiple Exchange 2010 servers with multiple breakout points to the Internet, so the issue was consistent and not really related to only one Exchange 2010 server. And it was only happening cross-premises, and only from Exchange Online to Exchange 2010. From Exchange 2010 to Exchange Online it was working flawlessly, so was between mailboxes in Exchange 2010.

You see this happen often right after configuring an Exchange hybrid configuration with the HCW, but in my case it had been working fine for quite some time, so it had to be related to something we had changed recently. We had WLAN changes (new provider), Windows Update, Exchange 2010 rollup updates, SSL certificates, new Send and Receive Connectors, but nothing that immediately pointed in the right direction. To make things worse, the Remote Connectivity Analyzer (your first stop when troubleshooting) didn’t see any issues, everything worked well. Autodiscover returned the correct information from mailboxes in Exchange Online and Exchange 2010, and the free/busy test in RCA worked well.

Note. A lot of people don’t know this feature, but in the Remote Connectivity Analyzer you can check free/busy in a hybrid environment. Just select the Free/Busy radio button under the Office 365 tab as shown in the following screenshot:

Remote Connectivity Analyzer FreeBusy

My mailbox was in Exchange Online and I also did experience the issue, but at the same time I was able to open the calendar cross-premises. Ok, this is Outlook Anywhere and not Exchange Web Services (which is used for free/busy) but at least it ruled out a firewall issue.

When running the Get-OrganizationRelationship command, I could verify the TargetAutodiscoverEpr property, which was set to the correct Autodiscover URL. Using the TargetSharingEpr property instead of the TargetAutodiscoverEpr property didn’t help.

Get-FederationInformation command with the -DomainName switch… all looks good.
Security on the Virtual Directory? Running these commands often solve unexpected issues:

Get-AutodiscoverVirtualDirectory | Set-AutodiscoverVirtualDirectory -WSSecurity $true
Get-WebServicesVirtualDirectory -Server WPGREXC01 | Set-WebServicesVirtualDirectory -WSSecurity $True

Again, no success… Time for deeper troubleshooting. At this moment Microsoft support was already involved as well….

When testing I could see the request entering the Exchange 2010 server in the IIS logs, but the servers returned a 500 Error, so something in the request was causing issues on the Exchange server:


2019-10-14 06:45:06 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.155.157 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

2019-10-14 06:45:33 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.140.165 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/Freebusy – 500 0 0 15

2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.158.5 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

The next step to try was to recycle the AutodiscoverAppPool and the ExchangeServicesAppPool in the IIS Manager, but unfortunately this didn’t help.
After looking out of the window what might be the issue I had another look at the eventlog on the Exchange server, and found the following certificate warning:

EventID 403 Certificate is expired


Log Name: Application
Source: MSExchange Common
Date: 10/16/2019 9:28:59 AM
Event ID: 403
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: Exchange.contoso.com
Description:
The certificate named ‘791BC6AD9893AA570DF03452B4F8069C8A743C29’ in the Federation Trust ‘Microsoft Federation Gateway’ is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

This rings a bell. The certificate with the thumbprint mentioned in the error message is not on the Exchange server, but it’s in the Microsoft Federation Gateway. I didn’t see this earlier, but when checking the federation with Get-FederationTrust | FL you can see certificate information, and one certificate expired some time ago. July 2019 to be precise.

You can also run the Test-FederationTrust on the Exchange server. If you ran into this issue, you should see an error message like Failed to validate delegation token in the TokenValidation section.

Fixing this is easy, just run the following command:

Get-FederationTrust | Set-FederationTrust -RefreshMetadata

After running this command it works like a charm.

This only happens in Exchange 2010 (in Exchange 2013 and higher it is fixed automatically), and looking at the support chances are you are not running Exchange 2010 anymore when other certificates are renewed. You can running the command mentioned before on a regular basis, or you can use a scheduled task to perform this automatically on a regular basis.

Schtasks /create /sc Daily /tn FedRefresh /tr “C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
-version 2.0 -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010;
$fedTrust = Get-FederationTrust;Set-FederationTrust -Identity $fedTrust.Name -RefreshMetadata” /ru System

So why didn’t we notice this in the first place? The certificate change was announced by Microsoft and other community blogs, but this was summer holiday time. Also lack of resources in IT Staff didn’t help either. Too bad, but in the end it was fixed (with help of Microsoft support 😊)

More information

 

Exchange Hybrid, Uncategorized

A hybrid deployment with Office 365 has been detected

1 Comment

There are lots of organizations running Exchange 2010 hybrid, but with the upcoming end-of-life of Exchange 2010 it’s time to move on. Sometimes it’s not always possible to move everything to Exchange Online, so then you must migrate from Exchange 2010 hybrid to Exchange 2016 hybrid.

When preparing Active Directory for Exchange 2016 using the /PrepareSchema swith the following error is (sometimes) raised:

hybrid deployment has been detected

A hybrid deployment with Office 365 has been detected. Please ensure that you are running setup with the /TenantOrganizationConfig switch. To use the TenantOrganizationConfig switch you must first connect to your Exchange Online tenant via PowerShell and execute the following command: “Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML”. Once the XML file has been generated, run setup with the TenantOrganizationConfig switch as follows “/TenantOrganizationConfig MyTenantOrganizationConfig.XML”.

If you continue to see this this message then it indicates that either the XML file specified is corrupt, or you are attempting to upgrade your on-premises Exchange installation to a build that isn’t compatible with the Exchange version of your Office 365 tenant. Your Office 365 tenant must be upgraded to a compatible version of Exchange before upgrading your on-premises Exchange installation. For more information, see: http://go.microsoft.com/fwlink/?LinkId=262888

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.DidTenantSettingCreatedAnException.aspx

(unfortunately the last link doesn’t show any useful information, and it hasn’t been updated since the early Exchange 2013 days).

Note. I’m not sure when and why this message pops up. Right now, it looks like it only happens in older tenants where Exchange 2010 hybrid is already running for some time. If you have more information about this, please feel free to leave a comment.

As stated in the error message you must run the following command in an Exchange Online PowerShell window:

Get-OrganizationConfig | Export-Clixml -Path C:\Install\MyTenantOrganizationConfig.XML

This retrieves the organization configuration from Exchange Online and exports it to an XML file. When opening the XML you can read the organization configuration as shown in the following screenshot:

TenantOrganizationConfig XML

There’s a pitfall when continuing, the /TenantOrganzationConfig switch that’s in the message is a switch that can only be combined with the /PrepareAD switch, it cannot be used with the /PrepareSchema switch. If you do, it will raise a “The parameter ‘tenantorganizationconfig’ is not valid for current operation ‘PrepareSchema’” Error message.

So, you must continue with the following command:

Setup.exe /PrepareAD /TenantOrganizationConfig C:\Install\TenantOrganizationConfig.XML /IAcceptExchangeServerLicenseTerms

Note. Running the /PrepareAD switch will automatically trigger the/PrepareSchema switch.

Now you can continue with the /PrepareDomain switch, followed by the installation of your first Exchange 2016 server in the existing Exchange 2010 environment.

Exchange Hybrid

Webinar: Top 5 Exchange hybrid considerations

1 Comment

This Thursday (May 16th) I’ll be doing a webinar on the Top 5 Exchange Hybrid Considerations with Jeff Guillet, MVP and MCM and well known for this ExPTA blogs.

The webinar will be hosted by Nicole Silva from Enow Software and will take approx 35 minutes, there is Q&A at the end and also the possibility to ask questions using a chat window during the call.

Topics are:

  • Identities.
  • Synchronization.
  • Authentication.
  • En two more 🙂

There are still a few seats left, you can register on the Enow website: https://enow.software/2WbwIQJ

Exchange Hybrid

Exchange Hybrid TLS negotiation failed with error NoCredentials

Leave a comment

Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.

When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.

But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:

LastError : [{LED=451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out};{MSG=};{FQDN=exchangelabsnl-mail-onmicrosoft-com.mail.protection.outlook.com}; {IP=104.47.10.36};{LRT=5/2/2019 6:32:14 AM}]

421 4.4.1 connection timed out

It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).

Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.

TLS Negotiation failed with error NoCredentials

This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).

The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:

Has Private Key

When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.

Certificate Thumbprint

But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.

When opening the certificate store using PowerShell using the following commands:

CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem

All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:

Check private key

That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:

Get-ChildItem | ?{$_.Thumbprint -like “B79*”} | Remove-Item

After restarting the Transport service cross-premises mail flow works again.

The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊

 

Exchange Hybrid, Uncategorized

Moving from Exchange 2010 to Office 365 Part III

7 Comments

Exchange 2010 hybrid and Edge Transport Server

In two previous blog posts I explained how to setup an Exchange 2010 hybrid environment. In these blog posts I used the Exchange 2010 (multi-role) server for the hybrid configuration, so both the Exchange Web Services (used for free/busy, Mailbox Replication Service, OOF, mail tips) and the SMTP connection between Exchange Online and Exchange 2010.

Now that Exchange 2010 end of support is getting closer (less than a year!) I get more questions regarding the move from Exchange 2010 to Exchange Online. And several questions include the use of an Exchange 2010 Edge Transport server in front of the Exchange 2010 multi-role server.

This configuration will look something like this:

exchange 2010 hybrid edge transport

Inbound mail from Internet is getting through Exchange Online Protection and when the mailbox is still on Exchange 2010 it is routed via the Edge Transport Server to the internal Exchange organization. Outbound mail is leaving the organization via the Edge Transport server or via Exchange Online Protection, depending of the location of the mailbox.

The challenge when configuring this in Exchange 2010 is shown in the following screenshot:

missing edge transport server

Compared to running the HCW on Exchange 2013 or Exchange 2016 there’s no option to configure the Edge Transport server for secure mail transport in Exchange 2010!!
The only option right now is to run the Hybrid Configuration Wizard, configure it using the Client Access and Mailbox servers option, but use the data for the Edge Transport where needed.

So, run the Hybrid Configuration Wizard, and when you need to enter the public IP address of the transport servers, enter the Public IP address of the Edge Transport server and not the public IP address of the load balancer VIP pointing to the Exchange 2010 internal servers). In my environment, the webmail.inframan.nl points to 176.62.196.253 and the Edge Transport Server smtphost.inframan.nl points to 176.62.196.245. This is the IP address I am going to use as shown in the following screenshot:

hcw public ip address

The next step is to select the certificate that’s used on the Edge Transport Server. By default, the HCW will only look at the internal Exchange 2010 server, so it won’t find any certificate installed on the Edge Transport server. To overcome this, I have imported the certificate of the Edge Transport Server in the certificate store of the internal Exchange 2010 server used by the HCW. In the HWC, click on the drop down box and select the certificate of the Edge Transport server, in my environment the smtphost.inframan.nl certificate:

hcw loading certificates

The last step is where the FQDN of the organization needs to be entered. I have a lot of discussion here because most admins want to enter something like ‘hybrid.domain.com’, but the FQDN of the transport server needs to be entered here, so in my environment this is the FQDN of the Edge Transport Server, i.e. smtphost.inframan.nl. This FQDN is used (together with the certificate information) to create a Send Connector from Exchange Online to the Edge Transport server.

hcw organization fqdn

Finish the Hybrid Configuration Wizard. It will be configured in Exchange 2010 and in Exchange Online and after a short time you can close the HCW:

hcw configure organization relationship

Now, when looking at the Exchange Management Console you can see the Send Connector from Exchange 2010 to Exchange Online. It is configured with the FQDN smtphost.inframan.nl as expected, but the source server of the Send Connector is still the internal Hub Transport server as shown in the following screen shot:

outbound to office 365 connector

Remove the Hub Transport server entry and add the Edge Transport server instead. If you have an Edge Synchronization in place you will see it immediately when you click the Add button.

In Exchange Online, the Receive Connector that’s created will check for any certificate with a wildcard like name, so the smtphost.inframan.nl certificate will automatically be accepted. The Send Connector is also created correctly. The FQDN of the Edge Transport server is used as the server to route message to, and the CN of the certificate that was selected in the HCW is also configured as shown in the following screenshot:

office 365 send connector certificate

We’re almost there. The only thing that needs to be done is to configure the Receive Connector on the Edge Transport Server for TLS from Exchange Online. You should have already configured the Edge Transport Server with the correct 3rd party certificate and when setting up an inbound connection it should use the 3rd party certificate. You can test this using the https://checktls.com tool online.

On the Edge Transport server, execute the following command in the Exchange Management Shell:

Get-ReceiveConnector "smtphost\Default internal receive connector SMTPHOST" | Set-ReceiveConnector -Fqdn smtphost.inframan.nl -TlsDomainCapabilities mail.protection.outlook.com:AcceptOorgProtocol

This will make sure the cross-premises email will be treated as internal email (SCL=-1). If you omit this step, there’s always the risk the email will be treated as external (I’ve seen SCL=5 in my environment) and will end up in the user’s Junk Email Folder.

Summary

When configuring an Exchange 2010 hybrid configuration it is not possible to configure an Edge Transport Server in the Hybrid Configuration Wizard. It is possible to configure this in the HCW for Exchange 2013 and Exchange 2016, but for Exchange 2010 this needs some manual changes.

In this blogpost I showed you the steps needed to configure an Edge Transport Server for secure messaging between Exchange Online and Exchange 2010. When configured this way cross-premises email will be seen as internal email and thus treated accordingly.