Tag Archives: hybrid

Free/busy not working from Exchange Online to Exchange on-premises

Recently users started to complain that free/busy information was not available, more specifically users that had their mailbox in Exchange Online were not able to retrieve availability information from their colleagues or meeting rooms that were still in Exchange 2010 on-premises.

Complaints came from multiple users from multiple countries, there are multiple sites with multiple Exchange 2010 servers with multiple breakout points to the Internet, so the issue was consistent and not really related to only one Exchange 2010 server. And it was only happening cross-premises, and only from Exchange Online to Exchange 2010. From Exchange 2010 to Exchange Online it was working flawlessly, so was between mailboxes in Exchange 2010.

You see this happen often right after configuring an Exchange hybrid configuration with the HCW, but in my case it had been working fine for quite some time, so it had to be related to something we had changed recently. We had WLAN changes (new provider), Windows Update, Exchange 2010 rollup updates, SSL certificates, new Send and Receive Connectors, but nothing that immediately pointed in the right direction. To make things worse, the Remote Connectivity Analyzer (your first stop when troubleshooting) didn’t see any issues, everything worked well. Autodiscover returned the correct information from mailboxes in Exchange Online and Exchange 2010, and the free/busy test in RCA worked well.

Note. A lot of people don’t know this feature, but in the Remote Connectivity Analyzer you can check free/busy in a hybrid environment. Just select the Free/Busy radio button under the Office 365 tab as shown in the following screenshot:

Remote Connectivity Analyzer FreeBusy

My mailbox was in Exchange Online and I also did experience the issue, but at the same time I was able to open the calendar cross-premises. Ok, this is Outlook Anywhere and not Exchange Web Services (which is used for free/busy) but at least it ruled out a firewall issue.

When running the Get-OrganizationRelationship command, I could verify the TargetAutodiscoverEpr property, which was set to the correct Autodiscover URL. Using the TargetSharingEpr property instead of the TargetAutodiscoverEpr property didn’t help.

Get-FederationInformation command with the -DomainName switch… all looks good.
Security on the Virtual Directory? Running these commands often solve unexpected issues:

Get-AutodiscoverVirtualDirectory | Set-AutodiscoverVirtualDirectory -WSSecurity $true
Get-WebServicesVirtualDirectory -Server WPGREXC01 | Set-WebServicesVirtualDirectory -WSSecurity $True

Again, no success… Time for deeper troubleshooting. At this moment Microsoft support was already involved as well….

When testing I could see the request entering the Exchange 2010 server in the IIS logs, but the servers returned a 500 Error, so something in the request was causing issues on the Exchange server:


2019-10-14 06:45:06 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.155.157 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:33 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.140.165 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/Freebusy – 500 0 0 15


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.158.5 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

The next step to try was to recycle the AutodiscoverAppPool and the ExchangeServicesAppPool in the IIS Manager, but unfortunately this didn’t help.
After looking out of the window what might be the issue I had another look at the eventlog on the Exchange server, and found the following certificate warning:

EventID 403 Certificate is expired


Log Name: Application
Source: MSExchange Common
Date: 10/16/2019 9:28:59 AM
Event ID: 403
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: Exchange.contoso.com
Description:
The certificate named ‘791BC6AD9893AA570DF03452B4F8069C8A743C29’ in the Federation Trust ‘Microsoft Federation Gateway’ is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

This rings a bell. The certificate with the thumbprint mentioned in the error message is not on the Exchange server, but it’s in the Microsoft Federation Gateway. I didn’t see this earlier, but when checking the federation with Get-FederationTrust | FL you can see certificate information, and one certificate expired some time ago. July 2019 to be precise.

You can also run the Test-FederationTrust on the Exchange server. If you ran into this issue, you should see an error message like Failed to validate delegation token in the TokenValidation section.

Fixing this is easy, just run the following command:

Get-FederationTrust | Set-FederationTrust -RefreshMetadata

After running this command it works like a charm.

This only happens in Exchange 2010 (in Exchange 2013 and higher it is fixed automatically), and looking at the support chances are you are not running Exchange 2010 anymore when other certificates are renewed. You can running the command mentioned before on a regular basis, or you can use a scheduled task to perform this automatically on a regular basis.

Schtasks /create /sc Daily /tn FedRefresh /tr “C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
-version 2.0 -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010;
$fedTrust = Get-FederationTrust;Set-FederationTrust -Identity $fedTrust.Name -RefreshMetadata” /ru System

So why didn’t we notice this in the first place? The certificate change was announced by Microsoft and other community blogs, but this was summer holiday time. Also lack of resources in IT Staff didn’t help either. Too bad, but in the end it was fixed (with help of Microsoft support 😊)

More information

 

Exchange 2010 and TLS 1.2

In a previous blogpost I discussed an issue I had with Outlook 2010 and TLS 1.2. At the same time this reminded me that Microsoft will remove support for TLS 1.0 and TLS 1.1 in Office 365 on October 31, 2018 as communicated in https://support.microsoft.com/en-us/help/4057306/preparing-for-tls-1-2-in-office-365. This means that when you have communication issues with Office 365 because of an older and weaker protocol, you won’t get any support. Time to do some research….

Existing Exchange 2010 environment

As you may have seen on this side, I still am a big fan of Exchange 2010 and also have an pure Exchange 2010 hybrid environment up-and-running and it looks like this:

Inframan-hybrid

MX records is pointing to my Exchange 2010 Edge Transport Server (running on Windows 2008 R2), webmail and Autodiscover are routed via an F5 LTM load balancer to an Exchange 2010 CAS/HUB/Mailbox server (also running on Windows 2008 R2), and hybrid is configured directly on Exchange 2010 (for hybrid mail flow I’m using a separate FQDN, o365mail.inframan.nl) without any Exchange 2013 or Exchange 2016 server.

So, how do you test which TLS version is used by your Exchange 2010 server? In Exchange 2010 this should be done using the protocol logfiles. Message headers in Exchange 2010 do not contain enough information for showing this TLS information. So, you must enable protocol logging for the appropriate Receive Connectors and Send Connectors. In my environment this means the Default Receive Connector on the Exchange 2010 Edge Transport server (for O365 traffic from other tenants), the Default-First-Site-Name to Internet Send Connector, and both connectors between the Exchange 2010 server and Office 365 for hybrid. Analyzing the protocol logfiles can best be done in Excel (import as CSV files). When analyzing, look for a string like TLS protocol SP_PROT_TLS1_0_SERVER (when receiving) or TLS protocol SP_PROT-TLS1_0_CLIENT (when sending). When TLS 1.2 is used, look for a string like TLS protocol SP_PROT_TLS1_2_SERVER and TLS protocol SP_PROT-TLS1_2_CLIENT.

Continue reading Exchange 2010 and TLS 1.2

Cannot find a recipient that has mailbox GUID when moving from Exchange Online to Exchange 2016

When moving mailboxes from Exchange Online to Exchange 2016 on-premises in a hybrid environment, the move fails with an error “Cannot find a recipient that has mailbox GUID ‘ ‘

image

The error is listed here for Search Engine purposes:

Error: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ‎’add02766-9698-48e6-9234-91c3077137bc’. –> Cannot find a recipient that has mailbox GUID ‎ add02766-9698-48e6-9234-91c3077137bc ‎’.
Report: bramwess@exchangelabs.nl

When checking the user account with ADSI Edit in the on-premises Active Directory it is obvious that this property is empty:

image

When checking the Mailbox in Exchange Online (using Remote PowerShell) the Exchange GUID is visible when using the command

Get-Mailbox -Identity name@exchangelabs.nl | Select Name,*GUID*

As shown in the following screenshot:

image

It took me some time to figure out why this property was empty. Normally when moving mailboxes from Exchange on-premises to Exchange Online the Mailbox GUID is retained. Keeping the Mailbox GUID makes sure you don’t have to download the .OST file again after moving to Exchange Online.

What happened here is that the user was created in Active Directory on-premises, and a Mailbox was directly created in Exchange Online using the Enable-RemoteMailbox command. In this scenario, there never was a Mailbox on-premises and thus never a Mailbox GUID.

The solution is to copy and paste the Mailbox Guid as found in the previous command into the Remote Mailbox object on-premises using the Set-RemoteMailbox command

Set-RemoteMailbox user@exchangelabs.nl -ExchangeGuid “copied value”

As shown in the following screenshot:

image

When setting the Mailbox Guid the mailbox can be moved from Exchange Online to Exchange on-premises.

Ps. Don’t forget to repeat this for anchive mailbox (if one exists)

Exchange 2010 hybrid, SMTP, SSL Certificates and Subject Alternative Names

On every Exchange server you need SSL certificates for authentication, validation and encryption purposes. For SMTP you can use the self-signed certificate. Exchange 2010 uses opportunistic TLS, so the self-signed certificate will do in this scenario. If you need to configure domain security (mutual TLS) on Exchange, you need a proper 3rd party SSL certificate for this.

SMTP communication between Office 365 and Exchange in a hybrid scenario is an example of mutual TLS or domain security. A proper 3rd party SSL certificate is needed on your Exchange server.

I was always under the impression that mutual TLS can only use the Common Name of the certificate, which in my scenario is CN=webmail.inframan.nl. After a previous blogpost there was an interesting discussion (see the comments of this particular blogpost) about this, so now it’s time to do some testing.

Originally I had a Digicert SSL certificate with Common Name CN=webmail.inframan.nl, and a Subject Alternative Name entry autodiscover.webmail.com. During the HCW I entered webmail.inframan.nl and selected the proper certificate.

It was time to renew my SSL certificate, so I added an additional SAN entry o365mail.inframan.nl.

image

Continue reading Exchange 2010 hybrid, SMTP, SSL Certificates and Subject Alternative Names

Exchange 2010 Hybrid cannot establish Mutual TLS wrong certificate is used

When configuring an Exchange 2010 hybrid environment a Receive Connector is created on the Exchange 2010 server. This Receive Connector is configured with the FQDN entered in the Hybrid Configuration Wizard (see previous blog post on Exchange 2010 Hybrid) and the source IP addresses of the Microsoft Exchange Online servers. If one of these servers access the Exchange 2010 environment, they end up on the Office 365 Receive Connector (based on the IP address) and the correct SSL certificate is returned. This way mutual TLS is established between Exchange 2010 on-premises and Exchange Online.

It sometimes happens that the wrong certificate is used for SMTP communication between Exchange on-premises and Exchange Online, thus resulting in SMTP mail flow failure between the two.

You can check this in the Exchange Admin Center (EAC) in Exchange Online. Logon to the EAC in Exchange Online, select Mail Flow and click the Connectors tab. You’ll see two connectors. One connector for mail from Exchange 2010 to Exchange Online, and one connector for mail from Exchange Online to Exchange 2010.

image

Continue reading Exchange 2010 Hybrid cannot establish Mutual TLS wrong certificate is used