Tag Archives: certificates

Free/busy not working from Exchange Online to Exchange on-premises

Recently users started to complain that free/busy information was not available, more specifically users that had their mailbox in Exchange Online were not able to retrieve availability information from their colleagues or meeting rooms that were still in Exchange 2010 on-premises.

Complaints came from multiple users from multiple countries, there are multiple sites with multiple Exchange 2010 servers with multiple breakout points to the Internet, so the issue was consistent and not really related to only one Exchange 2010 server. And it was only happening cross-premises, and only from Exchange Online to Exchange 2010. From Exchange 2010 to Exchange Online it was working flawlessly, so was between mailboxes in Exchange 2010.

You see this happen often right after configuring an Exchange hybrid configuration with the HCW, but in my case it had been working fine for quite some time, so it had to be related to something we had changed recently. We had WLAN changes (new provider), Windows Update, Exchange 2010 rollup updates, SSL certificates, new Send and Receive Connectors, but nothing that immediately pointed in the right direction. To make things worse, the Remote Connectivity Analyzer (your first stop when troubleshooting) didn’t see any issues, everything worked well. Autodiscover returned the correct information from mailboxes in Exchange Online and Exchange 2010, and the free/busy test in RCA worked well.

Note. A lot of people don’t know this feature, but in the Remote Connectivity Analyzer you can check free/busy in a hybrid environment. Just select the Free/Busy radio button under the Office 365 tab as shown in the following screenshot:

Remote Connectivity Analyzer FreeBusy

My mailbox was in Exchange Online and I also did experience the issue, but at the same time I was able to open the calendar cross-premises. Ok, this is Outlook Anywhere and not Exchange Web Services (which is used for free/busy) but at least it ruled out a firewall issue.

When running the Get-OrganizationRelationship command, I could verify the TargetAutodiscoverEpr property, which was set to the correct Autodiscover URL. Using the TargetSharingEpr property instead of the TargetAutodiscoverEpr property didn’t help.

Get-FederationInformation command with the -DomainName switch… all looks good.
Security on the Virtual Directory? Running these commands often solve unexpected issues:

Get-AutodiscoverVirtualDirectory | Set-AutodiscoverVirtualDirectory -WSSecurity $true
Get-WebServicesVirtualDirectory -Server WPGREXC01 | Set-WebServicesVirtualDirectory -WSSecurity $True

Again, no success… Time for deeper troubleshooting. At this moment Microsoft support was already involved as well….

When testing I could see the request entering the Exchange 2010 server in the IIS logs, but the servers returned a 500 Error, so something in the request was causing issues on the Exchange server:


2019-10-14 06:45:06 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.155.157 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:33 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.140.165 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/Freebusy – 500 0 0 15


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.158.5 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

The next step to try was to recycle the AutodiscoverAppPool and the ExchangeServicesAppPool in the IIS Manager, but unfortunately this didn’t help.
After looking out of the window what might be the issue I had another look at the eventlog on the Exchange server, and found the following certificate warning:

EventID 403 Certificate is expired


Log Name: Application
Source: MSExchange Common
Date: 10/16/2019 9:28:59 AM
Event ID: 403
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: Exchange.contoso.com
Description:
The certificate named ‘791BC6AD9893AA570DF03452B4F8069C8A743C29’ in the Federation Trust ‘Microsoft Federation Gateway’ is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

This rings a bell. The certificate with the thumbprint mentioned in the error message is not on the Exchange server, but it’s in the Microsoft Federation Gateway. I didn’t see this earlier, but when checking the federation with Get-FederationTrust | FL you can see certificate information, and one certificate expired some time ago. July 2019 to be precise.

You can also run the Test-FederationTrust on the Exchange server. If you ran into this issue, you should see an error message like Failed to validate delegation token in the TokenValidation section.

Fixing this is easy, just run the following command:

Get-FederationTrust | Set-FederationTrust -RefreshMetadata

After running this command it works like a charm.

This only happens in Exchange 2010 (in Exchange 2013 and higher it is fixed automatically), and looking at the support chances are you are not running Exchange 2010 anymore when other certificates are renewed. You can running the command mentioned before on a regular basis, or you can use a scheduled task to perform this automatically on a regular basis.

Schtasks /create /sc Daily /tn FedRefresh /tr “C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
-version 2.0 -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010;
$fedTrust = Get-FederationTrust;Set-FederationTrust -Identity $fedTrust.Name -RefreshMetadata” /ru System

So why didn’t we notice this in the first place? The certificate change was announced by Microsoft and other community blogs, but this was summer holiday time. Also lack of resources in IT Staff didn’t help either. Too bad, but in the end it was fixed (with help of Microsoft support 😊)

More information

 

Exchange Hybrid TLS negotiation failed with error NoCredentials

Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.

When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.

But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:

LastError : [{LED=451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out};{MSG=};{FQDN=exchangelabsnl-mail-onmicrosoft-com.mail.protection.outlook.com}; {IP=104.47.10.36};{LRT=5/2/2019 6:32:14 AM}]

421 4.4.1 connection timed out

It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).

Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.

TLS Negotiation failed with error NoCredentials

This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).

The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:

Has Private Key

When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.

Certificate Thumbprint

But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.

When opening the certificate store using PowerShell using the following commands:

CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem

All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:

Check private key

That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:

Get-ChildItem | ?{$_.Thumbprint -like “B79*”} | Remove-Item

After restarting the Transport service cross-premises mail flow works again.

The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊

 

Exchange 2010 hybrid, SMTP, SSL Certificates and Subject Alternative Names

On every Exchange server you need SSL certificates for authentication, validation and encryption purposes. For SMTP you can use the self-signed certificate. Exchange 2010 uses opportunistic TLS, so the self-signed certificate will do in this scenario. If you need to configure domain security (mutual TLS) on Exchange, you need a proper 3rd party SSL certificate for this.

SMTP communication between Office 365 and Exchange in a hybrid scenario is an example of mutual TLS or domain security. A proper 3rd party SSL certificate is needed on your Exchange server.

I was always under the impression that mutual TLS can only use the Common Name of the certificate, which in my scenario is CN=webmail.inframan.nl. After a previous blogpost there was an interesting discussion (see the comments of this particular blogpost) about this, so now it’s time to do some testing.

Originally I had a Digicert SSL certificate with Common Name CN=webmail.inframan.nl, and a Subject Alternative Name entry autodiscover.webmail.com. During the HCW I entered webmail.inframan.nl and selected the proper certificate.

It was time to renew my SSL certificate, so I added an additional SAN entry o365mail.inframan.nl.

image

Continue reading Exchange 2010 hybrid, SMTP, SSL Certificates and Subject Alternative Names