Category Archives: Exchange Hybrid

Webinar: Top 5 Exchange hybrid considerations

May 15, 2019 jaapwesselius 1 Comment

This Thursday (May 16th) I’ll be doing a webinar on the Top 5 Exchange Hybrid Considerations with Jeff Guillet, MVP and MCM and well known for this ExPTA blogs.

The webinar will be hosted by Nicole Silva from Enow Software and will take approx 35 minutes, there is Q&A at the end and also the possibility to ask questions using a chat window during the call.

Topics are:

Identities.
Synchronization.
Authentication.
En two more 🙂

There are still a few seats left, you can register on the Enow website: https://enow.software/2WbwIQJ

Exchange Hybrid

Exchange Hybrid TLS negotiation failed with error NoCredentials

May 7, 2019 jaapwesselius Leave a comment

Recently I ran the Hybrid Configuration Wizard in an Exchange 2016 and Exchange 2019 environment. There were also two Edge Transport servers in this environment. One Exchange 2016 CU12 Edge Transport server is used for internet communication, one Exchange 2019 CU1 Edge Transport server (running on Windows 2019 Server Core) is used for hybrid communication. This server was selected in the Hybrid Configuration Wizard, proper certificate was selected etc. and the Hybrid Configuration Wizard finished successfully.

When the wizard finished the Receive Connector on the Edge Transport server was modified for hybrid mail flow. Validating the Send Connector from Exchange Online to Exchange on-premises revealed no issues, the test message was successfully sent and received in my mailbox.

But message flow from Exchange on-premises to Exchange Online was not working and mail was stuck in the Queue on the Edge Transport server. Looking at the Queue it seems there’s a time-out issue since it says:

LastError : [{LED=451 4.4.395 Target host responded with error. -> 421 4.4.1 Connection timed out};{MSG=};{FQDN=exchangelabsnl-mail-onmicrosoft-com.mail.protection.outlook.com}; {IP=104.47.10.36};{LRT=5/2/2019 6:32:14 AM}]

421 4.4.1 connection timed out

It is not a firewall issue, I can use Telnet to connect on port 25 and send a message to myself (which arrives in the junk mail folder, but it arrives).

Opening the Send Connector protocollog file (enable in on the outbound connector first) shows a different error. When trying to execute the TLS handshake it fails with TLS negotiation failed with error NoCredentials.

TLS Negotiation failed with error NoCredentials

This is strange since the same certificate is used by the Receive Connector (you can check this using https://checktls.com and entering the FQDN of the Exchange server holding the Receive Connector).

The “TLS negotiation failed with error NoCredentials” looks like a private key issue with the certificate (according to Microsoft kb article KB4495258) but PowerShell shows it does have a private key:

Has Private Key

When going back to the protocol logfile you can see the certificate thumbprint in the data field, and this thumbprint didn’t match the thumbprint of the certificate that Get-ExchangeCertificate returned.

Certificate Thumbprint

But, Get-ExchangeCertificate only returns certificates that have a private key, if there isn’t a private key nothing is returned.

When opening the certificate store using PowerShell using the following commands:

CD Cert:
Cd LocalMachine
Set-Location my
Get-ChildItem

All certificates in the store are shown, and when checking the certificate with the thumbprint we got from the protocol log, this one does not have a private key:

Check private key

That explains the NoCredentials error messages. Use the following command to remove the wrong certificate:

Get-ChildItem | ?{$_.Thumbprint -like “B79*”} | Remove-Item

After restarting the Transport service cross-premises mail flow works again.

The main question is of course how this happened. I’m not sure, but I do remember requesting several certificates at the same time (a few weeks ago) and there were a few errors. I didn’t pay too much attention to this since everything seemed to work fine. But in the end it turned out to be not the case, and I didn’t notice in the first place because of inbound SMTP working fine. Sigh…. 😊

Uncategorized, Exchange Hybrid

Moving from Exchange 2010 to Office 365 Part III

January 16, 2019 jaapwesselius 5 Comments

Exchange 2010 hybrid and Edge Transport Server

In two previous blog posts I explained how to setup an Exchange 2010 hybrid environment. In these blog posts I used the Exchange 2010 (multi-role) server for the hybrid configuration, so both the Exchange Web Services (used for free/busy, Mailbox Replication Service, OOF, mail tips) and the SMTP connection between Exchange Online and Exchange 2010.

Now that Exchange 2010 end of support is getting closer (less than a year!) I get more questions regarding the move from Exchange 2010 to Exchange Online. And several questions include the use of an Exchange 2010 Edge Transport server in front of the Exchange 2010 multi-role server.

This configuration will look something like this:

exchange 2010 hybrid edge transport

Inbound mail from Internet is getting through Exchange Online Protection and when the mailbox is still on Exchange 2010 it is routed via the Edge Transport Server to the internal Exchange organization. Outbound mail is leaving the organization via the Edge Transport server or via Exchange Online Protection, depending of the location of the mailbox.

The challenge when configuring this in Exchange 2010 is shown in the following screenshot:

missing edge transport server

Compared to running the HCW on Exchange 2013 or Exchange 2016 there’s no option to configure the Edge Transport server for secure mail transport in Exchange 2010!!
The only option right now is to run the Hybrid Configuration Wizard, configure it using the Client Access and Mailbox servers option, but use the data for the Edge Transport where needed.

So, run the Hybrid Configuration Wizard, and when you need to enter the public IP address of the transport servers, enter the Public IP address of the Edge Transport server and not the public IP address of the load balancer VIP pointing to the Exchange 2010 internal servers). In my environment, the webmail.inframan.nl points to 176.62.196.253 and the Edge Transport Server smtphost.inframan.nl points to 176.62.196.245. This is the IP address I am going to use as shown in the following screenshot:

hcw public ip address

The next step is to select the certificate that’s used on the Edge Transport Server. By default, the HCW will only look at the internal Exchange 2010 server, so it won’t find any certificate installed on the Edge Transport server. To overcome this, I have imported the certificate of the Edge Transport Server in the certificate store of the internal Exchange 2010 server used by the HCW. In the HWC, click on the drop down box and select the certificate of the Edge Transport server, in my environment the smtphost.inframan.nl certificate:

The last step is where the FQDN of the organization needs to be entered. I have a lot of discussion here because most admins want to enter something like ‘hybrid.domain.com’, but the FQDN of the transport server needs to be entered here, so in my environment this is the FQDN of the Edge Transport Server, i.e. smtphost.inframan.nl. This FQDN is used (together with the certificate information) to create a Send Connector from Exchange Online to the Edge Transport server.

hcw organization fqdn

Finish the Hybrid Configuration Wizard. It will be configured in Exchange 2010 and in Exchange Online and after a short time you can close the HCW:

hcw configure organization relationship

Now, when looking at the Exchange Management Console you can see the Send Connector from Exchange 2010 to Exchange Online. It is configured with the FQDN smtphost.inframan.nl as expected, but the source server of the Send Connector is still the internal Hub Transport server as shown in the following screen shot:

outbound to office 365 connector

Remove the Hub Transport server entry and add the Edge Transport server instead. If you have an Edge Synchronization in place you will see it immediately when you click the Add button.

In Exchange Online, the Receive Connector that’s created will check for any certificate with a wildcard like name, so the smtphost.inframan.nl certificate will automatically be accepted. The Send Connector is also created correctly. The FQDN of the Edge Transport server is used as the server to route message to, and the CN of the certificate that was selected in the HCW is also configured as shown in the following screenshot:

office 365 send connector certificate

We’re almost there. The only thing that needs to be done is to configure the Receive Connector on the Edge Transport Server for TLS from Exchange Online. You should have already configured the Edge Transport Server with the correct 3rd party certificate and when setting up an inbound connection it should use the 3rd party certificate. You can test this using the https://checktls.com tool online.

On the Edge Transport server, execute the following command in the Exchange Management Shell:

Get-ReceiveConnector "smtphost\Default internal receive connector SMTPHOST" | Set-ReceiveConnector -Fqdn smtphost.inframan.nl -TlsDomainCapabilities mail.protection.outlook.com:AcceptOorgProtocol

This will make sure the cross-premises email will be treated as internal email (SCL=-1). If you omit this step, there’s always the risk the email will be treated as external (I’ve seen SCL=5 in my environment) and will end up in the user’s Junk Email Folder.

Summary

When configuring an Exchange 2010 hybrid configuration it is not possible to configure an Edge Transport Server in the Hybrid Configuration Wizard. It is possible to configure this in the HCW for Exchange 2013 and Exchange 2016, but for Exchange 2010 this needs some manual changes.

In this blogpost I showed you the steps needed to configure an Edge Transport Server for secure messaging between Exchange Online and Exchange 2010. When configured this way cross-premises email will be seen as internal email and thus treated accordingly.

Exchange Hybrid

Your message couldn’t be delivered because you weren’t recognized as a valid sender

November 15, 2018 jaapwesselius 2 Comments

Today a customer ran into an interesting issue. A user was not able to send out email to external recipients (this was already the case for a couple of weeks) but internal email, both in Office 365 as well as hybrid Exchange 2010 did work fine.

The NDR that was returned to the user said:

Delivery has failed to these recipients or groups:
jaapwess@gmail.com
Your message couldn’t be delivered because you weren’t recognized as a valid sender. The most common reason for this is that your email address is suspected of sending spam and it’s no longer allowed to send messages outside of your organization. Contact your email admin for assistance.

At first, the only I read was “Your message couldn’t be delivered because you weren’t recognized as a valid sender” so it took me some time to figure out what was wrong.
It’s not a permission issue (was my first thought) but Exchange Online Protection is blocking the account because of spam.

Even in a hybrid scenario with centralized mail transport this can happen, because Exchange Online outbound mail (to Exchange 2010 on-premises) is still handled by Exchange Online Protection.

To check the outbound spam and the user that is blocked, open the Exchange Online Admin Center, select protection in the navigation bar and click the action center tab. Here you can see the user account that is blocked, including the reason and date for blocking as shown in the following screenshot:

For this specific user:

Reason:
OutboundSpamLast24Hours=122;OutboundMailLast24Hours=128;OutboundSpamPercent=953;Last Message MessagetraceId:4495783e-13af-483c-b8d2-08d643c0f46c

Date:
11/6/2018 8:22 AM

So, it was already blocked for 9 days and 122 outbound spam messages were detected the last 24 hours.

I asked the local IT guys to go to this specific workstation, perform an ant-virus run to clean-up the workstation so I can unlock the account.

Update. Some items from the protection and/or compliance center are moving to the Security & Compliance Admin Center (https://protection.microsoft.com). You can find the restricted users (i.e. users that are blocked from sending outbound email) under Threat Mangement | Review and Restricted Users.

Exchange Hybrid, Uncategorized

Hybrid Configuration Wizard diagnostics

October 4, 2018 jaapwesselius 2 Comments

Life can be so simple sometimes… learned this nice feature at Microsoft Ignite last week… when running the Hybrid Configuration Wizard (HCW) and you press F12, the diagnostics tools becomes available:

hybrid-diagnostics

You can open the individual directories, open the log file itself or create a support package when you have to contact Microsoft support in case of issues. Very nice and useful!

support-package

Jaap Wesselius