Tag Archives: TLS

Exchange Resource Forest and Exchange Hybrid – Part III

In my previous two blogposts (part I and part II) I’ve explained more about the Exchange Resource Forest model and how to implement Azure AD Connect into such an environment. In this blogpost I’ll show you more about creating a hybrid environment with an Exchange Resource Forest model.

Exchange 2010 Hybrid

If you have been following my blog, or maybe my work as a consultant you most likely know I’m not a big fan of installing Exchange 2016 into an existing Exchange 2010 environment when creating a hybrid environment. It adds a lot of additional complexity since you are halfway a migration to Exchange 2016, you need network and client access changes and most likely hit users multiple times. Better is to create an Exchange 2010 hybrid scenario and when the migration to Exchange Online is done, upgrade the Exchange 2010 remains to Exchange 2016.

My Resource Forest environment is built on Exchange 2010 (that’s what most of my customers are still running) and I will create another Exchange 2010 hybrid environment, but this time built on the Exchange Resource Forest. The solution will look something like this:

image

The only more challenging part is the use of an Edge Transport server for inbound and outbound SMTP, but if your SSL certificates are ok, you’re good to go. In our example, the Edge Transport server is used for inbound and outbound SMTP, but the hybrid SMTP will be sent directly from Exchange Online to the Exchange 2010 multi-role server. Centralized Mail Transport will be used, so all mail will always go via the Edge Transport server, even outbound mail from Exchange Online.

Note. Before you continue, you have to make sure that your certificates are ok, that a valid 3rd party certificate is used and bound to IIS and SMTP, and that your load balancer is configured correctly. A common pitfall is that address translation occurs, and that all inbound connections originate from the IP address of the load balancer. In this case inbound SMTP ends up on the wrong connector, causing secure traffic between Exchange 2010 and Exchange Online to fail.

Logon to the Exchange 2010 server and download the Hybrid Configuration Wizard at https://aka.ms/TAPHCW and start the wizard by clicking the Install button.

Click the Next button a couple of times, the wizard will detect the optimal Exchange server to be used to create the hybrid configuration (this is the server where the hybrid configuration wizard is running, and is known as the ‘hybrid server’) and logon to the Office 365 tenant using a tenant administrator account as shown in the following figure:

image

Continue with the wizard, select Full Hybrid (or minimal hybrid if you need to), and create a federation trust (and enter this crazy TXT record in public DNS). When you reach the radio button for Configure my Client Access and Mailbox server window, you can select the enable centralized mail transport checkbox if you want to.

image

Select the Hub Transport server (or Mailbox server when running Exchange 2013 or Exchange 2016) that should be used for secure communication with Exchange Online. This server is configured in an Office 365 Send Connector and a Receive Connector from Office 365 is created on this server.

image

Select a proper certificate (which should already be present on the Exchange server of course), enter the Organization FQDN that’s used to access your on-premises environment (i.e. webmail.exchangefun.nl) and you’re ready to finalize the hybrid configuration wizard. The options you’ve selected in the wizard are now pushed to the Exchange server and Active Directory when you click the update button.

image

And after a minute or two the Hybrid Configuration Wizard should be finished, and of course no warning message should be shown:

image

We’ve now configured a hybrid configuration with an on-premises Exchange 2010 server that’s in a Resource Forest.

Move Mailbox

An easy way to test the new hybrid configuration is to test a mailbox move from Exchange 2010 on-premises to Exchange Online. To do so, logon to the Exchange (Online) Admin Center, go to Recipients | Migration and start a new migration batch. Select move to Exchange Online and select a user to move to Exchange Online as shown in the following figure:

image

Enter the on-premises administrator account to find a proper migration endpoint (through Autodiscover):

image

It will automatically detect and show the migration endpoint on the Exchange 2010 server:

image

Click Next to continue, enter a migration batch name, increase the bad item and large item limit if needed and follow the wizard. The migration batch is automatically started, but manually completed. I typically complete migration batches off business hours, but for a test or lab environment you can safely select to complete the batch automatically. When you click the new button a new migration batch is created, and the mailbox move is automatically initiated. When the mailbox is moved to Exchange Online you can logon to Office 365 and start testing.

image

The first test is to see if mail flows between Exchange 2010 on-premises to Exchange Online. In the previous figure the mailbox ‘Jaap Wesselius [Linked]’ is a mailbox that was not migrated, so this works fine. Checking the header of this message reveals the same:

image

The figure might be a bit blurry, but in the last column we can see that TLS 1.2 is used for communications between Exchange Online and Exchange 2010.

Sending from Gmail to the mailbox in Exchange Online reveals that Gmail sends the message to the Edge Transport server, which sends in to the Exchange 2010 server and to Exchange Online:

image

Inbound messaging is working as well. When mail is sent from Exchange Online to Gmail, we can see in the headers that mail goes from Exchange Online to the Exchange 2010 server, to the Edge Transport server and to Gmail.

image

Another important topic to test is free/busy information between Exchange 2010 and Exchange Online. When an on-premises mailbox wants to schedule a meeting with two migrated mailboxes in Exchange Online the following should be visible:

image

The Exchange 2010 server will contact Exchange Online using Exchange Web Services (EWS) to check the availability for the users Don and Duw.

Vice versa, when user Don wants to schedule a meeting the following should be visible:

image

The server in Exchange Online now contacts the Exchange 2010 server (via the load balancer) using EWS to check the availability of the on-premises mailboxes.

It happens a lot that availability information or free/busy information in the on-premises environment is not available. This can be an Autodiscover issue, a certificate issue or a pre-authentication issue in the load balancer. Enough stuff to troubleshoot in this case.

If free/busy is working properly, cross-premises Mail Tips are most likely working as well since this is also using EWS:

image

So, it looks like everything is working as expected.

Summary

In this blog post and the previous two blog posts I’ve explained more about the Exchange Resource Forest model, how linked mailboxes are related to their corresponding accounts, how to implement Azure AD Connect in a Resource Forest environment and how to setup a hybrid environment in this model.

This was built on top of Exchange 2010 but is very similar for Exchange 2013 or Exchange 2016. If all prerequisites are met it doesn’t make any difference if you’re running a single forest environment with Exchange installed or a Resource Forest model.

Since the Resource Forest is a fully supported scenario by Microsoft, the hybrid environment in a Resource Forest is fully supported as well.

In the next blog and final (part IV) of this series I’ll dive deeper into the provisioning part of linked mailboxes and Office 365.

Exchange 2016 CU9 and Exchange 2013 CU20 released

On March 20, 2018 Microsoft has released two new quarterly updates:

  • Exchange 2016 Cumulative Update 9 (CU9)
  • Exchange 2013 Cumulative Update 20 (CU20)

There aren’t too many new features in these CUs. The most important ‘feature’ is that TLS 1.2 is now fully supported (most likely you already have TLS 1.2 only on your load balancer). This is extremely supported since Microsoft will support TLS 1.2 ONLY in Office 365 in the last quarter of this year (see the An Update on Office 365 Requiring TLS 1.2 Microsoft blog as well).

Support for .NET Framework 4.7.1, or the ongoing story about the .NET Framework. The .NET Framework 4.7.1 is fully supported by Exchange 2016 CU9 and Exchange 2013 CU20. Why is this important? For the upcoming CUs in three months (somewhere in June 2018) the .NET Framework 4.7.1 is mandatory, so you need these to be installed in order to install these upcoming CUs.

Please note that .NET Framework 4.7 is NOT supported!

If you are currently running an older CU of Exchange, for example Exchange 2013 CU12, you have to make an intermediate upgrade to Exchange 2013 CU15. Then upgrade to .NET Framework 4.6.2 and then upgrade to Exchange 2013 CU20. If you are running Exchange 2016 CU3 or CU4, you can upgrade to .NET Framework 4.6.2 and then upgrade to Exchange 2016 CU9.

Schema changes

If you are coming from a recent Exchange 2013 CU, there are no schema changes since the schema version (rangeUpper = 15312) hasn’t changed since Exchange 2013 CU7. However, since there can be changes in (for example) RBAC, it’s always a good practice to run the Setup.exe /PrepareAD command. For Exchange 2016, the schema version (rangeUpper = 15332) hasn’t changed since Exchange 2016 CU7.

As always, check the new CUs in your lab environment before installing into your production environment. If you are running Exchange 2013 or Exchange 2016 in a DAG, use the PowerShell commands as explained in my earlier EXCHANGE 2013 CU17 AND EXCHANGE 2016 CU6 blog.

More information and downloads

Exchange 2010 hybrid, SMTP, SSL Certificates and Subject Alternative Names

On every Exchange server you need SSL certificates for authentication, validation and encryption purposes. For SMTP you can use the self-signed certificate. Exchange 2010 uses opportunistic TLS, so the self-signed certificate will do in this scenario. If you need to configure domain security (mutual TLS) on Exchange, you need a proper 3rd party SSL certificate for this.

SMTP communication between Office 365 and Exchange in a hybrid scenario is an example of mutual TLS or domain security. A proper 3rd party SSL certificate is needed on your Exchange server.

I was always under the impression that mutual TLS can only use the Common Name of the certificate, which in my scenario is CN=webmail.inframan.nl. After a previous blogpost there was an interesting discussion (see the comments of this particular blogpost) about this, so now it’s time to do some testing.

Originally I had a Digicert SSL certificate with Common Name CN=webmail.inframan.nl, and a Subject Alternative Name entry autodiscover.webmail.com. During the HCW I entered webmail.inframan.nl and selected the proper certificate.

It was time to renew my SSL certificate, so I added an additional SAN entry o365mail.inframan.nl.

image

Continue reading Exchange 2010 hybrid, SMTP, SSL Certificates and Subject Alternative Names

Upgrade Hybrid Server to Exchange 2016

I’m running a coexistence scenario with Exchange 2013 and Exchange 2016 without too many issues. My hybrid server is running on Exchange 2013 from the beginning, and it is time to upgrade this server to Exchange 2016.

If you have configured your Exchange environment correctly the hybrid server is nothing special. In my environment the hybrid server is just used for sending SMTP messages between Exchange Online and Exchange on-premises, and it is used for migrating Mailboxes back and forth.

Upgrading the existing Exchange 2013 hybrid server to Exchange 2016 is actually just a matter of installing a new Exchange 2016 Mailbox server, configure it correctly like the old Exchange 2013 hybrid server and rerun the Hybrid Configuration Wizard application.

image

Figure 1. The new hybrid server (hybrid02) will be installed next to the old hybrid server (hybrid01)

Continue reading Upgrade Hybrid Server to Exchange 2016

Exchange 2010 UM – A TLS API failure occurred

Recently I was implementing an Exchange 2010 UM server that was not willing to deliver any voicemail messages to the user’s inbox. On the UM server I was several EventID 1423 UMCore error messages in the application eventlog:

Log Name: Application
Source: MSExchange Unified Messaging
Date: 4-2-2013 14:53:25
Event ID: 1423
Task Category: UMCore
Level: Error
Keywords: Classic
User: N/A
Computer: EXUM02.contoso.com
Description:
The Unified Messaging server encountered an error while trying to process the message with header file "C:\Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\voicemail\2d831f7a-2a85-40f2-864c-70b4680a118f.txt". Error details: "Microsoft.Exchange.Net.ExSmtpClient.TlsApiFailureException: A TLS API failure occurred. Error = 0x80090301

At the same time I saw lots of EventID 36885 Schannel error messages in the system eventlog of the Hub Transport Server:

Log Name: System
Source: Schannel
Date: 4-2-2013 12:32:56
Event ID: 36885
Task Category: None
Level: Warning
Keywords:
User: SYSTEM
Computer: exhub01.contoso.local
Description:
When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

Note. As you may know the UM server records the voicemail message and the voicemail message is sent to the user’s mailbox using the Transport Server.

When looking at the Trusted Root Certification Authorities on the Hub Transport Server it turned out that there were 355 certificates stored here.

image

This is where things are breaking. The UM server is using TLS to communicate with the Hub Transport Server and during the handshake between the servers the list of root certs is sent. The maximum size of the package being sent by Schannel is only 16KB and the 355 root certificates never fit in these 16KB. Schannel fails, the list of certificates is truncated, resulting in EventID 36885 and the UM server only sees an invalid handshake with a truncated list of certificates and does not want to communicate.

So, the initial entry in the eventlog on the UM server is a result of a TLS issue and important to note, this is not an Exchange problem!

The way to solve this is delete a large number of root certificates from the Trusted Root Certification Authorities on all Hub Transport server. In my environment I reduced the number of root certs to about 85 which is sufficient. When the certificates are deleted the TLS handshake succeeds and the UM server starts sending the voicemails.

The question is how these 355 root certificates ended up in the trusted root store, a newly installed Windows 2008 R2 server in my test environment only reveals 11 certificates in the trusted root store.

image

Most likely the rootsupd.exe tool has been run sometime ago which updates the list of root certificates on the computer to the list that is accepted by Microsoft as part of the Microsoft Root Certificate Program. Now this is fine on a laptop or workstation, but you don’t want this to happen on your server because it can lead to unpredictable results.

This package however was released to Windows Update and WSUS on December 11, 2012 and was intended for client OS’es only. It also affected servers however and after customer reports the package was set to expired in Windows Update and WSUS.

For more information please check the following knowledge base articles.