Category Archives: Exchange Hybrid

Microsoft Teams and Exchange 2016

Microsoft Teams works best when your mailbox is in Exchange Online, and you have a license for SharePoint Online, and you have OneDrive for Business enabled.

When you have your mailbox in Exchange on-premises your options are limited, you have basic Teams functionality like audio/video and files, but that’s basically it. Not a trace of the calendar or the option to manage meetings in Teams. This is what ‘cloud partners’ have been telling me for a long time: “no, you must migrate your mailbox to Exchange Online” (and we can help you with that I was they were thinking…)

Teams with Exchange on-premises

But, if you have Exchange 2016 CU3 or higher things get better (I’m sorry if you are still running Exchange 2010). To enable the integration of Teams with on-premises Exchange 2016 you need to configure OAuth in your on-premises environment as outlined in my previous blog, and assuming you have Exchange hybrid configured of course.

When you have OAuth configured, Teams can access the on-premises mailbox on behalf of the user that’s logged on in Teams, making it possible to retrieve the user’s calendar. After configuring OAuth and without any additional configuration in Teams the user’s on-premises calendar (here in Exchange 2019) automagically appears:

Teams with Exchange on-premises OAuth

In the Teams client this is a view of your calendar, and using the new meeting option you can create new Teams meetings, invite people, set recurrence, select a channel etc. Invites are sent to other recipients using the regular Exchange process, and when accepted a response is sent (or not) back to your inbox. I never knew this was possible, but this is cool stuff.

Teams with Exchange on-premises new meeting

Integrating the calendar into Teams is a nice addition and it can help users with their daily productivity task. From a compliance perspective things are a bit different. Chats for example are not stored in the user’s mailbox, this is only possible for mailboxes in Exchange Online. The process responsible for copying Teams chat data from the Azure storage (outside of Office 365) simply cannot access the folder in the user’s mailbox.

But, for on-premises calendar integration in your Teams environment this is very nice.

More information

How Exchange and Microsoft Teams interact – https://docs.microsoft.com/en-us/microsoftteams/exchange-teams-interact

Configure OAuth authentication between Exchange and Exchange Online organizations – https://docs.microsoft.com/en-us/exchange/configure-oauth-authentication-between-exchange-and-exchange-online-organizations-exchange-2013-help

 

Configure OAuth authentication in Exchange 2016

As long as I can remember the Hybrid Configuration Wizard finishes successfully, and itgenerates the error about the OAuth portion of the hybrid configuration.

Configure Intra-Organization Connector

HCW8064 – The HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration. If you need features that rely on OAuth, you can try running the HCW again or manually configure OAuth using these manual steps.

The Learn more option redirects to the Microsoft page Configure OAuth authentication between Exchange and Exchange Online organizations. I used that article for the PowerShell commands in this blogpost.

OAuth is used cross-premises to logon to other services, on behalf of the user. So, if you are logged on to some Microsoft service, this service can use OAuth to access services in Exchange on-premises and vice versa.

Example of these cross-premises services are:

  • Message Records Management (MRM).
  • Exchange in-place eDiscovery.
  • Exchange in-place Archiving.
  • Teams calendaring.

The HCW can configure Azure Active Directory for OAuth authentication, it can create the IntraOrganizationConnectors, but it cannot export and import the (self-signed) certificate on the Exchange server, nor can it (or does it) create the authorization server objects in Active Directory. So, time to test, guided by the Microsoft article and write down my experiences.

Note. This only works for Exchange 2013 and higher, I have been working on this in a mixed Exchange 2016 and Exchange 2019 environment.

Configuring OAuth between Office 365 and Exchange Online involve a number of steps.

Create Authorization server objects in Exchange on-premises

To create the authorization server objects in your on-premises environment enter the following commands in the Exchange Management Shell.

New-AuthServer -Name "WindowsAzureACS" -AuthMetadataUrl "https://accounts.accesscontrol.windows.net/contoso.com/metadata/json/1"
New-AuthServer -Name "evoSTS" -Type AzureAD -AuthMetadataUrl https://login.windows.net/contoso.com/federationmetadata/2007-06/federationmetadata.xml

Your verified domain contoso.com (in the command) should be something like exchangelabs.nl, and not <your tenant name> as outlined in the Microsoft article.

New-AuthServer

Enable the partner application for use with Exchange Online

The partner application was created in the previous step (the first command) and this should be enabled. Do this using the following command in Exchange Management Shell (on-premises):

Get-PartnerApplication | ?{$_.ApplicationIdentifier -eq "00000002-0000-0ff1-ce00-000000000000" -and $_.Realm -eq ""} | Set-PartnerApplication -Enabled $true

Export the Exchange authorization certificate

Authentication cross-premises is using certificates, so the on-premises certificate needs to be exported to Azure Active Directory. In case you were wondering where the CN=Microsoft Exchange Server Auth Certificate certificate was coming from when running the Get-ExchangeCertificate command in Exchange Management Shell, here you go.

Use the following PowerShell commands and store them in a PowerShell script called ExportAuthCert.ps1 or something and run it. This should export the OAuth certificate to a file called OAuthCert.cer.

$ThumbPrint = (Get-AuthConfig).CurrentCertificateThumbprint
If((Test-Path $ENV:SYSTEMDRIVE\OAuthConfig) -eq $false)
{
md $ENV:SYSTEMDRIVE\OAuthConfig
}
CD $ENV:SYSTEMDRIVE\OAuthConfig
$oAuthCert = (dir Cert:\LocalMachine\My) | ?{$_.ThumbPrint -Match $ThumbPrint}
$CertType = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert
$CertBytes = $oAuthCert.Export($CertType)
$CertFile = "$env:SYSTEMDRIVE\OAuthConfig\OAuthCert.cer"
[System.IO.File]::WriteAllBytes($CertFile, $CertBytes)

ExportAuthCert

Note. The Export-ExchangeCertificate command doesn’t work in this scenario since the self-signed certificate isn’t exportable.

Import the Exchange authorization certificate into Azure AD

The next step is to import the OAuthCert.cer certificate into Azure AD. Connect to the Microsoft Online service (Connect-MSOLService, if you don’t have this installed you can use the Install-Module MSOnline command) and run the following commands when connected:

$Cred = Get-Credential
Connect-MSOLService -Credential $Cred
$CertFile = "$ENV:SYSTEMDRIVE\OAuthConfig\OAuthCert.cer"
$objFSO = New-Object -ComObject Scripting.FileSystemObject
$CertFile = $objFSO.GetAbsolutePathName($CertFile)
$CER = New-Object System.Security.Cryptography.X509Certificates.X509Certificate
$CER.Import($CertFile)
$binCert = $cer.GetRawCertData()
$CredValue = [System.Convert]::ToBase64String($binCert)
$ServiceName = "00000002-0000-0ff1-ce00-000000000000"
$P = Get-MsolServicePrincipal -ServicePrincipalName $ServiceName
New-MsolServicePrincipalCredential -AppPrincipalId $P.AppPrincipalId -Type asymmetric -Usage Verify -Value $credValue

This will import the self-signed certificate from the Exchange server into Azure AD so it can be used for mutual authentication.

I did not run the commands mentioned above on my Exchange server but on my Azure AD Connect server since the MSOL module was loaded on that server. For importing the certificate file I had to use the following command accessing the certificate file (instead of the $ENV:System variable):

$CertFile = "\\AMS-EXCH01\C$\OAuthConfig\OAuthCert.cer"

Register endpoints in Azure Active Directory

The last step is to register the endpoints of your on-premises Exchange environment into Azure Active Directory. You can use the following commands to register the endpoints:

$ServiceName = "00000002-0000-0ff1-ce00-000000000000";
$x = Get-MsolServicePrincipal -AppPrincipalId $ServiceName;
$x.ServicePrincipalnames.Add("https://webmail.exchangeserver.com/");
$x.ServicePrincipalnames.Add("https://autodiscover.exchangeserver.com/");
Set-MSOLServicePrincipal -AppPrincipalId $ServiceName -ServicePrincipalNames $x.ServicePrincipalNames;

Instead of webmail.exchangeserver.com and Autodiscover.exchangeserver.com you must use your own local FQDNs of the Exchange server (shown below in the verification screenshot).

You can use the the following command to check if this was configured correctly.

Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 | select -ExpandProperty ServicePrincipalNames

As shown in the following screenshot:

Register Endpoints

Note. Over the years I have run the HCW several times. Domains have been added, but not (automatically) deleted in Azure Active Directory.

IntraOrganizationConnectors and AvailabilityAddressSpace

The Hybrid Configuration Wizard created the IntraOrganizationConnectors (both in Exchange 2016 as well as Exchange Online) and configured the AvailabilityAddressSpace. There’s no need to create these, but you have to check them using the Get-IntraOrganizationConnector and the Get-AvailabilityAddressSpace commands)

Verify the OAuth configuration

To verify the OAuth configuration you can use the Test-OAuthConnectivity command. You must do this on the on-premises Exchange server and in Exchange Online.

On the on-premises Exchange server use the Exchange Online Uri and a mailbox on-premises:

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox onprem-user@exchangeserver.com -Verbose | Format-List

In Exchange Online, use the Exchange on-premises Uri with a mailbox in Exchange Online:

Test-OAuthConnectivity -Service EWS -TargetUri https://webmail.exchangeserver.com/metadata/json/1 -Mailbox Online-User@exchangeserver.com -Verbose | Format-List

The output is an extended list, but in the end you should see ResultType: Success and IsValid:True on your console:

Test-OAuthConnectivity

You have now configured the OAuth between Exchange Online and Exchange On-Premises.

 

Outlook 2010 stays offline with Exchange Online

One of my clients is running Exchange 2010 in hybrid mode, and they have Outlook 2010 and Outlook 365 ProPlus client. For testing purposes, I have two VMs, one with Windows 7 and Office 2010 and one with Windows 10 and Office 365 ProPlus. And every Monday morning I run the Windows 7 VM for an hour or so to see if everything is working fine 😊

This morning my Outlook 2010 was working offline, and it didn’t want to go online (OWA and Outlook 365 ProPlus were working fine). Remove the Outlook profile but creating a new Outlook profile didn’t work. After a minute the dreaded an encrypted connection to your mail server is not available error message appeared:

An encrypted connection to your mail server is not available

Mostly this is caused by Autodiscover that goes wrong somewhere, the Remote Connectivity Analyzer shows that Autodiscover to the on-premises Exchange 2010 goes well, but that the redirect to Exchange Online goes wrong and it generates the following error message:

An HTTP 456 Unauthorized response was received from the remote Unknown server. This indicates that the user may not have logged on for the first time, or the account may be locked. To logon, go to http://portal.microsoftonline.com.

And further down more details are revealed:

X-AutoDiscovery-Error: LiveIdBasicAuth:AppPasswordRequired:<RequestId=8a51c25b-9213-4873-aff8-ebc1da40544f>;

An HTTP 456 Unauthorized response was received from the remote Unknown server

The AppPasswordRequired explains more. Last week I changed the MFA settings (see previous authenticator app for Office 365 blogpost). This works fine for OWA and Office 365 ProPlus, but not for Outlook 2010. Since Outlook 2010 does not work with Office 365 MFA, especially not in a hybrid environment (not even with an App Password).

The only workaround here was to temporarily disable MFA for my user account, create a new Outlook profile (which worked fine without MFA) and re-enable MFA. Again, Outlook 2010 does not recognize the MFA and still works with Exchange Online using basic authentication, but all other Office 365 services work fine with Office 365 MFA (both SMS and Authenticator authentication).

Free/busy not working from Exchange Online to Exchange on-premises

Recently users started to complain that free/busy information was not available, more specifically users that had their mailbox in Exchange Online were not able to retrieve availability information from their colleagues or meeting rooms that were still in Exchange 2010 on-premises.

Complaints came from multiple users from multiple countries, there are multiple sites with multiple Exchange 2010 servers with multiple breakout points to the Internet, so the issue was consistent and not really related to only one Exchange 2010 server. And it was only happening cross-premises, and only from Exchange Online to Exchange 2010. From Exchange 2010 to Exchange Online it was working flawlessly, so was between mailboxes in Exchange 2010.

You see this happen often right after configuring an Exchange hybrid configuration with the HCW, but in my case it had been working fine for quite some time, so it had to be related to something we had changed recently. We had WLAN changes (new provider), Windows Update, Exchange 2010 rollup updates, SSL certificates, new Send and Receive Connectors, but nothing that immediately pointed in the right direction. To make things worse, the Remote Connectivity Analyzer (your first stop when troubleshooting) didn’t see any issues, everything worked well. Autodiscover returned the correct information from mailboxes in Exchange Online and Exchange 2010, and the free/busy test in RCA worked well.

Note. A lot of people don’t know this feature, but in the Remote Connectivity Analyzer you can check free/busy in a hybrid environment. Just select the Free/Busy radio button under the Office 365 tab as shown in the following screenshot:

Remote Connectivity Analyzer FreeBusy

My mailbox was in Exchange Online and I also did experience the issue, but at the same time I was able to open the calendar cross-premises. Ok, this is Outlook Anywhere and not Exchange Web Services (which is used for free/busy) but at least it ruled out a firewall issue.

When running the Get-OrganizationRelationship command, I could verify the TargetAutodiscoverEpr property, which was set to the correct Autodiscover URL. Using the TargetSharingEpr property instead of the TargetAutodiscoverEpr property didn’t help.

Get-FederationInformation command with the -DomainName switch… all looks good.
Security on the Virtual Directory? Running these commands often solve unexpected issues:

Get-AutodiscoverVirtualDirectory | Set-AutodiscoverVirtualDirectory -WSSecurity $true
Get-WebServicesVirtualDirectory -Server WPGREXC01 | Set-WebServicesVirtualDirectory -WSSecurity $True

Again, no success… Time for deeper troubleshooting. At this moment Microsoft support was already involved as well….

When testing I could see the request entering the Exchange 2010 server in the IIS logs, but the servers returned a 500 Error, so something in the request was causing issues on the Exchange server:


2019-10-14 06:45:06 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.155.157 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:33 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.140.165 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/Freebusy – 500 0 0 15


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.158.5 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31


2019-10-14 06:45:51 192.168.25.119 POST /ews/exchange.asmx/WSSecurity – 443 – 52.97.139.125 ASProxy/CrossForest/Directory/https/15.20.2347.020/MailTips – 500 0 0 31

The next step to try was to recycle the AutodiscoverAppPool and the ExchangeServicesAppPool in the IIS Manager, but unfortunately this didn’t help.
After looking out of the window what might be the issue I had another look at the eventlog on the Exchange server, and found the following certificate warning:

EventID 403 Certificate is expired


Log Name: Application
Source: MSExchange Common
Date: 10/16/2019 9:28:59 AM
Event ID: 403
Task Category: Configuration
Level: Error
Keywords: Classic
User: N/A
Computer: Exchange.contoso.com
Description:
The certificate named ‘791BC6AD9893AA570DF03452B4F8069C8A743C29’ in the Federation Trust ‘Microsoft Federation Gateway’ is expired. Please review the Federation Trust properties and the certificates installed in the certificate store of the server.

This rings a bell. The certificate with the thumbprint mentioned in the error message is not on the Exchange server, but it’s in the Microsoft Federation Gateway. I didn’t see this earlier, but when checking the federation with Get-FederationTrust | FL you can see certificate information, and one certificate expired some time ago. July 2019 to be precise.

You can also run the Test-FederationTrust on the Exchange server. If you ran into this issue, you should see an error message like Failed to validate delegation token in the TokenValidation section.

Fixing this is easy, just run the following command:

Get-FederationTrust | Set-FederationTrust -RefreshMetadata

After running this command it works like a charm.

This only happens in Exchange 2010 (in Exchange 2013 and higher it is fixed automatically), and looking at the support chances are you are not running Exchange 2010 anymore when other certificates are renewed. You can running the command mentioned before on a regular basis, or you can use a scheduled task to perform this automatically on a regular basis.

Schtasks /create /sc Daily /tn FedRefresh /tr “C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
-version 2.0 -command Add-PSSnapIn Microsoft.Exchange.Management.PowerShell.E2010;
$fedTrust = Get-FederationTrust;Set-FederationTrust -Identity $fedTrust.Name -RefreshMetadata” /ru System

So why didn’t we notice this in the first place? The certificate change was announced by Microsoft and other community blogs, but this was summer holiday time. Also lack of resources in IT Staff didn’t help either. Too bad, but in the end it was fixed (with help of Microsoft support 😊)

More information

 

A hybrid deployment with Office 365 has been detected

There are lots of organizations running Exchange 2010 hybrid, but with the upcoming end-of-life of Exchange 2010 it’s time to move on. Sometimes it’s not always possible to move everything to Exchange Online, so then you must migrate from Exchange 2010 hybrid to Exchange 2016 hybrid.

When preparing Active Directory for Exchange 2016 using the /PrepareSchema swith the following error is (sometimes) raised:

hybrid deployment has been detected

A hybrid deployment with Office 365 has been detected. Please ensure that you are running setup with the /TenantOrganizationConfig switch. To use the TenantOrganizationConfig switch you must first connect to your Exchange Online tenant via PowerShell and execute the following command: “Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML”. Once the XML file has been generated, run setup with the TenantOrganizationConfig switch as follows “/TenantOrganizationConfig MyTenantOrganizationConfig.XML”.

If you continue to see this this message then it indicates that either the XML file specified is corrupt, or you are attempting to upgrade your on-premises Exchange installation to a build that isn’t compatible with the Exchange version of your Office 365 tenant. Your Office 365 tenant must be upgraded to a compatible version of Exchange before upgrading your on-premises Exchange installation. For more information, see: http://go.microsoft.com/fwlink/?LinkId=262888

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.DidTenantSettingCreatedAnException.aspx

(unfortunately the last link doesn’t show any useful information, and it hasn’t been updated since the early Exchange 2013 days).

Note. I’m not sure when and why this message pops up. Right now, it looks like it only happens in older tenants where Exchange 2010 hybrid is already running for some time. If you have more information about this, please feel free to leave a comment.

As stated in the error message you must run the following command in an Exchange Online PowerShell window:

Get-OrganizationConfig | Export-Clixml -Path C:\Install\MyTenantOrganizationConfig.XML

This retrieves the organization configuration from Exchange Online and exports it to an XML file. When opening the XML you can read the organization configuration as shown in the following screenshot:

TenantOrganizationConfig XML

There’s a pitfall when continuing, the /TenantOrganzationConfig switch that’s in the message is a switch that can only be combined with the /PrepareAD switch, it cannot be used with the /PrepareSchema switch. If you do, it will raise a “The parameter ‘tenantorganizationconfig’ is not valid for current operation ‘PrepareSchema’” Error message.

So, you must continue with the following command:

Setup.exe /PrepareAD /TenantOrganizationConfig C:\Install\TenantOrganizationConfig.XML /IAcceptExchangeServerLicenseTerms

Note. Running the /PrepareAD switch will automatically trigger the/PrepareSchema switch.

Now you can continue with the /PrepareDomain switch, followed by the installation of your first Exchange 2016 server in the existing Exchange 2010 environment.