Category Archives: Exchange

Exchange 2016 Cumulative Update 11

Most likely you’ve seen this information before, because of my vacation in Dallas and New Orleans I’m a bit behind with blogging 😊

But on October 16, 2018 Microsoft has released Cumulative Update 11 (CU11) for Exchange 2016, this is a little later than expected to align the release of Exchange 2016 Cumulative Updates with the upcoming release of Exchange 2019. . There’s only a release for Exchange 2016, there won’t be any new CU’s for Exchange 2013 since Exchange 2013 is already in extended support. There will be security updates for Exchange 2013 though.

Exchange server and .NET Framework is not a happy marriage and it continues to be a struggle, or at least it looks that way. Exchange 2016 CU11 now supports .NET Framework 4.7.2. This version of .NET Framework is not mandatory, installation of .NET Framework 4.7.2 can be before installing of CU11 or after CU11. The .NET Framework 4.7.2 will be required for a future CU of Exchange 2016.

Another dependency is Visual C++, you might have seen this in previous CU’s and also in Exchange 2010 Update Rollup 23 as well. To avoid any issue, install Visual C++ 2012 (https://www.microsoft.com/download/details.aspx?id=30679) before installing Exchange 2016 CU11.

Exchange 2016 CU11 does not have any schema changes. If you’re upgrading from an older version of Exchange 2016, Active Directory changes (in the configuration container) might be needed. These will automatically be applied by the setup application, but you can also choose to update the configuration partition manually by running setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

As always, you should test a Cumulative Update thoroughly before bringing it to production, it won’t be the first time something goes wrong in production with a CU. But I have to say, I haven’t seen any major blocking issues so far…

More information and downloads of Exchange 2016 CU11:

Exchange 2010 and TLS 1.2

In a previous blogpost I discussed an issue I had with Outlook 2010 and TLS 1.2. At the same time this reminded me that Microsoft will remove support for TLS 1.0 and TLS 1.1 in Office 365 on October 31, 2018 as communicated in https://support.microsoft.com/en-us/help/4057306/preparing-for-tls-1-2-in-office-365. This means that when you have communication issues with Office 365 because of an older and weaker protocol, you won’t get any support. Time to do some research….

Existing Exchange 2010 environment

As you may have seen on this side, I still am a big fan of Exchange 2010 and also have an pure Exchange 2010 hybrid environment up-and-running and it looks like this:

Inframan-hybrid

MX records is pointing to my Exchange 2010 Edge Transport Server (running on Windows 2008 R2), webmail and Autodiscover are routed via an F5 LTM load balancer to an Exchange 2010 CAS/HUB/Mailbox server (also running on Windows 2008 R2), and hybrid is configured directly on Exchange 2010 (for hybrid mail flow I’m using a separate FQDN, o365mail.inframan.nl) without any Exchange 2013 or Exchange 2016 server.

So, how do you test which TLS version is used by your Exchange 2010 server? In Exchange 2010 this should be done using the protocol logfiles. Message headers in Exchange 2010 do not contain enough information for showing this TLS information. So, you must enable protocol logging for the appropriate Receive Connectors and Send Connectors. In my environment this means the Default Receive Connector on the Exchange 2010 Edge Transport server (for O365 traffic from other tenants), the Default-First-Site-Name to Internet Send Connector, and both connectors between the Exchange 2010 server and Office 365 for hybrid. Analyzing the protocol logfiles can best be done in Excel (import as CSV files). When analyzing, look for a string like TLS protocol SP_PROT_TLS1_0_SERVER (when receiving) or TLS protocol SP_PROT-TLS1_0_CLIENT (when sending). When TLS 1.2 is used, look for a string like TLS protocol SP_PROT_TLS1_2_SERVER and TLS protocol SP_PROT-TLS1_2_CLIENT.

Continue reading Exchange 2010 and TLS 1.2

Ignite 2018 – Exchange sessions, the good and a bit ugly

My first blog about Ignite 2018 was more about the keynotes on the first day, Microsoft’s strategy around the cloud and how various application integrate with each other and take advantage of the cloud. But I’m a technical consultant and more interested in the technical stuff. And as a MVP in ‘Office Apps and Service’  (used to be ‘Office Servers and Services’, and before that just ‘Exchange MVP’) my heart is still with Exchange. Although I also attended lots of other sessions, there are better blogs available for these technologies instead of mine 😊

Welcome to Exchange 2019

The first break-out session I attended was BRK “Welcome to Exchange 2019” by Greg Taylor and Brent Alinger. Lots of information was already available since Microsoft released the preview version of Exchange 2019, but some other interesting points were mentioned as well:

  • Exchange 2019 runs on Windows 2019 only. There are so much security features in Windows 2019 that Exchange is using, features that are not available in Windows 2016. The preview version was running on Windows 2016, but the final version won’t.
  • Windows 2019 Server Core is the recommended platform because of the lower footprint and attack surface.
  • Required Forest Functional Level is Windows 2012 R2, which may cause issues with customers I guess.
  • Minimum recommended RAM is 128 GB. Be aware, this is the recommended amount of RAM, not the required amount of RAM. This amount comes from .NET usage in Windows 2019 that does much better performance with lots amount of memory. If an Exchange 2019 server does not have 100GB or more, it won’t take advantage of a lot of these improvements. There’s also a correlation with the amount of processors in the Exchange 2019 server, and this 128 GB is related to 48 processors. If you are using less processors, the memory usage decreases as well.
  • Exchange 2019 will (at least for now) only be available via Volume Licensing. Discussions are going on whether this will lead to piracy via download sites. Microsoft is aware of this, but at this moment it’s only Volume Licensing.
  • A very cool new feature is the MCDB, or metacache database. This is a cache stored on SSD drives where metadata from the Mailbox databases is stored, like search information, folder structure etc. This will improve performance for Outlook clients running in Online Mode, not only search information, but also logon to the Mailbox is dramatically improved (I’m starting to sound like a marketing guy 😊
  • Related to the previous bullet, Search is improved (or rewritten actually) using Bing technology. The indexing information is no longer stored in a separate directory on the disk, but it is stored in the Mailbox, and thus inside the Database. This means that passive copies have the same information, and the same search indexes. A failover will now never fail because search is not healthy in replication, speeding up fail-over times.
  • And Microsoft also released the documentation for Exchange 2019, which can be found on https://docs.microsoft.com/en-us/Exchange/exchange-server?view=exchserver-2019.

Unfortunately this session is not available online yet. It is recorded, but somehow not yet available.

As a side note, Microsoft has organized a side meeting with the Exchange and Outlook Product Groups and a couple of (Exchange) MVPs. In this 2 hour session we could have a decent discussion with all Program Managers in the room, and we were able to express our deepest concerns regarding the announcements and presentations at Ignite. Some people prefer to do this on Twitter, but we think it’s better to discuss with the Program Managers directly. They gave us additional background information, but at the same time were impressed by the feedback we gave them, and will take it back to HQ. They won’t change the product of course, but the marking messeag and documentation (background information) around Exchange 2019 will certainly change.

Securing Exchange Online from modern threats

Another interesting session was BRK3148 “Securing Exchange Online from modern threats”. It all about security, and what steps the bad guys usually take to attach a platform like Exchange Online. And it’s incredibly easy. Every heard of ‘password spray’? This a brute force attack, but the other way around. The bad guy has a list of usernames (UPN = Email address to make their life easy) and standard passwords like Summer2018 or Spring2018. But with a spray attack they take a password, and try this against all users. If not successful then try the next password against all users in the list. Incredibly simple, and unlike a regular brute force attack this does not result in locked out accounts. And we all know it, it’s a matter of little time before a simple password is compromised. In the demo the presenter shows how easy it is, and once logged on how to continue with elevated privileges.

The good news is, this presentation is available on YouTube: https://youtu.be/jnUUioUU_eY

Hybrid Exchange: making it easier and faster to move to the cloud

Exchange hybrid is also a hot topic. Last year Jeff Kizner did a session on hybrid, and announced Microsoft was working on removing the last Exchange server when all Mailboxes are moved to the cloud. Expectations were high when attending this years session BRK3143 “Hybrid Exchange: making it easier and faster to move to the cloud”.

The first part of this presentation is about the work done on the Organization Configuration Transfer in the Hybrid Configuration Wizard. Still not finished, but a lot of the configuration information cannot be copied over to Exchange Online. It is copying to Exchange Online, there’s no synchronization. So when making changes in Exchange on-premises, they are not transferred automatically to Exchange Online. You have to either make the changes in Exchange Online, or run the Hybrid Configuration Wizard again.

Completely new (and not available yet) is the Hybrid Agent that runs on-premises. The hybrid agent works with an endpoint in Microsoft Azure, and is outbound HTTPS traffic only. Exchange Online is configured with the HCW to use the same endpoint in Microsoft Azure. This way only outbound connections are used, and it is no longer needed to make all kinds of firewall changes when configuring Exchange Hybrid. Even better, when Autodiscover and Exchange on-premises are not published this still works, since only Outbound connections are used, and configuration information is stored safely in the endpoint in Microsoft Azure.

Exchange Hybrid Agent

TargetSharingEpr

At this moment it is only going to work with Free/Busy and Mailbox moves using the MRS, but it’s a good start. Next versions can include more features, and using this technique everything is possible, imaging Microsoft Search that’s searching your on-premises Mailbox servers…. I’m not sure if that’s a good idea, and I have an idea how the average security officer will react to a solution like this. Some people will refer to this as a man in the middle that has full access to your Exchange environment (something with Exchange Trusted Subsystem). Also, the Hybrid Agent only supports auto-update, and I’m not sure if I want that to happen on my Exchange servers. The good news, you can run the Agent on dedicated servers instead of the Exchange servers, as long as these servers have a decent Internet connection.

The Exchange Product Group released a blogpost on both topics, which can be found on https://blogs.technet.microsoft.com/exchange/2018/09/26/announced-improvements-to-hybrid-publishing-and-organization-configuration-transfer/.

Also, the presentation itself is available on YouTube: https://youtu.be/QhOh5RCcLu8

Unfortunately not a single word about that last Exchange server on-premises, so at this point this will need to be available for some time I’m afraid.

Email search in a flash! Accelerating Exchange 2019 with SSDs

As already mentioned in the first section of this blog, Microsoft introduced Metacache Database (MCDB) in Exchange 2019. Exchange 2019 will now work in the regular JBOD solutions, but now added are SSD disks in the servers. These SSD disks are used as an additional cache (special data from the mailbox database is stored additionally on the SSD disks) to speed up performance. Think about the message table, mailbox information, message metadata information, all kinds of information that’s regularly needed by Outlook clients, and can now be retrieved by the Exchange 2019 at a much faster pace.

Personally I think this is a very impressive technology, but I don’t see it appear at customers anytime soon. It is build on top of a DAG (should be no issue), but is also using AutoReseed as outlined in the Preferred Architecture. The SSD versus spinning disks ration is 1:3, so for a 12 disk Exchange server, three SSD disks are needed. Now, I don’t see a lot of customers deploying Exchange 2019 this way, at least not the smaller organizations, but maybe these customers are better off with Exchange Online, but that’s a different discussion.

metacache database

metacache guidance

The technology is impressive, and I’m looking forward to test this is a lab. Unfortunately this feature is not yet available in the preview version of Exchange, so you have to wait until the official release of Exchange 2019 later this year.

The presentation is available on YouTube: https://youtu.be/VHrScskhCQk

Stay tuned for more information…

 

Outlook 2010 disconnected with TLS 1.2

When my normal laptop died last week I had to use an older laptop, and this laptop had Windows 7 and Outlook 2010 installed, one of my personal favorite Outlook clients.

However, Outlook 2010 did work correctly with Mailboxen in Exchange Online, but Outlook refused to work with Mailboxen on my on-premises Exchange 2016 server. The only thing I saw in the lower right corner was “Disconnected” and every now and then Outlook tried to connect, but no luck.

image

When checking the Connection Status in Outlook I could see that the directory connection was established, but the Exchange Connections disconnected. The Exchange sever and mailbox were ok since I was able to connect using OWA and my Outlook for iPhone client.

image

The Test Email AutoConfiguration option in Outlook wasn’t very helpful either, it just showed that it was unable to determine the settings and none of the Autodiscover options worked.

image

image

Using the Internet Explorer browser I tried to access my Autodiscover.exchangelabs.nl site, and after a logon prompt I got the famous ErrorCode 600. This is good, so I know my Autodiscover is at least listening properly.

image

The Exchange Remote Connectivity Analyzer (http://aka.ms/exrca) showed that there was an issue with my SSL certificate:

image

The SSL certificate however is a valid Digicert UC certificate and there’s nothing wrong with this certificate. IE does use it, and the Digicert help utility doesn’t show anything strange either.

image

Oh, and my Outlook 2016 running on another computer did work correctly, so there should be a configuration error impacting Outlook 2010 only.

Then I realized that a week before I accidentally ruined the Virtual Service on my Kemp Load Balancer and I quickly created a new Virtual Service using the correct template. As a security measure I only selected TLS 1.2 on the SSL properties of the Virtual Service.

image

After enabling TLS 1.0 on the Virtual Service, Outlook 2010 started to work correctly again and (to my surprise) so did the Remote Connectivity Analyzer.

image

image

So, obviously TLS 1.0 was the culprit here and by enabling TLS 1.0 Outlook 2010 started to work again.

When checking my laptop using the SSLLABS website (https://www.ssllabs.com/ssltest/viewMyClient.html), all looks fine and TLS 1.2 is fully supported by my Windows 7 client:

image

It must be something with Outlook 2010 and TLS1.2. I found an interesting article on Technet regarding enabling of TLS 1.1 and TLS 1.2. Create a DWORD value DefaultSecureProtocols in the registry under the following keys:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp


HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Internet Settings\WinHttp

Its value should be one of the following:

For only TLS 1.1 and 1.2: A00 (hexadecimal)


For TLS 1.0, 1.1, and 1.2: A80 (hexadecimal)

clip_image002

Also, create the following DWORD values DisabledByDefault in the following locations and assign it the value of ‘0’:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Client


HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Client

image

When needed create the necessary subkeys under the \Protocols key.

Now your Windows 7 and Outlook 2010 will support a TLS 1.2 environment only (this is also true for Windows 8 BTW).

Summary

Outlook 2010 does not support TLS 1.2 out of the box. This can be an issue if you or your network department starts implementing a TLS 1.2 environment only. You have to enable TLS 1.2 on the workstation by setting a registry key. After this it works fine.

Next October Microsoft will stop support for TLS 1.0 and TLS 1.1. This means that if you run into an issue caused by TLS 1.0 or TLS 1.1 it won’t be fixed. Please note that Microsoft will continue to accept TLS 1.0 and TLS 1.1 connection from clients, it just won’t be supported anymore.

Microsoft is working on a plan to disable TLS 1.0 and TLS 1.1 but that won’t happen anytime soon. When this is going to happen, Microsoft will give notification 6 months in advance of disabling TLS 1.0 and TLS 1.1.

More information

https://www.ssllabs.com/ssltest/viewMyClient.html

https://blogs.technet.microsoft.com/schrimsher/2016/07/08/enabling-tls-1-1-and-1-2-in-outlook-on-windows-7/

https://support.microsoft.com/en-us/help/4057306/preparing-for-tls-1-2-in-office-365

https://technet.microsoft.com/en-us/library/dn786418(v=ws.11).aspx

Last edited on October 26, 2018.

Exchange 2016 Setup RecoverServer fails with internal transport certificate warning

I am currently working with a customer on their Exchange 2016 design, implementation and disaster recovery process. While writing a new Exchange 2016 disaster recovery document I ran into this issue in my lab environment while running “Setup.exe /Mode:RecoverServer /IAcceptExchangeServerLicenseTerms”.

image

For search engine options this is a part of the actual error message.

Mailbox role: Transport service FAILED

The following error was generated when “$error.Clear();

Install-ExchangeCertificate -DomainController

$RoleDomainController -Services SMTP

” was run: “System.InvalidOperationException: The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

The solution looks simple since it says “the problem has been fixed”. However, running the setup application again results in the next error message.

image

Again, for search engine possibilities:

Performing Microsoft Exchange Server Prerequisite Check

Configuring Prerequisites COMPLETED

Prerequisite Analysis FAILED

A Setup failure previously occurred while installing the HubTransportRole role. Either run Setup again for just this role, or remove the role using Control Panel.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.InstallWatermark.aspx

The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the

:\ExchangeSetupLogs folder.

Z:\>

To remove the watermark, start the registry editor on the Exchange 2016 server and go to HKLM\Software\Microsoft\ExchangeServer\v15\HubTransportRole and delete the Watermark and Action entries.

image

Rerunning the setup application unfortunately results in the 1st error, despite the “the problem has been fixed” and the removal of the watermark entries.

It turns out that I have two Edge Transport servers in my environment, with an Edge Subscription. This Edge subscription is using the self-signed certificate for encryption purposes, and since this self-signed certificate on the new Exchange 2016 server differs from the original (before the crash) self-signed certificate the encryption possibilities fail.

To resolve this, using ADSI Edit to find the msExchEdgeSyncCredential on the Exchange 2016 server you are recovering, and delete all credential entries.

image

When running the Setup application with the /RecoverServer option again (for the third time ) it will succeed and successfully recover the Exchange 2016 server.