Category Archives: Exchange

Exchange 2016 Setup RecoverServer fails with internal transport certificate warning

I am currently working with a customer on their Exchange 2016 design, implementation and disaster recovery process. While writing a new Exchange 2016 disaster recovery document I ran into this issue in my lab environment while running “Setup.exe /Mode:RecoverServer /IAcceptExchangeServerLicenseTerms”.

image

For search engine options this is a part of the actual error message.

Mailbox role: Transport service FAILED

The following error was generated when “$error.Clear();

Install-ExchangeCertificate -DomainController

$RoleDomainController -Services SMTP

” was run: “System.InvalidOperationException: The internal transport certificate for the local server was damaged or missing in Active Directory. The problem has been fixed. However, if you have existing Edge Subscriptions, you must subscribe all Edge Transport servers again by using the New-EdgeSubscription cmdlet in the Shell.

The solution looks simple since it says “the problem has been fixed”. However, running the setup application again results in the next error message.

image

Again, for search engine possibilities:

Performing Microsoft Exchange Server Prerequisite Check

Configuring Prerequisites COMPLETED

Prerequisite Analysis FAILED

A Setup failure previously occurred while installing the HubTransportRole role. Either run Setup again for just this role, or remove the role using Control Panel.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.InstallWatermark.aspx

The Exchange Server setup operation didn’t complete. More details can be found in ExchangeSetup.log located in the

:\ExchangeSetupLogs folder.

Z:\>

To remove the watermark, start the registry editor on the Exchange 2016 server and go to HKLM\Software\Microsoft\ExchangeServer\v15\HubTransportRole and delete the Watermark and Action entries.

image

Rerunning the setup application unfortunately results in the 1st error, despite the “the problem has been fixed” and the removal of the watermark entries.

It turns out that I have two Edge Transport servers in my environment, with an Edge Subscription. This Edge subscription is using the self-signed certificate for encryption purposes, and since this self-signed certificate on the new Exchange 2016 server differs from the original (before the crash) self-signed certificate the encryption possibilities fail.

To resolve this, using ADSI Edit to find the msExchEdgeSyncCredential on the Exchange 2016 server you are recovering, and delete all credential entries.

image

When running the Setup application with the /RecoverServer option again (for the third time ) it will succeed and successfully recover the Exchange 2016 server.

Update Rollup 23 for Exchange Server 2010 SP3

I was a bit surprised finding this one in my mailbox this morning, but Microsoft has released Update Rollup 23 for Exchange 2010 SP3. It’s a security update, and it solves the vulnerability that’s described in CVE-2018-8302 (Exchange memory corruption vulnerability).

A couple of things to be aware of:

  • This update is available via Windows Update and as such can be installed automatically.
  • The Visual C++ 2013 Redistributable package is now a required component. You can download this from https://www.microsoft.com/download/details.aspx?id=40784
    If it’s not installed, a pop-up warning will appear:
    vcdistri
  • If you run the update manually, make sure you use evelated privileges (‘Run as Administrator’). Since you cannot run a .MSP file this way, open a command prompt with elevated privileges and start the Update Rollup from the command prompt. If you don’t use elevated privileges the update won’t install correctly, but doesn’t show a warning in this case. The result is that OWA and ECP might stop working.

Update Rollup 23 is available via the Download Center: https://www.microsoft.com/en-us/download/details.aspx?id=57219

As always…. please test before updating your production environment!

Exchange 2019 Preview Introduction

Why does the fun part always happen when you are on vacation? I’m glad I brought my laptop to Spain…. You might have seen by now that Microsoft released a preview version of Exchange 2019. Some people expected that Microsoft was not releasing any new server software anymore, but the opposite is true. If there’s sufficient demand Microsoft will release software.

From a version perspective, Exchange 2019 is a major upgrade, but from a technical perspective Exchange 2019 is more a minor upgrade. You can see this under the hood, Exchange 2019 is referenced as “version 15.2”, whereas version 15.1 is Exchange 2016 and version 15.0 is Exchange 2013.

So, what’s new in Exchange 2019? New features for Exchange 2019 can be seen in four different areas:

  • Security
  • Performance and Manageability
  • User Experience

I will briefly go through these topics in the next sections.

Security

One major improvement is the support for Windows Server Core. Yes, finally… Exchange 2019 runs on Server Core. And I must admit, it runs fine. You have to get used to the fact there’s no GUI, and you have to start EMS using the ‘LaunchEMS’ command, but it runs great. Be careful though, it’s a new development and it runs on Windows 2019 Server Core, not a word yet about supportability on older versions of Windows Server Core….

So, when you want to look at Server Core, it’s time to dust off your knowledge about SCONFIG

image

What else can you show about Server Core?

image

Oh, and now we’re talking about supportability, when it comes to coexistence, my guess is (and this has always been the case in the past) that only Exchange 2013 and Exchange 2016 are supported in a coexistence scenario. Exchange 2010 is most likely not supported (N-2 support) so when you are still running Exchange 2010 it’s time to think about a migration strategy (did I already mention Exchange 2010 support will end in April 2020?).

If you want to test with Windows 2019 you can grab one from the Windows Insider Preview: https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewserver

Performance and Manageability

If you are running Exchange 2013 or Exchange 2016 you must have experienced issues with indexing and health status of Database copies being ‘unhealthy’. This can become an issue when you want to failover to another Exchange server.

In Exchange 2019 this will improve with the introduction of a new search engine called ‘Big Funnel’. Indexing will no longer be in separate files, but it will be included in the Mailbox database. And since Mailbox database copies are always in sync (or at least they should be) this should result is faster failover times, less complexity and less issues.

When looking at the database directory structure it is obvious that the index files are missing. You can use the Get-MailboxStatistics cmdlet to retrieve information regarding the new search engine:

image

Microsoft does have experience with Big Funnel, it is already running in Exchange Online and in Outlook.com, so it’s not an entire new technology.

New developments in performance as well, since Microsoft is supporting SSD disks for Mailbox databases. But it’s a bit more granular than this. Only parts of the Mailbox database need to be stored on the SSD disks. For regular Mailbox items it doesn’t make sense to store them on SSD, and regular JBOD storage will do, but the ‘metache’ information, stuff that gets accessed frequently and randomly can be stored on SSD disks. Unfortunately, the Exchange 2019 build that I have used so far doesn’t have the PowerShell cmdlets to manage this, so I haven’t been able to test this. I do hope the Preview has these available, so I can start testing this after my vacation in a couple of weeks

According to Microsoft Exchange 2019 should support up to 48 processor cores and up to 256 GB. Nice to know, maybe if you’re running datacenters like Office 365, but I don’t have the hardware to validate this I’m afraid. But it’s good Microsoft is also expanding in this area.

User Experience

Some improvements in the User Experience area. Calendaring has always been an issue, and in Exchange 2019 we will see improvements like ‘do not forward’, simplified sharing and better OOF (Out-of-Office) handling. And there will be a Remove-CalendarEvent cmdlet in PowerShell, which allows administrator to remove (orphaned) calendar events, very useful! And more PowerShell improvements, it will be possible to assign delegate permissions using PowerShell!

So, how does Exchange 2019 look like from a user perspective. Since it is very similar to Exchange 2013 and Exchange 2016, the logon screen looks like this:

image

And after logging on:

image

And it even works with Office Online Server (I used the 2016 version of OOS here):

image

The User Interface is not very exciting compared to Exchange 2013 or Exchange 2016, the good thing here is that you don’t have to educated your end users when moving to Exchange 2019 so that’s a win as well.

Message Hygiene

I have been playing with Exchange 2019 for some time now. When working on-premises it’s best to use Exchange Online Protection (EOP) for message hygiene purposes. The Edge Transport server is still around in Exchange 2019, but nothing special here. For DKIM and DMARC you still need to use EOP, and using Azure AD Connect is works great. Of course, you can use any message hygiene solution you want, there’s no mandatory requirement to use Office 365 or EOP at all. I have heard rumours in the past, that you need to have a tenant in Office 365, but that’s not true.

Of course other message hygiene solutions can be used as well, both on-premises as online, but I happened to test with EOP for Exchange 2019.

What will be removed?

With every new version features are introduced, and other features are deprecated or even removed.

The Unified Messaging server role is removed from Exchange 2019, so if you are using UM you should stick on Exchange 2016 for some time to see how this will develop with cloud alternatives. Or move to Exchange 2019 and also move to Skype for Business 2019 and move to Cloud VoiceMail, but at this moment that’s a bit outside my comfort zone yet.

Summary

Exchange 2019 Preview is a first look at the new upcoming Exchange 2019, which should be released later this year. I don’t have any insight in a release date, but according to Microsoft a lot more information will be released at Ignite this September (I am going to Ignite, are you? And I’m looking forward to hearing more information).

There are some interesting new features, where large organizations will benefit from like the new search engine and the database metcache on SSD improvements. For sure there must be more, but the upcoming months will tell.

Exchange 2016 CU10 and Exchange 2013 CU21 released

On June 19, 2016 Microsoft released Exchange 2016 CU10 and Exchange 2013 CU21, exactly 90 days after the previous CUs. Perfectly aligned with their regular quarterly release 🙂

Besides regular hotfixes there are a couple of important things to notice:

  • Exchange 2016 CU10 and Exchange 2013 CU21 need the .NET Framework 4.7.1. This is a hard requirement, so if .NET Framework 4.7.1 is not installed, the setup application will halt and generate an error message. You can use the Get-DotnetVersion.ps1 script that fellow MVP Michel de Rooij wrote to check the .NET version in advance.
  • A new requirement is the VC++ 2013 runtime library. This component provides WebReady Document Viewing in Exchange Server 2010 and 2013 and Data Loss Prevention in Exchange Server 2013 and 2016. In the (near) future the VC++ 2013 runtime library will be forced to install.
  • Standard support for Exchange 2013 ended on April 10th, 2018 and thus Exchange 2013 entered extended support. Exchange 2013 CU21 is the last planned CU. Customers need to install this CU to stay in a supported configuration, and to be able to install future Security Releases.
  • When running a hybrid configuration with Exchange Online, customers are required to install the latest Cumulative Update for Exchange 2013 or Exchange 2016, or install the latest Update Rollup for Exchange 2010 SP3.
  • None of these releases bring Active Directory Schema changes. You have to run Setup.exe /PrepareAD to activate new features like the following:
  • A new feature in Exchange 2016 CU10 and Exchange 2013 CU21 is the option to create shared mailboxes in Office 365 using the *-RemoteMailbox cmdlets. For example, after creating a user account in Active Directory you can use the following command to create a Shared Mailbox in Office 365 directly:
    Enable-RemoteMailbox -Identity <account> -Shared -RemoteRoutingAddress account@contoso.mail.onmicrosoft.com

Microsoft also released Update Rollup 22 for Exchange 2010 SP3. This Update Rollup brings support for Windows 2016 Domain Controllers (and corresponding Domain Functional Level and Forest Functional Level) and it fixes an issue with Web Services impersonation.

As always you should thoroughly test the new Cumulative Updates or Update Rollups in your test environment before installing in your production environment.

Installing a Cumulative Update hasn’t changed much over the years, so you can follow my previous blogpost about installing Exchange 2013 CU9, which is especially important when installing a Cumulative Update in a Database Availability Group.

More information and downloads:

Source based routing in Exchange

In Exchange server Send Connectors are used to route messages from the Exchange organization to external recipients. Routing is based on the namespace of the recipient. You can create an Internet Send Connector with a namespace “*”, which means that all outbound messages are routed via this Send Connector.

You can also create a separate Send Connector with a namespace “fabrikam.com”. All messages with destination user@fabrikam.com are sent via this Send Connector, all other messages are sent via the other Send Connector. Routing via specific smart hosts or implementing domain security (i.e. Forced TLS) are good examples of using dedicated Send Connectors.

In Exchange unfortunately it is not possible to route message based on (properties of) the sender in Exchange. For example, users in a communications department should send all messages via a dedicated, high priority Send Connector, or members of the Sales Group should always send their messages via smarthosts in the DMZ, while other users send their messages via Exchange Online Protection. You can think of various examples, specific for your organization.

A customer wanted to implement source-based routing. Based on Active Directory Group membership or a property in Active Directory, outbound email should either be routed via the Symantec Messaging Gateway (SMG) or via Exchange Online Protection.

In Exchange you need a 3rd party solution, and one of the 3rd party solutions is the Transport Agent of Egress Software Technologies (www.egress.com). The Egress Transport Agents is a small software package that is installed on the Exchange 2010 Hub Transport server, of the Exchange 2013/2016 Mailbox server. It can also be installed on an Edge Transport server. It is installed in the C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\agents\SmtpAgents directory, and the Transport Agent is configured using a configuration file.

In my lab environment I’ve installed it on two Edge Transport servers (one Exchange 2013 and the other Exchange 2016). I can route messages via Trend Micro Hosted Email Security (HES) or directly via the Edge Transport servers. Of course, there are two Send Connectors to facilitate each route.

The Edge Transport servers make a routing decision based on a header in the email message. If the message contains a header ‘X-RoutethroughTM’ the message is routed via Trend Micro HES, if this x-header is not present it is routed via the regular Send Connector.

The X-RoutethroughTM header is added on the internal Mailbox server. When a user is a member of a Security Group called TrendUsers, then this X-header is added. This is achieved using a Transport Rule on the Mailbox servers:

image

When a message is sent from a mailbox which is a member of this Security Group, it is routed via Trend Micro. When sending to a Gmail mailbox it is visible in the message headers:

image

The Transport Agent does extensive logging, and the outbound messages including the trigger is visible in this logfile:

2018-04-19T10:25:05.9304327Z TID=24 ID=13306 [#016636d20b19] Event: [OnResolved]. Processing message
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 ID=[]
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 Details=[Interpersonal]
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 Delivery=[Smtp/2018-04-19T10:25:05.8246623Z]
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 Subject=[Routing via TM?]
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 From=[jaap@Wesselius.info]
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 To=[jaapwess@gmail.com]
2018-04-19T10:25:05.9304327Z TID=24 ID=13306 Attachments=[]
2018-04-19T10:25:05.9304328Z TID=24 ID=13307 [#016636d20b19] Event: [OnResolved]. Rule execution log:
2018-04-19T10:25:05.9304328Z TID=24 ID=13307 > Rule #1. Found header [x-routethroughtm: On]
2018-04-19T10:25:05.9304328Z TID=24 ID=13307 > Rule #1. Recipients: [jaapwess@gmail.com]
2018-04-19T10:25:05.9304328Z TID=24 ID=13307 > Rule #1. Recipient [jaapwess@gmail.com]. Rerouted to [trend.local] via [UseOverrideDomain].
2018-04-19T10:25:05.9304329Z TID=24 ID=13308 [#016636d20b19] Event: [OnResolved]. Message ID=[] processing completed. Result: [Routed].

This was added after the blog was published because of visibility/readability:

message based routing X-header

A message from a mailbox that’s not part of this Security Group shows different headers:

image

And the Transport Agent logfile:

2018-04-19T10:24:56.1352082Z TID=48 ID=13306 [#1f1e25edd0da] Event: [OnResolved]. Processing message
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 ID=[]
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 Details=[Interpersonal]
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 Delivery=[Smtp/2018-04-19T10:24:56.0250359Z]
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 Subject=[Routing via Edge?]
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 From=[administratie@Wesselius.info]
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 To=[jaapwess@gmail.com]
2018-04-19T10:24:56.1352082Z TID=48 ID=13306 Attachments=[]
2018-04-19T10:24:56.1352083Z TID=48 ID=13307 [#1f1e25edd0da] Event: [OnResolved]. Rule execution log:
2018-04-19T10:24:56.1352083Z TID=48 ID=13307 > Rule #1. Header [x-routethroughtm] was not found.
2018-04-19T10:24:56.1352084Z TID=48 ID=13308 [#1f1e25edd0da] Event: [OnResolved]. Message ID=[] processing completed. Result: [NoAction].

And again an added screenshot for readability:

Message based routing X-header

The config file is easy to configure. When using the X-header as shown above it would contain:

image

Note. Unfortunately I was not able to post the config info itself on my blog, as WordPress does not accept this.

The header rule can be configured extensively using a headerValuePattern or a headerValueNotPattern. These are regular expressions, like:

image

In this example, all messages with the X-RouteThroughTM header are routed to the trend.local connector. However, if the X-RouteThroughTM header has label “public”, do not route. Also, do not route for recipients in @secure.com and @secure.eu domains.

Summary

Source Based Routing is not possible in Exchange server and you need a 3rd party solution to achieve this. The Transport Agent solution from Egress Software is a highly customizable tool that can achieve this and the last couple of months it has been proven to be stable.

On general Exchange remark though, after upgrading to a newer CU you have to redeploy the Transport Agent. Not a big deal, only a matter of executing a setup.ps1 PowerShell script (but easy to forget)