Exchange Resource Forest and Office 365 Part V

Upgrade to Exchange 2016

Two years ago I wrote a number of blogpost regarding an Exchange Resource Forest model and Office 365 starting at Exchange Resource Forest and Office 365 part I. There are four articles, all based on Exchange 2010 and at the end of each article you’ll find a link to the next article. In the more information section at the end of this blogpost you’ll find all links.

Over the years I have received numerous requests on how to continue; moving to Exchange 2016 in a Resource Forest or when all mailboxes are in Exchange Online, decommission the Resource Forest. This is not a problem, but brings a new challenge when it comes to the last Exchange server for management purposes that need to be kept. And yes, this blogpost is the preparation for the ‘decommision the resource forest’ blogpost 😉

I have a few customers running a resource forest environment, and all have this same problem. They must move away from Exchange 2010, either to Exchange 2016 or to Exchange Online (or both). In this blogpost I’ll discuss the upgrade from Exchange 2010 in a Resource Forest to Exchange 2016 (in the same Resource Forest of course) in a hybrid scenario. In one of the next blogs I’ll discuss the way to decommission the Resource Forest in a hybrid scenario.

Note. I did a webinar recently for Kemp on the Upgrading Exchange 2010 topic. It’s a Brighttalk hosted webinar that you can find here The Best Known Methods on Upgrading Microsoft Exchange.

Upgrading Exchange 2010 to Exchange 2016

Upgrading Exchange 2010 in a resource forest is not different then upgrading any other Exchange environment, not even when running this in a hybrid environment. These are the steps that need to be taken:

Design your Exchange 2016 environment (won’t cover that in this blogpost).

  1. Prepare Active Directory for Exchange 2016.
  2. Build your Exchange environment.
  3. Change client access to Exchange 2016.
  4. Run Hybrid Configuration Wizard.
  5. Move Resources to Exchange 2016.
  6. Decommission Exchange 2010.

Prepare Active Directory for Exchange 2016

Before you can install the first Exchange 2016 server, Active Directory needs to be prepared. This is only true for the resource forest, there’s no need to do this at the account forest.

You can prepare the Active Directory schema using the following command:

C:\> Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

It is possible that the following error message is shown:

A hybrid deployment with Office 365 has been detected. Please ensure that you are running setup with the /TenantOrganizationConfig switch. To use the TenantOrganizationConfig switch you must first connect to your Exchange Online tenant via PowerShell and execute the following command: “Get-OrganizationConfig | Export-Clixml -Path MyTenantOrganizationConfig.XML”. Once the XML file has been generated, run setup with the TenantOrganizationConfig switch as follows “/TenantOrganizationConfig MyTenantOrganizationConfig.XML”.

This issue is covered in a previous blogpost: A hybrid deployment with Office 365 has been detected.

Build your Exchange environment

Exchange 2016 can be installed on Windows Server 2012 R2 and Windows Server 2016, I always recommend the latter because of support lifecycle. Windows Server 2019 and Windows Server Core are not supported for running Exchange 2016.

I prefer to install the Exchange 2016 server unattended, especially if you must install multiple servers. You can use a command similar to this:

C:\> Setup.exe /Mode:Install /Roles:Mailbox /DbFilePath:”F:\MDB02\MDB02.edb” /LogFolderPath:”F:\MDB02\LogFiles” /InstallWindowsComponents /IAcceptExchangeServerLicenseTerms

Unattended Setup

After installing the Exchange server, the server should be configured:

  • Virtual Directories; make sure you use the same internalURL and externalURL as the Exchange 2010 servers.
  • SSL Certificates.
  • Storage, Databases and Database Availability Group (when needed of course).
  • Relocate (IIS) logfiles.
  • Send and Receive Connectors (SMTP Relay).

Change client access to Exchange 2016.

When you have configured the Exchange 2016 it is time to test the proxy functionality from Exchange 2016 to Exchange 2010. The easiest way to test this is to edit the HOSTS file for the webmail.contoso.com and Autodiscover.contoso.com entries on a client and point it to the Exchange 2016 server.

hosts file

From the client, connect to the new Exchange 2016 server and start OWA with an Exchange 2010 mailbox. You should see the initial blue OWA screen:

Exchange 2016 OWA Login

And after logging on the yellowish Exchange 2010 OWA user interface. It will automatically proxy to the correct Exchange 2010 mailbox server:

Exchange 2010 OWA Login

The same should happen when you request the Exchange Web Services page on https://webmail.contoso.com/ews/exchange.asmx, this should also successfully proxy to Exchange 2010:

EWS proxy

One important thing here is that this is a down-level proxy only. This means that when a client does a request to Exchange 2016 for a resource on Exchange 2010 (down-level proxy, like in the previous example) it should work. If a client does a request to Exchange 2010 for a request on Exchange 2016 (i.e. up-level proxy) it fails. This is shown in the following figure, and it does not work.

downlevel uplevel

When all is configured and tested, DNS (internal and external) can be changed so that webmail.contoso.com and Autodiscover.contoso.com point to the new Exchange 2016 server. All clients will now connect to the new Exchange 2016 server and connect to their mailbox on Exchange 2010 without a problem.

One thing you must be aware of is RPC/TCP traffic (the original MAPI connection) between Outlook and Exchange 2010 if you are not running Outlook Anywhere. Outlook will still use RPC/TCP to connect to the CAS Array (this is the RPC Endpoint for Outlook) which is running on the Exchange 2010 servers. You can see that when checking the connection status in Outlook. In the following screenshot, outlook.exchangefun.nl is the CAS Array.

Outlook 2010 Connection Status

There are two entries shown, one is for the regular mailbox, the second is for accessing a shared mailbox on Exchange 2010.

When performing the Test E-mail AutoConfiguration check on Outlook 2010, you can see that RPC Connects to outlook.exchangefun.nl, but that OWA, EWS and OAB want to connect to webmail.exchangefun.nl, which is the Exchange 2016 server now.

Outlook 2010 Test Email AutoConfiguration

So, Outlook connects using RPC to the Exchange 2010 server and using HTTPS to connect to the Exchange 2016 server. From Exchange 2016 the connection is proxied to the correct Exchange 2010 server.

Note. Be aware that if you have configured Kerberos authentication in your Exchange 2010 environment you have to upgrade this to Exchange 2016 as well. This is documented in the Microsoft articles Recommendation: Enabling Kerberos Authentication for MAPI Clients, How to Enable Kerberos Authentication for Accessing Exchange in a Resource Forest andExchange 2013 and Exchange 2010 Coexistence with Kerberos Authentication. For Exchange 2010/2016 coexistence you can follow that last link.

Run the Hybrid Configuration Wizard

When all is running well it is time to upgrade the Hybrid Configuration by running the HCW on the Exchange 2016 server. You can find the latest version of the HCW on http://aka.ms/hybridwizard.

The HCW will detect the optimal Exchange server to configure (which should be automatically the Exchange 2016 server). You should select the Exchange 2016 server for the Receive Connector Configuration and the Send Connector Configuration. For the Organization FQDN I typically use the default FQDN (like webmail.contoso.com) to keep everything consistent. Sometimes I see admins use something like hybrid.contoso.com but that doesn’t bring much, except for more complexity.

HCW Organization FQDN

When reaching the end of the HCW you can select to upgrade the Hybrid Configuration. The original Hybrid Configuration was created on Exchange 2010, but now it is upgrade to Exchange 2016 so I don’t see any reason to not check the checkbox.

HCW Upgrade current configuration

The easiest way to check the new Hybrid Configuration is to see if free/busy works cross-premises. Open a mailbox in Exchange Online and check if you can plan a meeting with a mailbox in Exchange 2010 and check the availability of this mailbox. Baron’s mailbox is in Exchange Online, my mailbox is in Exchange 2010 (still). The free/busy request is sent to webmail.exchangefun.nl which is now the Exchange 2016 server. From there it is proxied to the Exchange 2010, and it works like a charm:

Cross-premises freebusy

To check if cross-premises mail security you can send an email from an online mailbox to an on-premises mailbox. If you check the headers, the SCL should have a value of -1 (which means it is treated as an internal message) and the X-MS-Exchange-CrossTenant-AuthAs should have a value ‘Internal’.

At this moment there is an Account and Resource Forest environment, where the Resource Forest contains an Exchange 2010/2016 coexistence environment. All clients, including Exchange Online connect to Exchange 2016 while recipients are still on Exchange 2010.

Exchange 2010 2016 Coexistence

Move Resources to Exchange 2016

At this moment, mailboxes can be migrated from Exchange 2010 to Exchange 2016. And to be honest, this is not that exciting. It is fully transparent for users. It is an online process, so the mailbox content is copied from Exchange 2010 to Exchange 2016 while the user continues to work. Only when the move is finalized, the user is presented the following message box and needs to restart Outlook.

The Microsoft Exchange Administrator has made a change

Now back to the RPC/TCP (MAPI) part earlier in this post. Exchange 2016 does not support RPC/TCP anymore, so when a mailbox is moved from Exchange 2010 to Exchange 2016, the protocol changes as well. In this case, it changes from RPC/TCP to Mapi/Http for Mailboxes, and to Outlook Anywhere for Public Folders (still on Exchange 2010, but Exchange 2010 does not support Mapi/Http).

Outlook 2010 Connection Status Exchange 2016

Public Folders (if any) must also be moved to Exchange 2016. This can only be done when all mailboxes are moved to Exchange 2016. A mailbox on Exchange 2016 can access Public Folders on Exchange 2010, but a mailbox on Exchange 2010 cannot access Public Folders on Exchange 2016. The Public Folder migration is covered in my previous blogpost Exchange 2010 Public Folder migration, which was also performed on my Exchange resource forest environment.

In an Exchange 2010/2016 coexistence environment there are two Offline Address Books:

  • Default Offline Address Book – used in Exchange 2010
  • Default Offline Address Book (Ex2013) – used in Exchange 2016 (the name was not changed when Exchange 2016 was introduced, not sure if this is a cosmetic bug but it does work correctly).

When configuring the Mailbox databases makes sure the \Default Offline Address Book (Ex2013) is set on the OfflineAddressBook property of the Mailbox database.

One pitfall that always takes more time than expected is the Receive Connector configuration, especially for SMTP relay purposes. Multiple devices (scanners, faxes), printers, applications, access on IP level, lack of documentation etc. Use the Protocol logging features on the various Receive Connectors to figure out what devices or applications are using these Connectors and configure them to use the Exchange 2016 server. But beware, this can take a lot of time ☹

Edge Transport Server

You might have noticed I have an Exchange 2010 Edge Transport server running. This is used for SMTP traffic to and from the Internet. My MX record is pointing to the Edge Transport server, but cross-premises SMTP is delivered directly on the Mailbox server.
When is the best time to upgrade the Edge Transport server to Exchange 2016? There’s no real rule of thumb for this. The Exchange 2010 Edge Transport server works fine with an Exchange 2016 Mailbox server, but the other way around also works fine. I typically upgrade the Edge Transport servers to Exchange 2016 after installing the Mailbox servers, but before moving Mailboxes. But there are much more options when doing this.

Decommissioning Exchange 2010

When all resources have been moved to Exchange 2016 you can decommission the Exchange 2010 environment. This is just a matter of:

  • Decommissioning Mailbox database copies.
  • Decommission the Database Availability Group.
  • Remove Mailbox databases.
  • Remove the Public Folder databases (if not removed earlier).
  • Change/Remove Send and Receive Connectors (also see previous section).

When all prerequisites for the decommissioning process have been met, the actual Exchange 2010 servers can be uninstalled. I still see admins just removing the Exchange VM’s but this is not a good idea since the Exchange configuration continue to exist in Active Directory.

Properly remove the Exchange 2010 server using Control Panel and Uninstall or change a program by deselecting all options as shown in the following screenshot.

Uninstall Exchange 2010

After a few minutes the Exchange 2010 server will be fully installed (the official way).

Summary

In this blogpost I explained how to upgrade Exchange 2010 to Exchange 2016 when running in a resource forest configuration, and in a hybrid configuration. The process is not very different from an upgrade in a ‘normal’ single forest, single domain environment, not even from a hybrid perspective.

The process is simple, upgrade Active Directory, install and configure the servers, change client access to Exchange 2016 and run the Hybrid Configuration Wizard. When done you can move resources to Exchange 2016 and when finished decommission the Exchange 2010 environment.

More information

 

Exchange 2010 Public Folder migration

So far, all my customers had decommissioned Public Folders in the Exchange 2010 timeframe, all migration I did have never included Public Folders, until now. The Exchange 2010 (hybrid) to Exchange 2016 (hybrid) migration included approx. 1000 mailboxes and maybe 100 Public Folders (in only one Public Folder database) but they were real Public Folders with real users using them 😊

Migrating Public Folders looks easy, it’s a matter of gathering information regarding the Public Folders and its permissions, copying that over to a Public Folder database and keeping the contents in sync and when you’re ready finalize the migration. Or in steps:

  1. Downloading scripts from Microsoft from https://www.microsoft.com/en-us/download/details.aspx?id=38407.
  2. Preparing the organization for public folder migration.
  3. Generating CSV files using the Microsoft scripts.
  4. Creating public folder mailboxes in Exchange 2016 databases.
  5. Starting the migration, and waiting for initial synchronization to complete.
  6. Locking the public folders for final migration (this requires an outage, usually of at least an hour, but longer for very large environments).
  7. Finalize the public folder migration.
  8. Testing modern public folders, and then unlocking them for access by users (the outage is now over).

I know there are several other blogs available on this topic, but this way I also have my own reference available.

Note. Migrating Public Folders can only be done when all mailboxes are migrated to Exchange 2016. Mailboxes still on Exchange 2010 cannot access Public Folders on Exchange 2016!

Take snapshot of existing Public Folders on Exchange 2010

The first step is to take a snapshot of the existing environment, both the Public Folders, their statistics and permissions. Basic PowerShell and the output is exported to an XML file:

Get-PublicFolder -Recurse | Export-CliXML C:\PFMigration\Legacy_PFStructure.xml
Get-PublicFolderStatistics | Export-CliXML C:\PFMigration\Legacy_PFStatistics.xml
Get-PublicFolder -Recurse | Get-PublicFolderClientPermission | Select-Object Identity,User -ExpandProperty AccessRights | Export-CliXML C:\PFMigration\Legacy_PFPerms.xml

Get-PublicFolderStatistics

Most customers will run into issues with naming of their Public Folders. Customers tend to use strange characters in the Public Folders names (back slash, forward slash) and these are not supported. Hence, they should be replaced. To retrieve a list of all Public Folders with a back slash or forward slash in their name, you can use the following command:

Get-PublicFolderStatistics -ResultSize Unlimited | Where {($_.Name -like "*\*") -or ($_.Name -like "*/*") } | Format-List Name, Identity

You can use any tool you like to change the name, depending of the number of Public Folders of course. If you have 300,000 Public Folders you won’t use the Exchange Management Console to changes names of Public Folders (well, I cannot imagine you would).
Check for any previous migration attempts in your Exchange environment:

Get-OrganizationConfig | Format-List PublicFoldersLockedforMigration, PublicFolderMigrationComplete

Both values should have a value of False:

PF Locked for Migration

You can (should) use the Get-MigrationBatch cmdlet on Exchange 2016 (!) to see if there are any previous migration batches for Public Folders. If there are, delete them using the Remove-MigrationBatch cmdlet before continuing.

Generate the CSV files

The next step is to export the Public Folder statistics using the scripts downloaded from the Microsoft site. This will create a CSV file which is later used to map it to one or more Public Folders mailboxes. Run the following command on the Exchange 2010 server:

.\Export-PublicFolderStatistics.ps1 C:\PFMigration\PFStatistics.csv

Export-PublicFolderStatistics

The PublicFolderToMailboxMapGenerator scripts is used to create a mapping file. This maps the output of the previous command to one or more Public Folder mailboxes.
For this, you need to set the maximum size of a Public Folder mailbox. For example, to use a maximum size of 30GB, the value (in bytes) would be 32212254720. So, for a 30GB Public Folder mailbox you can use the following command:

.\PublicFolderToMailboxMapGenerator.ps1 32212254720 C:\PFMigration\PFStatistics.csv C:\PFMigration\folder-to-mailbox.csv

PublicFolderToMailboxMapGenerator

Create the Public Folder mailboxes in Exchange 2016

The Public Folder database(s) in Exchange 2016 are created using the Create-PublicFolderMailboxesForMigration.ps1 script. This script takes the mapping file that was created in the previous step, and it takes the estimated number of concurrent users. In my environment the number of concurrent users is approx. 900, so the command will be:

.\Create-PublicFolderMailboxesForMigration.ps1 -FolderMappingCsv C:\PFMigration\folder-to-mailbox.csv -EstimatedNumberOfConcurrentUsers:900

CreatePublicFolderMailboxesForMigration

This will create a mailbox called Mailbox1 in a random Exchange 2016 database. If you want to use a specific database, you can edit the Create-PublicFolderMailboxesForMigration.ps1 script and add the -Database parameter to the New-Mailbox command.

PublicFolderMailboxLocation

Public Folder Migration Batch

At this point everything is in place for the Public Folder migration. The Public Folder database has been created, the Public Folders are created in this Mailbox and the permissions are set. A migration batch can be created using the following command on the Exchange 2016 server:

New-MigrationBatch -Name PFMigration -SourcePublicFolderDatabase (Get-PublicFolderDatabase -Server ) -CSVData (Get-Content -Encoding Byte) -NotificationEmails

This would translate to something like:

New-MigrationBatch -Name PFMigration -SourcePublicFolderDatabase PF01 -CSVData (Get-Content “C:\PFMigration\folder-to-mailbox.csv” -Encoding Byte) -NotificationEmails j.wesselius@exchangelabs.nl

Start the migration batch using the Start-MigrationBatch cmdlet:

New-MigrationBatch

It is possible to use the Get-PublicFolderMailboxMigrationRequest to get more information regarding the migration from Exchange 2010 to Exchange 2016:

Get-PublicFolderMailboxMigrationRequestStatistics

You can see this migration batch in the Exchange Admin Center as well (under Recipients | Migration).

Exchange Admin Center

When you click the View Details option, all details regarding the migration, including the number of Public Folders, sizes, performance etc. will be downloaded as a text file.

Migration Status Report

To get more information regarding the batch you can also use the Get-PublicFolderMailboxMigrationRequest in combination with the Get-PublicFolderMailboxMigrationRequest Statistics command. This also gives a lot of information, and using the Select parameter you can only show the information you’re interested in. For example:

Get-PublicFolderMailboxMigrationRequest | Get-PublicFolderMailboxMigrationRequestStatistics | Select StartTimeStamp, CompleteAfter,BytesTransferred, ItemsTransferred, PercentComplete,Message

Get-PublicFolderMailboxMigrationRequestStatistics

The ItemsTransferred value can be compared against the ItemsCount value when running Get-PublicFolder -Recurse | Get-PublicFolderStatistics command on the old Exchange 2010 server. This should closely match.

Finalize the Public Folder migration

All users still connect to the old Public Folders in Exchange 2010 and all changes are replicated to the Public Folders in Exchange 2016. The last stop in the migration is finalizing. In this step the Public Folders in Exchange 2010 are closed (users are disconnected) and the remaining contents are synchronized with Exchange 2016.

Be aware that this automatically means downtime for users. I typically recommend finalizing migrations off business hours. Since finalizing Public Folder migration can take quite some time (and this is unpredictable) I recommend starting this on Friday night.

The migration batch is now completed, and the new Public Folders in Exchange 2016 can be tested. Once ok the migration is finalized, and all users can connect to Public Folders in Exchange 2016. This is fully transparent for users, so no need to do anything on the client side.

To lock the Public Folders for migration, change the organization configuration and complete the migration batch, use the following commands:

Set-OrganizationConfig -PublicFoldersLockedForMigration:$true
Set-OrganizationConfig -PublicFoldersEnabled Remote

It can take some time before the previous commands are activated. When not fully active and continue with the Complete-MigrationBatch cmdlet, you will receive an error message stating:

Before finalizing the migration, it is necessary to lock down public folders on the legacy Exchange server (downtime required). Make sure public folder access is locked on the legacy Exchange server and then try to complete the batch again.

lock-down

You can speed up this process by restarting the Information Store on the Exchange 2010 server (although difficult when running multiple Exchange 2010 servers).

Continue with the following cmdlet to complete the migration batch:

Complete-MigrationBatch PFMigration

To test the new Public Folders, you can enable it for a test mailbox using the following command:

Set-Mailbox -Identity -DefaultPublicFolderMailbox

In our environment, the mailbox identity is Mailbox1. You should check for the following:

  • View hierarchy
  • Check permissions.
  • Access content
  • Create and delete public folders.
  • Post content to and delete content from a public folder.

If all is well, you can finalize the migration using the following commands:

Get-Mailbox -PublicFolder | Set-Mailbox -PublicFolder -IsExcludedFromServingHierarchy $false
Set-OrganizationConfig -PublicFolderMigrationComplete:$true
Set-OrganizationConfig -PublicFoldersEnabled Local

Finalizing

Outlook clients will now (seamlessly) connect to the new Public Folder infrastructure on Exchange 2016. Outlook will use Autodiscover to retrieve Public Folder mailbox information, so it can take some time (again) before this is picked up correctly. You can see this when running the Test E-mail Autoconfiguration option in Outlook (2010 in this example):

Autodiscover

Summary

When moving from Exchange 2010 to Exchange 2016 you have to move Public Folders too. Starting in Exchange 2013, Microsoft introduced the concept of Modern Public Folders. Migrating to Modern Public Folders is not that difficult, but it needs proper preparation.

In our scenario we had issues with Public Folder names (strange unsupported characters) and Public Folders that were no longer in use. These can be fixed or removed before the actual migration takes place.

Another issue we ran into was that it took a long time before all clients discovered that Public Folders had moved to a new environment. Sometimes up to 48 hours (don’t know why and how). When this happens users cannot open Public Folders and start logging tickets at the Servicedesk. The only thing I could say at that point is “be patient”.

Other than that the migration went smooth, all clients (Outlook 2010 and 2016, mailboxes in Exchange 2016 and Exchange Online) were able to access the Public Folders after the migration.

More information

 

Exchange 2016 End of (mainstream) support

As you should (must) know, Exchange 2010 support will end this October. At that point, Microsoft will stop all support for Exchange 2010, including all security fixes. If you are still running Exchange 2010, you must act now and start moving to Exchange 2016 or to Office 365. For an Exchange 2010 to Office 365 migration I have written a couple of blogs before:

Moving from Exchange 2010 to Office 365.

Moving from Exchange 2010 to Office 365 Part II.

But what most people don’t realize is that Exchange 2016 mainstream support will also end this October. From that point forward, Exchange 2016 will be in extended support. This means no more Cumulative Updates and only Security Updates will be released when there updates are marked as ‘critical’.

Note. There’s no direct upgrade path from Exchange 2010 to Exchange 2019, so if you want to follow this route, you must move to Exchange 2016 first, followed by a migration to Exchange 2019.

If you move to Office 365 and have moved all your Mailboxes to Exchange Online, things are getting interesting. In this situation, you still need at least one Exchange server on-premises for management purposes. Microsoft supplies a free Exchange 2016 hybrid license for this situation (there is no free Exchange 2019 hybrid license!), and Microsoft is committed to support this configuration. At least until the moment a final solution is delivered by Microsoft to remove that last Exchange server from your on-premises organization. According to Microsoft, “this does not increase your risk profile in any way” as stated in their article “Exchange Server 2016 and End of Mainstream Support”.
If you still have mailboxes on-premises, the Microsoft recommendation is to move to Exchange 2019. Mainstream support for Exchange 2019 will end on January 1st, 2024, and extended support for Exchange 2019 will end on October 14, 2025 (this is the same date as end of extended support for Exchange 2016).

What to do

  1. If you are still on Exchange 2010, I would urge you to move to Exchange 2016 as soon as possible. Mainstream support for Exchange 2016 will stop this October, but according to Microsoft you are still safe since Security Updates will be released when needed. There’s no direct need to upgrade to Exchange 2019 at this moment, but this is something you must consider the upcoming time. I do know customers however that only want products that are in mainstream support, so if you are in this boat you must move to Exchange 2019 of course.
  2. If you are running Exchange 2013, you must start moving to Exchange 2019 anytime soon for optimal support and skip Exchange 2016.
  3. If you are in an Exchange 2016 hybrid scenario and all your mailboxes are in Exchange Online, you are safe to stay in this situation until Microsoft releases a final solution for that dreaded last Exchange server on-premises for management purposes.
  4. If you are in Exchange 2016 hybrid and you still have mailboxes on-premises, you must move this to Exchange 2019 hybrid to stay in a supported configuration. Please keep in mind that Microsoft only supports the version n-1 in a hybrid scenario, which means automatically Exchange 2019 after this October.