Category Archives: Exchange

This could be due to CredSSP encryption oracle remediation

From a security perspective this is not really a best practice, but sometimes you get into this horrible situation where you cannot logon to a server using RDP, and you don’t have access to the server console… sometimes necessity knows no law…

When you try to logon to a remote server using RPD an authentication error occurs, and you are not able to logon the following error is shown:

cred-ssp-issue

An authentication error has occurred.
The function requested is not supported
This could be due to CredSSP encryption oracle remediation

Unfortunately, the link provided in the error message points to a non-existing page on the Microsoft website…

In March 2018 Microsoft released a fix that addresses a CredSSP, “Remote Code Execution” vulnerability (CVE-2018-0886) that could impact RDP connections. If the host you are working on has this fix, and the server you are connecting to does not have this fix (can occur when deploying new VM’s remotely) the error shown above pops-up.

The best solution is to update the host you’re connecting to, but if it’s not possible to get access to the console for whatever reason, you can also lower the security on your own host (ouch!).

To do this, add the following registry entry to your own host:

REG ADD HKLM\Software\Microsoft\Windows\CurrentVersion\Policies\System\CredSSP\Parameters\ /v AllowEncryptionOracle /t REG_DWORD /d 2

cred-ssp

I strongly recommend raising security again when you have updated the remote server.

The version of the Client Access server selected is not supported

When running the Hybrid Configuration Wizard in an Exchange 2010 environment (I reproduced this with Exchange 2010, but didn’t try this with Exchange 2013 or Exchange 2016) the following error message is generated:

unsupported version

The version of the Client Access server selected, <ServerName>, is not supported. Please go back and select a server that is supported or upgrade the server to a supported version. If Exchange Server 2010, please install the wizard on the same machine.

Note. The HCW is not run on the Exchange 2010 since it requires .NET Framework 4.6.2 and this version of .NET Framework is not supported on Exchange 2010. Even worse, I’ve seen issues with Exchange 2010 after installing .NET Framework 4.6.2 so it’s a bad idea after all.

Running Exchange 2010 on a server with .NET Framework 4.5.x installed is fully supported, but the HCW won’t install on such an Exchange 2010 server since HCW depends on .NET Framework 4.6.2 and the following error message is generated:

unable to install or run this application

So, we are in a deadlock situation. HCW requires .NET Framework 4.6.2 which is not supported on the Exchange server, and when running the HCW on a non-Exchange 2010 server with the correct version of .NET Framework it fails with an error message.

We have been working with Microsoft CSS (product support) on this case. While it should be fixed in the HCW in the first place, under supervision of CSS the following workaround is available.

If you have HCW open and face this error, press F12 and a few other options appear as can be seen in the following screenshot:

HCW Error

If you click Open Logging Folder you get to the folder where the HCW Logs are stored. If you open the correct logfile and search for *ERROR* you can find something similar to:

server error

Obviously the HCW does an incorrect version check (at least when not running on the Exchange 2010 server itself) so it stops. Version checking is something that was built recently into the HCW so Microsoft can check for N-2 version of the implemented Exchange version.

Back to the error message, if you click Open Process Folder a new HCW command prompt is opened on the correct location:

HCW Prompt

Now when you start the Hybrid Configuration Wizard from the Command Prompt you can use the /dv switch (Microsoft.Online.CSE.Hybrid.App.exe /dv) and now the HCW will not do a version check and continue running and finish successfully.

Important note. This was done under the supervision of Microsoft CSS and should not be done by customers directly. If you are running into this issue, please contact Microsoft support to get the right support. Before you know things break beyond repair (and beyond support).

More information

Updated: December 6, 2018

 

Autodiscover in an Exchange interorg migration with Quest QMM

Outlook clients get their configuration information using the Autodiscover protocol from the Exchange server where their mailbox resides and the underlying Active Directory. This works fine, until you are in an interorg migration scenario using the Quest Migration Manager (QMM) for Exchange.

When using Microsoft tools (ADMT, Prepare-MoveRequest.ps1 and New-MoveRequest) the source Mailbox in Exchange is converted to a Mail-Enabled user at the moment of migration finalization. At the same moment the Mail-Enabled User property targetAddress is stamped with the SMTP address of the Mailbox in the new forest. SMTP works fine now, and also Autodiscover will follow the SMTP domain that’s in the targetAddress property. This is true for an interorg migration on-premises, but it is also true when moving Mailboxes from Exchange on-premises to Exchange Online in a hybrid scenario.

When using Quest tooling things are a bit different. The source Mailbox is not converted to a Mail-Enabled User, but it continues to exist at a regular Mailbox. The Outlook profile on the desktop is converted using the CPUU tool using local Autodiscover.xml files so that the Outlook client no longer connects to the old Mailbox but to the new Mailbox.

This works fine for the existing client, but when a user gets a new laptop, or has to configure the Outlook profile again, Outlook will use the Autodiscover process and thus connect to the old Mailbox. Since this isn’t converted to a Mail-Enabled User, Outlook will find the (old) Mailbox, it will stop searching (and thus will not follow the targetAddress property) and return the configuration information for the old Mailbox.

To fix this, we have to export the Autodiscover information from the new Exchange organization to the old organization. I found an old blogpost written by Andread Kapteina (Senior Consultant at Microsoft) in the Google cache (since his blogs no longer exist at the Microsoft Technet Site) about this scenario in an interorg Exchange 2007 to Exchange 2010 migration, but I found that it is also valid in an interorg Exchange 2013 to Exchange 2016 migration. And it should be valid in every interorg Exchange migration from Exchange 2007 and higher.

Export-AutodiscoverConfig

To export the Autodiscover configuration from the new Exchange 2016 to the old Exchange 2013 organization, execute the following commands on the Exchange 2016 server:

$OldCred = Get-Credential OldForest\administrator
Export-AutodiscoverConfig -DomainController <NewForestFQDN> -TargetForestDomainController <OldForestFQDN> -TargetForestCredential $OldCred -MultipleExchangeDeployments $true

The -MultiplExchangeDeployments options should be set to $true since both forests contain an Exchange organization.

The Exchange Management Shell does not report anything back, so no need to show it here 😊

When we look in the AD Configuration container we can now see two SCP records:

  • One record can be found under CN=Services, CN=Microsoft Exchange, CN=<Organization>, CN=Administrative Groups, CN=Exchange Adminstrative Groups (FYDIBOHF23SPDLT), CN=Servers, CN=<ServerName>, CN=Protocols, CN=Autodiscover, CN=<ServerName>. This will contain the regular SCP information that Outlook needs to connect to the existing Exchange organization to retrieve its information.
  • The second record can be found under CN=Services, CN=Microsoft Exchange Autodiscover, CN=<FQDN of new Forest>. This will contain information regarding the target (i.e. new Exchange 2016) forest that the Outlook needs for migrated Mailboxes.
    This second record can be seen in the following screenshot:

CN=Microsoft Exchange Autodiscover

The keywords property of the SCP record contains the Accepted Domains of the new Exchange 2016 organization, like Exchangefun.nl, Corporate.Exchangefun.nl and target.qmm (the Quest target domain). This means when a new Accepted Domain is added to Exchange 2016, the Export-AutodiscoverConfig command needs to be run again.

The serviceBindingInformation property contain an LDAP link to the Exchange 2016 forest where Outlook clients can find information from the migrated Mailboxes.

Granting permissions

To avoid issues with Outlook clients of not migrated Mailboxes (that need to retrieve information from the old Exchange 2013 organization) we have to hide the exported SCP for these users. At the same time, we have to hide the original SCP record in Exchange 2013 for Mailboxes that have been migrated to Exchange 2016 (and where Outlook should NEVER receive old Exchange 2013 information).

To achieve this, create a Universal Security Group with a name like “Migrated_Users”, remove the Authenticated Users group from the exported SCP and grant the Migrated_Users Security Group Read permissions on this object as shown in the following screen shot:

Remove Authenticated Users

At the same time we have to grant an explicit deny Read permission to the Migrated_Users Security Group on the original SCP record as shown in the following screenshot:

Explicit Deny

Summary

Now when a Mailbox is migrated using QMM and the CPUU tool, add the user to the Migrated_Users Security Group. At this moment its Outlook client will no longer find the original (Exchange 2013) SCP record but the exported SCP record. Outlook will then connect to the target Active Directory forest with Exchange 2016 and retrieve the correct information.

Note. It took us quite some time when testing this scenario with different versions of Outlook (2010, 2013 and 2016) but the scenario explained here turned out to be working fine with these versions. But please test in your own test environment with various clients as well.

More information

 

SSL Certificate warning during or after Exchange server setup

When installing a new Exchange server (2013/2016/2019) in an existing environment, Microsoft recommends installing this new Exchange server in a separate Active Directory site, configure the server there and then move the server to its production Active Directory site.

The reason for this is Outlook and the Service Connection Point (SCP) in Active Directory. Somewhere during the installation process a new SCP is created in Active Directory, but when created it is not configured and points to the FQDN of the Exchange server instead of the more general Autodiscover.contoso.com/Autodiscover/Autodiscover.xml URL. When an Outlook client accidentally discovers this unconfigured SCP it will try to connect to the new server instead of the Autodiscover FQDN which will result in a certificate warning message similar to the following:

image404

To avoid this, the SCP should be configured as soon as it is created in Active Directory (and this is during setup itself).

Tony Murray, also an MVP, has written a PowerShell script (Set-AutodiscoverSCPValue.ps1) that will check the existence of the Exchange server object in Active Directory, and when it is created by the Exchange setup application, it immediately sets the correct Autodiscover value in its SCP.

When you run the script it will check every 5 seconds (time is configurable) for the newly created server object, and when it finds it, it will set the correct value as shown in the following screenshot:

set-AutodiscoverSCPValue

From this moment on Outlook client can safely discover this SCP record, and it will be automatically connected to the correct Autodiscover URL and therefore the SSL Certificate warning will not appear (assuming the original servers are configured correctly of course).

More information and download – https://gallery.technet.microsoft.com/office/set-autodiscoverserviceinte-3930e163

Exchange 2019 released and available on Microsoft Volume License Center

On Monday October 22, 2018 Microsoft has released Exchange 2019 publicly. Well… Publicly… Exchange 2019 is only available via Volume Licensing, so if you don’t have a VL agreement with Microsoft I’m afraid getting a legal version of Exchange 2019 can become a bit annoying.

How exciting is Exchange 2019? Personally, I think it’s a nice improvement compared to Exchange 2016, there are some new admin features for performance and security, and Exchange 2019 also includes some nice features for end users as well. But be aware, at this moment these features are only available in OWA and sometimes not (yet) in Outlook. But let’s have a look at Exchange 2019…

Engineering

In Exchange 2016 a Cumulative Update was released every 90 days, and this Cumulative Update was directly derived from Exchange Online. Within the Exchange product team there was only one ‘branch’ and all versions were coming from this branch.

In Exchange 2019 a separate branch is used, so within Microsoft there are now two branches. One for Exchange Online and one for Exchange 2019. This means that changes and improvements that are introduced in Exchange Online do not necessarily make it into Exchange 2019. At the same time, it is also possible that Microsoft introduces changes in Exchange 2019 that won’t make it into Exchange Online.

building-2019

Look at calendaring for example, a lot of new features are introduced recently in Exchange Online, a new look and feel etc. but these will never make it to Exchange 2019 unfortunately.

Windows Server 2019

Exchange 2019 runs on Windows Server 2019, and it only runs on Windows 2019. There are a lot of security and .NET improvement in Windows Server 2019 which Exchange 2019 uses, and these are not available in Windows 2016. So, Windows 2019 is the way forward when it comes to Exchange 2019….

Also, Windows 2019 Server Core is the recommended version of Windows 2019. Because of the lower footprint of Windows, Server Core reduces the attack surface and thus improves security of the server.

When it comes to Active Directory, the Domain Functional Level (DFL) and Forest Functional Level (FFL) should be at Windows 2012 R2 level. By enforcing this, Microsoft ensures customers are using a recent version of Windows for Domain Controllers, thus improving security of Active Directory.

From a personal perspective I’m curious about the impact of this. A lot of my customers are still running at a lower level of DFL and FFL (and thus a lower version of Windows), and also Windows 2019 might be a challenge to bring into production at this time…

Exchange 2019 Requirements

When it comes to server memory, the amount of recommended server memory is 128 GB (64 GB for Exchange 2019 Edge Transport Server). Exchange 2019 will run fine with less memory of course, but to make use of the performance improvements (in Windows 2019 and .NET Framework) 100 GB or more is needed, hence the 128 GB recommendation. The maximum amount of memory that’s supported is 256 GB.

One might argue that the minimum amount of server memory in Exchange 2016 was ‘only’ 8 GB. While this is true, Exchange 2016 won’t hardly start with 8 GB of memory, and a properly sized Exchange 2016 server also has 48 GB or 64 GB or RAM, so the step to 128 GB is not that big.

The maximum number of processor cores is now 48 (was 24 in Exchange 2016), and the amount of server memory is also related to the amount of processor cores, so less processor cores will certainly result in less memory.

The requirements calculator will be updated soon, and once released we will have more information regarding proper sizing.

If you are still interested, server virtualization is supported, but you can ask yourself if the advantages of server virtualization with these requirements outnumber the added layer of complexity (and potential hit on performance).

Metacache Database (MCDB)

The current hard disk technology is improving, and disk sizes continue to grow and will do the upcoming years. Disk I/O doesn’t grow, and a 16TB disk has approx. the same IOPS as a 4TB disk. The existing Exchange disk usage will suffer from performance issues with these larger disks, so in Exchange 2019 Microsoft has added ‘Metacache Database’ technology, where meta information from Mailbox databases is stored on SSD disks. So, an Exchange server will have JBOD disks where Mailbox databases are stored, but at the same time have SSD disks where this meta information is stored. The ratio SSD to spinning disk is 1:3, so for every spinning disk one SSD disk is used. This meta information can be mailbox information within a database, the folder structure within mailboxes, meta information about individual items or when items are small enough, even these individual items.

For a MCDB implementation, the Exchange server needs to be configured with a reseed-enabled DAG and a symmetric SSD count and size on each server (preferred architecture!), so all servers need to be completely identical.

mcdb-config

Meta information is automatically replicated from JBOD to SSD disks in a background task, so no need to worry about that. What happens if one SSD disk fails? It is a cached system, so all information will always be available in the Mailbox database, so when one SSD disk fails the server will automatically use the Mailbox database, but of course will be hit by a performance penalty.

MCDB will result is faster logon times, faster response times and less latency, something that will be extremely useful when Outlook is running online mode (in a Citrix environment).

Dynamic Database Cache

In previous versions of Exchange, database cache was automatically and equally used by all Mailbox databases, whether they were active copies or passive copies.

Exchange 2019 uses the concept of Dynamic Database Cache, where active Mailbox databases will be assigned (much) more memory than passive copies of Mailbox databases. This will result in much better performance. When a Mailbox database failover occurs, server memory is automatically re-aligned to the new situation.
Compared to Exchange 2016, the dynamic database cache will result in more memory per active mailbox database, which (again) will result in better performance.

dynamic-database

Exchange 2019 Search Engine

Exchange 2019 introduces a new search engine called ‘Big Funnel’, based on Bing technologies. Search indexes are not longer stored on disk per Mailbox database, but search indexes are now stored inside the Mailbox database on a per Mailbox basis. As such, Search Indexes are also replicated to passive copies of the Mailbox database and these are always in sync. When a corrupt page containing search index information is found, page patching occurs to copy a healthy page from a passive copy of the Mailbox database and the active copy of a Mailbox database.

Now that the content indexes are always good, issues with Mailbox database failovers no longer occur, and Mailbox database failover times should decrease dramatically.
Combined with the new MCDB and dynamic database cache search results should be increase dramatically, which is again especially interesting for Outlook running in online mode.

Client improvements

Exchange 2019 also comes with some client improvements but be aware that there’s a difference in client. OWA client behaves differently than an Outlook client on Windows, which is again differs from an Outlook client on Mac…

Recurring meetings with default end date

When creating a recurring appointment there’s a default end date on the appointments. This will prevent your users creating a recurring appointment without an end date, which is annoying when the user leaves the organization ending in a orphaned recurring meeting.

default-end-date

Block calendar during OOF

Very nice improvement in Exchange 2019 is the possibility to block your calendar when setting your Out-of-Office during this period. Besides blocking, you can also select to automatically decline new meeting requests or decline new requests and cancel existing meetings in your OOF time. Very useful!

block-calendar

Remove calendar events

One annoying issue with Exchange is that an Exchange admin cannot remove calendar items created by users. This is an issue when a recurring or future appointment is created by a user that has left the organization, and the meeting is an orphaned meeting.
Exchange 2019 introduces the Remove-CalendarEvents cmdlet, which makes it possible to remove future calendar items created by users.

The following example will remove all meetings created from this day forward for user Chris:

Remove-CalendarEvents -Identity chris@contoso.com -CancelOrganizedMeetings

The next example will remove meetings created by Kim Akers starting on November 1, 2018 for a period of 120 days. This is useful when Kim is on leave for a 4 months period:

Remove-CalendarEvents -Identity “Kim Akers” -CancelOrganizedMeetings -QueryStartDate 11-1-2018 -QueryWindowInDays 120

Email Address Internationalization

Email Address Internationalization (EAI) was already supported in Exchange Online, but now it has arrived in Exchange 2019 as well.

Wat is EAI? As Microsoft announced on the Exchange Team blog “Out of 7.6 billion people in the world, only 360 million are native English speakers” so lots of people don’t even use the character set that we are using currently in Exchange. There are a lot of character sets that are not supported in Exchange.

For example, I’m working with a European organization that acquired a company in Colombia. Importing these users into the existing Active Directory was quite a challenge, because they are using a Latin alphabet with diacritic. Before importing we had to convert all names and email addresses into our regular character set.

Exchange 2019 now supports EAI, and you can send email to users with an EAI address. Unfortunately, you cannot add an EAI proxy or Accepted Domain yet.

Examples of EAI email addresses are:

  • Latin alphabet (with diacritics): Pelé@example.com
  • Greek alphabet: δοκιμή@παράδειγμα.δοκιμή
  • Traditional Chinese characters: 我買@屋企.香港
  • Japanese characters: 甲斐@黒川.日本
  • Cyrillic characters: чебурашка@ящик-с-апельсинами.рф
  • Hindi email address: संपर्क@डाटामेल.भारत

For more information regarding Email Address Internationalization (in Office 365) please check Jeff Guillet’s blog on http://www.expta.com/2017/12/email-address-internalization-eai.html.

Unified Messaging

In Exchange 2019 the Unified Messaging server role is completely removed, so Exchange 2019 does not offer voice mail or auto attendant processing.

You can use the Microsoft cloud voice mail option or a 3rd party PBX solution that can record voicemail messages and have these sent by SMTP to your Exchange 2019 mailbox, but that’s not really a viable replacement I would say.

When moving UM enabled mailboxes from Exchange 2013 or Exchange 2016 to Exchange 2019, these mailboxes will automatically be UM disabled, but existing voicemail messages will remain in the user’s mailbox of course.

The options you have as a UM customer are:

  • Move all users and mailboxes to Office 365.
  • Migrate to Skype for Business Server 2019.
  • Remain on Exchange Server 2016 (supported until 2023).
  • Deploy 3rd party voice mail solution options

Summary

Exchange 2019 was released by Microsoft on October 22nd, 2018 and is the latest version of Exchange Server, targeted toward (large) enterprise customers that cannot move to the cloud (yet). As it is targeted towards enterprise customers, Exchange 2019 is only available via Volume Licensing only. This is not only true for the initial version of Exchange 2019, future CU’s will be made available via VL only as well.

There are a lot of new features in Exchange 2019 like the dynamic database cache and metacache database. Combined with the new hardware requirements and SSD disk this will certainly improve performance, and Outlook clients running in Online Mode (I say Citrix) will benefit from this.

Some new client features as well, like the possibility to block forwarding of meeting requests, block invites during Out-Of-Office and support for Email Address Internationalization.

Time to start playing with Exchange 2019, find out what the true benefits are, and hopefully blog about it in the near future 😊

More information

During Ignite 2018 in Orlando there were a few presentation on Exchange 2019 that are certainly worth having a look at: