Category Archives: Exchange

SSL Certificate warning during or after Exchange server setup

When installing a new Exchange server (2013/2016/2019) in an existing environment, Microsoft recommends installing this new Exchange server in a separate Active Directory site, configure the server there and then move the server to its production Active Directory site.

The reason for this is Outlook and the Service Connection Point (SCP) in Active Directory. Somewhere during the installation process a new SCP is created in Active Directory, but when created it is not configured and points to the FQDN of the Exchange server instead of the more general Autodiscover.contoso.com/Autodiscover/Autodiscover.xml URL. When an Outlook client accidentally discovers this unconfigured SCP it will try to connect to the new server instead of the Autodiscover FQDN which will result in a certificate warning message similar to the following:

image404

To avoid this, the SCP should be configured as soon as it is created in Active Directory (and this is during setup itself).

Tony Murray, also an MVP, has written a PowerShell script (Set-AutodiscoverSCPValue.ps1) that will check the existence of the Exchange server object in Active Directory, and when it is created by the Exchange setup application, it immediately sets the correct Autodiscover value in its SCP.

When you run the script it will check every 5 seconds (time is configurable) for the newly created server object, and when it finds it, it will set the correct value as shown in the following screenshot:

set-AutodiscoverSCPValue

From this moment on Outlook client can safely discover this SCP record, and it will be automatically connected to the correct Autodiscover URL and therefore the SSL Certificate warning will not appear (assuming the original servers are configured correctly of course).

More information and download – https://gallery.technet.microsoft.com/office/set-autodiscoverserviceinte-3930e163

Exchange 2019 released and available on Microsoft Volume License Center

On Monday October 22, 2018 Microsoft has released Exchange 2019 publicly. Well… Publicly… Exchange 2019 is only available via Volume Licensing, so if you don’t have a VL agreement with Microsoft I’m afraid getting a legal version of Exchange 2019 can become a bit annoying.

How exciting is Exchange 2019? Personally, I think it’s a nice improvement compared to Exchange 2016, there are some new admin features for performance and security, and Exchange 2019 also includes some nice features for end users as well. But be aware, at this moment these features are only available in OWA and sometimes not (yet) in Outlook. But let’s have a look at Exchange 2019…

Engineering

In Exchange 2016 a Cumulative Update was released every 90 days, and this Cumulative Update was directly derived from Exchange Online. Within the Exchange product team there was only one ‘branch’ and all versions were coming from this branch.

In Exchange 2019 a separate branch is used, so within Microsoft there are now two branches. One for Exchange Online and one for Exchange 2019. This means that changes and improvements that are introduced in Exchange Online do not necessarily make it into Exchange 2019. At the same time, it is also possible that Microsoft introduces changes in Exchange 2019 that won’t make it into Exchange Online.

building-2019

Look at calendaring for example, a lot of new features are introduced recently in Exchange Online, a new look and feel etc. but these will never make it to Exchange 2019 unfortunately.

Windows Server 2019

Exchange 2019 runs on Windows Server 2019, and it only runs on Windows 2019. There are a lot of security and .NET improvement in Windows Server 2019 which Exchange 2019 uses, and these are not available in Windows 2016. So, Windows 2019 is the way forward when it comes to Exchange 2019….

Also, Windows 2019 Server Core is the recommended version of Windows 2019. Because of the lower footprint of Windows, Server Core reduces the attack surface and thus improves security of the server.

When it comes to Active Directory, the Domain Functional Level (DFL) and Forest Functional Level (FFL) should be at Windows 2012 R2 level. By enforcing this, Microsoft ensures customers are using a recent version of Windows for Domain Controllers, thus improving security of Active Directory.

From a personal perspective I’m curious about the impact of this. A lot of my customers are still running at a lower level of DFL and FFL (and thus a lower version of Windows), and also Windows 2019 might be a challenge to bring into production at this time…

Exchange 2019 Requirements

When it comes to server memory, the amount of recommended server memory is 128 GB (64 GB for Exchange 2019 Edge Transport Server). Exchange 2019 will run fine with less memory of course, but to make use of the performance improvements (in Windows 2019 and .NET Framework) 100 GB or more is needed, hence the 128 GB recommendation. The maximum amount of memory that’s supported is 256 GB.

One might argue that the minimum amount of server memory in Exchange 2016 was ‘only’ 8 GB. While this is true, Exchange 2016 won’t hardly start with 8 GB of memory, and a properly sized Exchange 2016 server also has 48 GB or 64 GB or RAM, so the step to 128 GB is not that big.

The maximum number of processor cores is now 48 (was 24 in Exchange 2016), and the amount of server memory is also related to the amount of processor cores, so less processor cores will certainly result in less memory.

The requirements calculator will be updated soon, and once released we will have more information regarding proper sizing.

If you are still interested, server virtualization is supported, but you can ask yourself if the advantages of server virtualization with these requirements outnumber the added layer of complexity (and potential hit on performance).

Metacache Database (MCDB)

The current hard disk technology is improving, and disk sizes continue to grow and will do the upcoming years. Disk I/O doesn’t grow, and a 16TB disk has approx. the same IOPS as a 4TB disk. The existing Exchange disk usage will suffer from performance issues with these larger disks, so in Exchange 2019 Microsoft has added ‘Metacache Database’ technology, where meta information from Mailbox databases is stored on SSD disks. So, an Exchange server will have JBOD disks where Mailbox databases are stored, but at the same time have SSD disks where this meta information is stored. The ratio SSD to spinning disk is 1:3, so for every spinning disk one SSD disk is used. This meta information can be mailbox information within a database, the folder structure within mailboxes, meta information about individual items or when items are small enough, even these individual items.

For a MCDB implementation, the Exchange server needs to be configured with a reseed-enabled DAG and a symmetric SSD count and size on each server (preferred architecture!), so all servers need to be completely identical.

mcdb-config

Meta information is automatically replicated from JBOD to SSD disks in a background task, so no need to worry about that. What happens if one SSD disk fails? It is a cached system, so all information will always be available in the Mailbox database, so when one SSD disk fails the server will automatically use the Mailbox database, but of course will be hit by a performance penalty.

MCDB will result is faster logon times, faster response times and less latency, something that will be extremely useful when Outlook is running online mode (in a Citrix environment).

Dynamic Database Cache

In previous versions of Exchange, database cache was automatically and equally used by all Mailbox databases, whether they were active copies or passive copies.

Exchange 2019 uses the concept of Dynamic Database Cache, where active Mailbox databases will be assigned (much) more memory than passive copies of Mailbox databases. This will result in much better performance. When a Mailbox database failover occurs, server memory is automatically re-aligned to the new situation.
Compared to Exchange 2016, the dynamic database cache will result in more memory per active mailbox database, which (again) will result in better performance.

dynamic-database

Exchange 2019 Search Engine

Exchange 2019 introduces a new search engine called ‘Big Funnel’, based on Bing technologies. Search indexes are not longer stored on disk per Mailbox database, but search indexes are now stored inside the Mailbox database on a per Mailbox basis. As such, Search Indexes are also replicated to passive copies of the Mailbox database and these are always in sync. When a corrupt page containing search index information is found, page patching occurs to copy a healthy page from a passive copy of the Mailbox database and the active copy of a Mailbox database.

Now that the content indexes are always good, issues with Mailbox database failovers no longer occur, and Mailbox database failover times should decrease dramatically.
Combined with the new MCDB and dynamic database cache search results should be increase dramatically, which is again especially interesting for Outlook running in online mode.

Client improvements

Exchange 2019 also comes with some client improvements but be aware that there’s a difference in client. OWA client behaves differently than an Outlook client on Windows, which is again differs from an Outlook client on Mac…

Recurring meetings with default end date

When creating a recurring appointment there’s a default end date on the appointments. This will prevent your users creating a recurring appointment without an end date, which is annoying when the user leaves the organization ending in a orphaned recurring meeting.

default-end-date

Block calendar during OOF

Very nice improvement in Exchange 2019 is the possibility to block your calendar when setting your Out-of-Office during this period. Besides blocking, you can also select to automatically decline new meeting requests or decline new requests and cancel existing meetings in your OOF time. Very useful!

block-calendar

Remove calendar events

One annoying issue with Exchange is that an Exchange admin cannot remove calendar items created by users. This is an issue when a recurring or future appointment is created by a user that has left the organization, and the meeting is an orphaned meeting.
Exchange 2019 introduces the Remove-CalendarEvents cmdlet, which makes it possible to remove future calendar items created by users.

The following example will remove all meetings created from this day forward for user Chris:

Remove-CalendarEvents -Identity chris@contoso.com -CancelOrganizedMeetings

The next example will remove meetings created by Kim Akers starting on November 1, 2018 for a period of 120 days. This is useful when Kim is on leave for a 4 months period:

Remove-CalendarEvents -Identity “Kim Akers” -CancelOrganizedMeetings -QueryStartDate 11-1-2018 -QueryWindowInDays 120

Email Address Internationalization

Email Address Internationalization (EAI) was already supported in Exchange Online, but now it has arrived in Exchange 2019 as well.

Wat is EAI? As Microsoft announced on the Exchange Team blog “Out of 7.6 billion people in the world, only 360 million are native English speakers” so lots of people don’t even use the character set that we are using currently in Exchange. There are a lot of character sets that are not supported in Exchange.

For example, I’m working with a European organization that acquired a company in Colombia. Importing these users into the existing Active Directory was quite a challenge, because they are using a Latin alphabet with diacritic. Before importing we had to convert all names and email addresses into our regular character set.

Exchange 2019 now supports EAI, and you can send email to users with an EAI address. Unfortunately, you cannot add an EAI proxy or Accepted Domain yet.

Examples of EAI email addresses are:

  • Latin alphabet (with diacritics): Pelé@example.com
  • Greek alphabet: δοκιμή@παράδειγμα.δοκιμή
  • Traditional Chinese characters: 我買@屋企.香港
  • Japanese characters: 甲斐@黒川.日本
  • Cyrillic characters: чебурашка@ящик-с-апельсинами.рф
  • Hindi email address: संपर्क@डाटामेल.भारत

For more information regarding Email Address Internationalization (in Office 365) please check Jeff Guillet’s blog on http://www.expta.com/2017/12/email-address-internalization-eai.html.

Unified Messaging

In Exchange 2019 the Unified Messaging server role is completely removed, so Exchange 2019 does not offer voice mail or auto attendant processing.

You can use the Microsoft cloud voice mail option or a 3rd party PBX solution that can record voicemail messages and have these sent by SMTP to your Exchange 2019 mailbox, but that’s not really a viable replacement I would say.

When moving UM enabled mailboxes from Exchange 2013 or Exchange 2016 to Exchange 2019, these mailboxes will automatically be UM disabled, but existing voicemail messages will remain in the user’s mailbox of course.

The options you have as a UM customer are:

  • Move all users and mailboxes to Office 365.
  • Migrate to Skype for Business Server 2019.
  • Remain on Exchange Server 2016 (supported until 2023).
  • Deploy 3rd party voice mail solution options

Summary

Exchange 2019 was released by Microsoft on October 22nd, 2018 and is the latest version of Exchange Server, targeted toward (large) enterprise customers that cannot move to the cloud (yet). As it is targeted towards enterprise customers, Exchange 2019 is only available via Volume Licensing only. This is not only true for the initial version of Exchange 2019, future CU’s will be made available via VL only as well.

There are a lot of new features in Exchange 2019 like the dynamic database cache and metacache database. Combined with the new hardware requirements and SSD disk this will certainly improve performance, and Outlook clients running in Online Mode (I say Citrix) will benefit from this.

Some new client features as well, like the possibility to block forwarding of meeting requests, block invites during Out-Of-Office and support for Email Address Internationalization.

Time to start playing with Exchange 2019, find out what the true benefits are, and hopefully blog about it in the near future 😊

More information

During Ignite 2018 in Orlando there were a few presentation on Exchange 2019 that are certainly worth having a look at:

 

Exchange 2016 Cumulative Update 11

Most likely you’ve seen this information before, because of my vacation in Dallas and New Orleans I’m a bit behind with blogging 😊

But on October 16, 2018 Microsoft has released Cumulative Update 11 (CU11) for Exchange 2016, this is a little later than expected to align the release of Exchange 2016 Cumulative Updates with the upcoming release of Exchange 2019. . There’s only a release for Exchange 2016, there won’t be any new CU’s for Exchange 2013 since Exchange 2013 is already in extended support. There will be security updates for Exchange 2013 though.

Exchange server and .NET Framework is not a happy marriage and it continues to be a struggle, or at least it looks that way. Exchange 2016 CU11 now supports .NET Framework 4.7.2. This version of .NET Framework is not mandatory, installation of .NET Framework 4.7.2 can be before installing of CU11 or after CU11. The .NET Framework 4.7.2 will be required for a future CU of Exchange 2016.

Another dependency is Visual C++, you might have seen this in previous CU’s and also in Exchange 2010 Update Rollup 23 as well. To avoid any issue, install Visual C++ 2012 (https://www.microsoft.com/download/details.aspx?id=30679) before installing Exchange 2016 CU11.

Exchange 2016 CU11 does not have any schema changes. If you’re upgrading from an older version of Exchange 2016, Active Directory changes (in the configuration container) might be needed. These will automatically be applied by the setup application, but you can also choose to update the configuration partition manually by running setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

As always, you should test a Cumulative Update thoroughly before bringing it to production, it won’t be the first time something goes wrong in production with a CU. But I have to say, I haven’t seen any major blocking issues so far…

More information and downloads of Exchange 2016 CU11:

Exchange 2010 and TLS 1.2

In a previous blogpost I discussed an issue I had with Outlook 2010 and TLS 1.2. At the same time this reminded me that Microsoft will remove support for TLS 1.0 and TLS 1.1 in Office 365 on October 31, 2018 as communicated in https://support.microsoft.com/en-us/help/4057306/preparing-for-tls-1-2-in-office-365. This means that when you have communication issues with Office 365 because of an older and weaker protocol, you won’t get any support. Time to do some research….

Existing Exchange 2010 environment

As you may have seen on this side, I still am a big fan of Exchange 2010 and also have an pure Exchange 2010 hybrid environment up-and-running and it looks like this:

Inframan-hybrid

MX records is pointing to my Exchange 2010 Edge Transport Server (running on Windows 2008 R2), webmail and Autodiscover are routed via an F5 LTM load balancer to an Exchange 2010 CAS/HUB/Mailbox server (also running on Windows 2008 R2), and hybrid is configured directly on Exchange 2010 (for hybrid mail flow I’m using a separate FQDN, o365mail.inframan.nl) without any Exchange 2013 or Exchange 2016 server.

So, how do you test which TLS version is used by your Exchange 2010 server? In Exchange 2010 this should be done using the protocol logfiles. Message headers in Exchange 2010 do not contain enough information for showing this TLS information. So, you must enable protocol logging for the appropriate Receive Connectors and Send Connectors. In my environment this means the Default Receive Connector on the Exchange 2010 Edge Transport server (for O365 traffic from other tenants), the Default-First-Site-Name to Internet Send Connector, and both connectors between the Exchange 2010 server and Office 365 for hybrid. Analyzing the protocol logfiles can best be done in Excel (import as CSV files). When analyzing, look for a string like TLS protocol SP_PROT_TLS1_0_SERVER (when receiving) or TLS protocol SP_PROT-TLS1_0_CLIENT (when sending). When TLS 1.2 is used, look for a string like TLS protocol SP_PROT_TLS1_2_SERVER and TLS protocol SP_PROT-TLS1_2_CLIENT.

Continue reading Exchange 2010 and TLS 1.2

Ignite 2018 – Exchange sessions, the good and a bit ugly

My first blog about Ignite 2018 was more about the keynotes on the first day, Microsoft’s strategy around the cloud and how various application integrate with each other and take advantage of the cloud. But I’m a technical consultant and more interested in the technical stuff. And as a MVP in ‘Office Apps and Service’  (used to be ‘Office Servers and Services’, and before that just ‘Exchange MVP’) my heart is still with Exchange. Although I also attended lots of other sessions, there are better blogs available for these technologies instead of mine 😊

Welcome to Exchange 2019

The first break-out session I attended was BRK “Welcome to Exchange 2019” by Greg Taylor and Brent Alinger. Lots of information was already available since Microsoft released the preview version of Exchange 2019, but some other interesting points were mentioned as well:

  • Exchange 2019 runs on Windows 2019 only. There are so much security features in Windows 2019 that Exchange is using, features that are not available in Windows 2016. The preview version was running on Windows 2016, but the final version won’t.
  • Windows 2019 Server Core is the recommended platform because of the lower footprint and attack surface.
  • Required Forest Functional Level is Windows 2012 R2, which may cause issues with customers I guess.
  • Minimum recommended RAM is 128 GB. Be aware, this is the recommended amount of RAM, not the required amount of RAM. This amount comes from .NET usage in Windows 2019 that does much better performance with lots amount of memory. If an Exchange 2019 server does not have 100GB or more, it won’t take advantage of a lot of these improvements. There’s also a correlation with the amount of processors in the Exchange 2019 server, and this 128 GB is related to 48 processors. If you are using less processors, the memory usage decreases as well.
  • Exchange 2019 will (at least for now) only be available via Volume Licensing. Discussions are going on whether this will lead to piracy via download sites. Microsoft is aware of this, but at this moment it’s only Volume Licensing.
  • A very cool new feature is the MCDB, or metacache database. This is a cache stored on SSD drives where metadata from the Mailbox databases is stored, like search information, folder structure etc. This will improve performance for Outlook clients running in Online Mode, not only search information, but also logon to the Mailbox is dramatically improved (I’m starting to sound like a marketing guy 😊
  • Related to the previous bullet, Search is improved (or rewritten actually) using Bing technology. The indexing information is no longer stored in a separate directory on the disk, but it is stored in the Mailbox, and thus inside the Database. This means that passive copies have the same information, and the same search indexes. A failover will now never fail because search is not healthy in replication, speeding up fail-over times.
  • And Microsoft also released the documentation for Exchange 2019, which can be found on https://docs.microsoft.com/en-us/Exchange/exchange-server?view=exchserver-2019.

Unfortunately this session is not available online yet. It is recorded, but somehow not yet available.

As a side note, Microsoft has organized a side meeting with the Exchange and Outlook Product Groups and a couple of (Exchange) MVPs. In this 2 hour session we could have a decent discussion with all Program Managers in the room, and we were able to express our deepest concerns regarding the announcements and presentations at Ignite. Some people prefer to do this on Twitter, but we think it’s better to discuss with the Program Managers directly. They gave us additional background information, but at the same time were impressed by the feedback we gave them, and will take it back to HQ. They won’t change the product of course, but the marking messeag and documentation (background information) around Exchange 2019 will certainly change.

Securing Exchange Online from modern threats

Another interesting session was BRK3148 “Securing Exchange Online from modern threats”. It all about security, and what steps the bad guys usually take to attach a platform like Exchange Online. And it’s incredibly easy. Every heard of ‘password spray’? This a brute force attack, but the other way around. The bad guy has a list of usernames (UPN = Email address to make their life easy) and standard passwords like Summer2018 or Spring2018. But with a spray attack they take a password, and try this against all users. If not successful then try the next password against all users in the list. Incredibly simple, and unlike a regular brute force attack this does not result in locked out accounts. And we all know it, it’s a matter of little time before a simple password is compromised. In the demo the presenter shows how easy it is, and once logged on how to continue with elevated privileges.

The good news is, this presentation is available on YouTube: https://youtu.be/jnUUioUU_eY

Hybrid Exchange: making it easier and faster to move to the cloud

Exchange hybrid is also a hot topic. Last year Jeff Kizner did a session on hybrid, and announced Microsoft was working on removing the last Exchange server when all Mailboxes are moved to the cloud. Expectations were high when attending this years session BRK3143 “Hybrid Exchange: making it easier and faster to move to the cloud”.

The first part of this presentation is about the work done on the Organization Configuration Transfer in the Hybrid Configuration Wizard. Still not finished, but a lot of the configuration information cannot be copied over to Exchange Online. It is copying to Exchange Online, there’s no synchronization. So when making changes in Exchange on-premises, they are not transferred automatically to Exchange Online. You have to either make the changes in Exchange Online, or run the Hybrid Configuration Wizard again.

Completely new (and not available yet) is the Hybrid Agent that runs on-premises. The hybrid agent works with an endpoint in Microsoft Azure, and is outbound HTTPS traffic only. Exchange Online is configured with the HCW to use the same endpoint in Microsoft Azure. This way only outbound connections are used, and it is no longer needed to make all kinds of firewall changes when configuring Exchange Hybrid. Even better, when Autodiscover and Exchange on-premises are not published this still works, since only Outbound connections are used, and configuration information is stored safely in the endpoint in Microsoft Azure.

Exchange Hybrid Agent

TargetSharingEpr

At this moment it is only going to work with Free/Busy and Mailbox moves using the MRS, but it’s a good start. Next versions can include more features, and using this technique everything is possible, imaging Microsoft Search that’s searching your on-premises Mailbox servers…. I’m not sure if that’s a good idea, and I have an idea how the average security officer will react to a solution like this. Some people will refer to this as a man in the middle that has full access to your Exchange environment (something with Exchange Trusted Subsystem). Also, the Hybrid Agent only supports auto-update, and I’m not sure if I want that to happen on my Exchange servers. The good news, you can run the Agent on dedicated servers instead of the Exchange servers, as long as these servers have a decent Internet connection.

The Exchange Product Group released a blogpost on both topics, which can be found on https://blogs.technet.microsoft.com/exchange/2018/09/26/announced-improvements-to-hybrid-publishing-and-organization-configuration-transfer/.

Also, the presentation itself is available on YouTube: https://youtu.be/QhOh5RCcLu8

Unfortunately not a single word about that last Exchange server on-premises, so at this point this will need to be available for some time I’m afraid.

Email search in a flash! Accelerating Exchange 2019 with SSDs

As already mentioned in the first section of this blog, Microsoft introduced Metacache Database (MCDB) in Exchange 2019. Exchange 2019 will now work in the regular JBOD solutions, but now added are SSD disks in the servers. These SSD disks are used as an additional cache (special data from the mailbox database is stored additionally on the SSD disks) to speed up performance. Think about the message table, mailbox information, message metadata information, all kinds of information that’s regularly needed by Outlook clients, and can now be retrieved by the Exchange 2019 at a much faster pace.

Personally I think this is a very impressive technology, but I don’t see it appear at customers anytime soon. It is build on top of a DAG (should be no issue), but is also using AutoReseed as outlined in the Preferred Architecture. The SSD versus spinning disks ration is 1:3, so for a 12 disk Exchange server, three SSD disks are needed. Now, I don’t see a lot of customers deploying Exchange 2019 this way, at least not the smaller organizations, but maybe these customers are better off with Exchange Online, but that’s a different discussion.

metacache database

metacache guidance

The technology is impressive, and I’m looking forward to test this is a lab. Unfortunately this feature is not yet available in the preview version of Exchange, so you have to wait until the official release of Exchange 2019 later this year.

The presentation is available on YouTube: https://youtu.be/VHrScskhCQk

Stay tuned for more information…