Tag Archives: Exchange

Azure, Exchange

Exchange Server in Azure – Part III

Leave a comment

In two previous post I blogged about a site-to-site VPN connection to Azure and about installing a Domain Controller (and Azure AD Connect) in Azure. In this blog I will discuss that last server, the Exchange 2019 server in Azure.

The last in most interesting part (in my opinion) is installing an Exchange 2019 server in Azure. This can be your last Exchange server for management purposes, or maybe an Exchange server still hosting (some) mailboxes or performing SMTP Relay. It can be quite expensive to host mailboxes on an Exchange server in Azure, moving Mailboxes to Exchange Online is much cheaper. But then…. If you are already decommissioning your own datacenter… All business requirements are different of course 🙂

Exchange in Azure Setup

Currently Exchange server are located on-premises in a hybrid scenario. There’s already a Domain Controller in Azure, and the Azure AD Connect is running in Azure too. Now I want to install an Exchange 2019 server in Azure and run the Hybrid Configuration Wizard to create a hybrid configuration with the Exchange 2019 server in Azure. Webmail and Autodiscover will point to the Exchange server in Azure as well. In short, the configuration will look something like this:

Inbound mail will be sent to Exchange Online Protection and delivered to mailboxes in Exchange Online. When a mailbox is located on-premises, it will be routed to the Exchange server in Azure and when needed, routed to the Exchange server on-premises.

But first we have to think about sizing the Exchange server in Azure.

Sizing Exchange in Azure

The first question about Exchange server in Azure is sizing the Exchange server. First of all, deploying Exchange on Microsoft Azure virtual machines is supported if all storage volumes used for Exchange databases and database transaction logs (including transport databases) are configured for Azure Premium Storage (please see the Exchange Server Virtualization article on Microsoft Learn.

When you are planning to host active mailboxes on the Exchange server running in Azure then you should use the Exchange 2019 storage requirements calculator to figure out the correct size for the Exchange server. You can find the storage calculator on the Exchange 2019 ISO images in the \Support directory.

If you are planning to install an Exchange server in Azure just for management purposes, or for lab purposes, I would recommend using a ‘Standard d8s v3’ VM. This is a VM with 8 vCPU, 32 GB memory and a maximum of 16 additional disks.
When you create a new VM in the Azure Portal and select ‘see all sizes’ it will show a list of most used virtual machines by Azure users as shown below:

For my own lab environment I selected the DS3_V2 VM. Not the fastest VM, but for my own test- and management purposes it is sufficient, but your mileage may vary of course. Attached to my VM are two SSD disks of each 1TB. Be aware that the disks can be a costly thing, especially in a lab environment! The network interface is connected to the internal VNET that was created earlier, and it is connected to the Microsoft network using a public IP address. I also created a DNS name (exchlabsnl.westeurope.cloudapp.azure.com which can be used as a CNAME record for some other DNS name) but that’s not really needed since I can use the public IP address in a regular DNS A record.

Installing and Configuring Exchange in Azure

When you have the VM up and running in Azure and you have configured the storage as required, you can install the Exchange server. This is not different from a regular Exchange server. Install the prerequisite software, patch the machine, install the latest Exchange 2019 Cumulative Update and install the latest security update for this CU.

Configuring the Exchange server in Azure is similar to configuring the Exchange server on-premises. Install a proper certificate, configure the virtual directories (hopefully needless to say, but use a different FQDN in Azure since it is a different site in Active Directory) and configure the transport database on a different disk.

Configure the Network Security Group (NSG) to allow access on port 443 and port 25 from the Internet. Depending on your security requirements, you can configure the NSG of the Exchange server with the IP ranges of Exchange Online (https://learn.microsoft.com/en-gb/microsoft-365/enterprise/urls-and-ip-address-ranges) so only Microsoft can fully access your Exchange server in Azure. If you have clients that need to connect to the Exchange server, they can use a VPN connection (that’s what I typically recommend to customers these days, and what I do in my lab environment).

If you are planning an Exchange server just for management purposes, I would recommend not using a public IP at all and only connect to the Exchange server using the private Vnet. In this scenario there’s no need to publish the Exchange server to the Internet, and all changes to recipients are replicated to Exchange Online using Azure AD Connect. No Internet connection is less prone to external attacks of course. And another interesting thing when using a management server only, you can turn it off when not in use. This can save you quite some money.

The next step is to run the Hybrid Configuration Wizard which you can download from https://aka.ms/hybridwizard. When going through the wizard, select the Exchange server in Azure for the Receive Connector configuration and for the Send Connector configuration.

When finished, add the public IP address of your Exchange server in Azure to your SPF record for outbound mail, or configure the SMTP connector to Exchange Online to route all outbound mail via Exchange Online, that’s up to your requirements of course.

What happened in my scenario, my public IP address was assigned by Microsoft and you never know what happened previously with this IP address, but I had to delist it from SpamHaus before I was able to send out email from my Exchange server.

Then move your resources from the on-premises Exchange server to the Exchange server in Azure and decommission your Exchange server on-premises when needed.

Summary

In the past three blogpost I explained what it takes to install an Exchange server in Microsoft Azure and it contains of the following steps:

  • Create a site-to-site connection between your on-premises network and a Vnet in Azure.
  • Install a Domain Controller in Azure, connected to the Vnet. Optionally you can install an Azure AD Connect server in Azure as well.
  • Install and configure an Exchange server in Azure, connected to the Vnet (and to the Internet when needed)

An Exchange server in Azure is fully supported, as long as the Mailbox database and transaction logfiles are located on premium storage, that should be no problem.

Sizing your Exchange server in Azure is like an on-premises Exchange server. Use the storage calculator to determine the size of your Exchange server, and look for a VM that matches these requirements.

When installing an Exchange server in Azure just for management purposes it’s much easier. You can use a relatively lightweight Exchange server, there’s no need to publish it to the Internet (much safer) and you can turn it off when not in use.

Azure, Exchange

Exchange Server in Azure – Part II

Leave a comment

In my previous blog I wrote about creating a virtual network in Microsoft Azure and a site-to-site VPN connection to connect your on-premises network to the virtual network in Azure. The next step is to create a Domain Controller and (optional) an Azure AD Connect server in Microsoft Azure.

On a Domain Controller on-premises, create a new site in Active Directory and a new subnet and assign it to the new site. In my environment, the new site is called ‘Azure’ and the new subnet is ‘172.16.1.0/24’ as shown in the following screenshot:

Adding a new VM to Azure is not too difficult, but you must be careful with change the network properties. Do not change the DNS setting on the NIC inside the VM, but change it on the properties of the networking interface object in Azure.

Open the properties of the Virtual Machine in Azure, scroll down to Networking (under Settings) and click on the Network Interface. Select DNS Servers (under Settings) and add the (AD Integrated) DNS server on-premises shown in the following screenshot:

It takes some time before the new settings are active. Once active, you should be able to resolve other machines on the on-premises network and more important, join the new VM to Active Directory. And once joined, install the ADDS components on the new VM and promote it to a new Domain Controller, running in Azure.

Another thing I have changed in Azure is the DNS setting of the virtual network. Open the virtual network properties and scroll down to DNS Servers (again under settings). Open the DNS servers properties, select the ‘custom’ radio button and add the IP address of the new Domain Controller (with Integrated DNS of course). All VM’s on this virtual network now use the new Domain Controller for name resolution.

For external recurring DNS queries, and the Azure forwarder with IP address 168.63.129.16 on the forwarders tab of the DNS server as shown in the following screenshot:

The next step is to create a VM that will host the Azure AD Connect server. It still is my test environment so VMs don’t have to be that big, so for the Azure AD Connect server I’m using the Standard DS1 v2 (1 vcpu, 3.5 GiB memory) as well. You can also install Azure AD Connect on the Domain Controller in Azure, but personally I’m not a big fan of installing other services on Domain Controllers.

Make sure the new VM is connected to the Virtual Network that was created earlier, it will automatically get an IP address in the correct range.

Once installed, logon to the new server, join it to the domain and reboot again, just you would do with a server on-premises.
Since we are working post migration to Exchange Online we do have an Azure AD Connect server running on-premises. Moving this server to Azure is just like upgrading an Azure AD Connect server. Export the configuration of the existing server to a JSON file, and import this JSON file using the Azure AD Connect setup application. I have blogged about this process in an earlier upgrade blog: https://jaapwesselius.com/2021/12/24/upgrade-azure-ad-connect-from-1-x-to-2-x/ but it’s similar to moving to Azure.

When using this method, the new server will automatically be in ‘staging mode’, so it will fill its database with information, but it won’t sync anything with Azure AD. When you are ready, set the old server in staging mode and get the new server out of staging mode. At this point you are synchronizing using Azure AD Connect in Azure instead of on-premises.

Azure, Exchange

Exchange server in Azure Part I

1 Comment

As an Exchange consultant I still do a lot of work with Exchange server on-premises, like migrations to Exchange 2019 and hybrid migrations to Exchange Online. In a typical Exchange hybrid environment 99% of all mailboxes are in Exchange Online, and only a handful of mailboxes are in Exchange on-premises. As a result, only one or two Exchange servers are left on-premises; for management purposes, for SMTP routing purposes and host these leftover mailboxes. A logical follow-up question is “why not put them in Azure?”.

Placing the last Exchange server in Azure can be costly, but a lot of customers are working hard to decommission their on-premises server room, so from that point of view it’s a valid question.

Before going into detail about this Exchange server in Azure, a couple of other things need to be in place:

  1. Site-to-Site VPN connection between on-premises and Azure.
  2. Domain Controller in Azure (optional: Azure AD Connect in Azure).
  3. Exchange server in Azure.

These make a couple of interesting blogs 🙂

Site-to-Site VPN with Ubiquity EdgeRouter

The first step is to create a site-to-site VPN connection between the internal network and Microsoft Azure, and we begin with a virtual network in Microsoft Azure.

The internal network is 10.83.4.0/24 and the Azure Virtual Network is 172.16.0.0/22, separated in a gateway subnet (172.16.0.0/24) and a server network (172.16.1.0/24).

My configuration looks like this (but my external IP addresses are different):

These are the steps I executed in the Microsoft Azure Portal:

  1. Create a new Azure Virtual Network:
    • Name: ServerNetwork
    • Address Space: 172.16.0.0/22
    • Subnet name: Default
    • Subnet Address Space: 172.16.1.0/24
    • Resource Group: MyTestRG
  2. Create a Gateway Subnet (under Virtual Networks, select Subnets and click +Gateway Subnet. The properties of the Gateway Subnet are automatically populated.
    • Name: GatewaySubnet
    • Address space: 172.16.0.0/24
  3. Create a new Azure Virtual Network Gateway:
    • Name: VirtualGateway
    • Gateway type: VPN
    • VPN type: Route-Based
    • SKU: Basic
    • Basic is using a dynamic external IP address. Static IP Address is only available in better (ie more expensive) versions. More info on https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways
    • Virtual Network: ServerNetwork
    • Public IP Address: Create new during creation of VirtualGateway
    • Public IP Name: ..cloudapp.azure.com
    • Public IP Address SKU (dynamic)
    • The creation of the VirtualGateway can take quite some time, so be patient…
  4. Create a local network gateway
    • Name: LocalGateway
    • IP Address:
    • Address Space: 10.83.4.0/24 (your own internal IP address space)
    • Create the VPN Connection (under VirtualGateway  Connections)
    • Name: IPsecER
    • Connection type: Site-to-Site (IPSec)
    • Virtual Network Gateway: VirtualGateway
    • Local Network Gateway: LocalGateway
    • Shared Key: <secret>

For testing purposes, I created a new Virtual Machine in Azure (Standard DS1 v2 (1 vcpu, 3.5 GiB memory). This is automatically connected to the newly created Virtual Network. Using the external IP address, you can use RDP for the initial connection. For security purposes, use the Network Security Group (NSG) to restrict RDP only to your own IP address. After connecting, enable the ‘File and Printer Shared (Echo request – ICMPv4-In) Inbound Rule in Windows Defender Firewall. This makes connectivity testing a lot easier.

The Ubiquite EdgeRouter supports site-to-site VPN solutions, including site-to-site to Microsoft Azure which is documented in the EdgeRouter Route Based Site-to-Site VPN to Azure article. Not all devices are supported by Microsoft as documented in the Microsoft article About VPN devices and IPsec/IKE parameters for Site-to-Site VPN Gateway connections. Keep in mind though that not tested by Microsoft means ‘not supported by Microsoft’ but that does not automatically include ‘does not work’. If the vendor (Ubiquity in my case) supports it, you’re good to go.

When the router is configured correctly the site-to-site connection is automatically setup and the VPN Connection status should change to ‘Connected’ as shown in the following screenshot:

At this point you should be able to ping the VM in Azure (that’s connected to the virtual network that we created earlier) from a machine in your local network and vice versa.

In my next blog I will discuss installing a Domain Controller in Azure.

Exchange

January 2022 Exchange Security Updates

Leave a comment

On january 11, 2022 Microsoft released new Security Updates for Exchange versions:

  • Exchange 2013 CU23
  • Exchange 2016 CU21, Exchange 2016 CU22
  • Exchange 2019 CU10, Exchange 2019 CU11

The following vulnerabilities have been addressed in these Security Updates:

No exploits have been found in the wild, but it is recommended to install these Security Updates as soon as possible.

These updates are targeted toward Exchange server on-premises, including Exchange servers used in a hybrid configuration.

Please note the following:

  • Run the Exchange Server Healthcheck script on your Exchange server to get an overview of all issues in your environment, including installed Security Updates and Cumulative Updates versions.
  • If running an old (and unsupported!) version of Exchange server, please update to the latest CU to get in a supported state and install these Security Updates.
  • When installing manually, start the update from a command prompt with elevated privileges. If you fail to do so, it will look like installation successfully finishes, but various issues will occur. This is not needed when installing using Windows Update or WSUS.
  • Security Updates are also cumulative, so this Security Updates contains all previous Security Updates for this specific Cumulative Update. There’s no need to install previous Security Updates before installing this Security Update.
  • The December 2021 Cumulative Update is postponed, check the link on the Microsoft site. Microsoft does not release Security Updates and Security Updates in the same month, so do not except a new Cumulative Update anytime soon.
  • This Security Update does not contain a fix for the Y2K22 problem that popped up on January 1, see the Email stuck in Exchange on-premises Transport Queues article which also contains the solution.
  • As always, download and deploy in your test environment to see if it all works well in your environment.
Exchange versionDownloadKnowledge base
Exchange 2013 CU23https://www.microsoft.com/en-us/download/details.aspx?id=103857KB5008631
Exchange 2016 CU21https://www.microsoft.com/en-us/download/details.aspx?id=103856KB5008631
Exchange 2016 CU22https://www.microsoft.com/en-us/download/details.aspx?id=103855KB5008631
Exchange 2019 CU10https://www.microsoft.com/en-us/download/details.aspx?id=103853KB5008631
Exchange 2019 CU11https://www.microsoft.com/en-us/download/details.aspx?id=103854KB5008631
Exchange

The FIP-FS Scan Process failed initialisation. Mail is queued on Exchange servers.

6 Comments

According to a post on Reddit it seems that Exchange 2016 and Exchange 2019 on-premises are queueing email messages starting januari 1, 2022 at midnight (UTC).

Besides mail queueing you will also see EventID 1106 in the application event log stating “The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error”

This might be a bug is how the date is handles inside the scan engine, causing it to fail after midnight UTC on Januari 1, 2022 (or is it December 31, 2021).

As there is no fix from Microsoft yet, the workaround is to disable anti-malware scanning on all your Exchange servers and restart the Transport service:

CD $ExScripts
.\Disable-AntiMalwareScanning.ps1
Restart-Service MSExchangeTransport

Update 1 on January 1 at 9PM GMT+1. Microsoft is aware of this issue and working on a solution. Check the Exchange team blog: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

Update 2 on January 2 at 10AM GMT+1. Microsoft has released a solution for this issue. A script can be downloaded from https://aka.ms/ResetScanEngineVersion. This will stop the services, delete the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft and the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata directories and download new scan engines. This can take a couple of minutes, but the script can be run in parallel on all your Exchange servers.

When the script has finished, check the eventlog and you should see EventID 6036 that all is well.

Although not documented, I had to reboot my Exchange servers. I’ve several other reports from people that they had to reboot too. Another thing, when you disabled the anti-malware as a workaround, you have to re-enable the anti-malware manually. You can check this using the Get-TransportAgent “malware agent” command.

It is also possible to manually update your Exchange servers, this is also documented in the Microsoft article https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

One warning though….. All Exchange 2016 and 2019 servers worldwide are suffering from this issue and they all will queue messages. Queues expire after 48 hours, so when not fixed the Exchange servers will generate NDR messages on Sunday night. Worst case scenario, millions of NDRs will be generated which in turn will result in tons of helpdesk calls. If you read this, most likely you have fixed your Exchange servers and it is the Exchange environment of the intended recipients.

Let’s hope it will be quiet again for some time now 🙂