Improve autodiscover performance

Autodiscover can be a lengthy process, especially if you are in a hosted environment or if your mailbox is in Office 365.

The autodiscover process consists of five different steps, it depends on your environment where autodiscover stops and returns the information. Autodiscover is using the following mechanisms:

  • Service Connection Point (SCP) in Active Directory. This is used by domain clients.
  • Root domain discovery, used by non domain joined clients or clients not being able to access Active Directory. All other steps are used by these clients as well.
  • Autodiscover.contoso.com (standard autodiscover mechanism)
  • Autodiscover redirect to autodiscover site (often used by hosting companies)
  • Autodiscover SRV records in DNS (sometimes used by hosting companies)
  • Autodiscover redirect to Office 365 (outlook.com)

If your mailbox is in Office 365, outlook will go through all these steps until it finds the information in Office 365. All steps will fail with the accompanying time-out and this will take quite some time. This can be seen in the Outlook Test Email AutoConfiguration option:

image

Continue reading Improve autodiscover performance

Cisco Email Security Appliance and DKIM Signing

In a previous blogpost, I already discussed DKIM signing with Exchange 2016:  SenderID, SPF, DKIM and DMARC in Exchange 2016 – Part II

Exchange out-of-the-box does support SPF checking, but DKIM signing/verifying and DMARC verifying are not supported. There are free 3rd party tools for DKIM Signing that can be found on GitHub, but at the moment of writing this tool only supports DKIM Signing, but does not support DKIM verifying. I have to admit that DKIM signing with this tool works very well.

I already explained earlier that I’ve installed and configured a Cisco Email Security Appliance (ESA, previously known as IronPort) appliance in my lab environment, and this is installed like this:

image

Figure 1. My testlab Exchange environment.

All outbound SMTP mail is via the ESA. The FQDN of the ESA is smtphost.exchangelabs.nl, of course it has a public IP address with a corresponding PTR record.

Continue reading Cisco Email Security Appliance and DKIM Signing

Cisco IronPort and Exchange 2016

If you have been following my blogs over the years you should be aware that I’ve always been using Exchange Edge Transport servers in front of my Mailbox servers for message hygiene purposes. My last (well known) environment looked like this:

image

There are two Mailbox servers (Exchange 2013 and Exchange 2016) and two Edge Transport servers (also Exchange 2013 and Exchange 2016). MX records point to both Edge Transport servers and there are two Edge Synchronizations. And the Edge Transport servers were capable for DKIM signing (as posted in a previous blogpost), but lacked DKIM verification and DMARC validation.

The most important part in the Edge Transport server is the Real Time Blocklist, configured to use Spamhaus for connection filtering. While this works pretty well (there still is quite some spam that gets delivered into mailboxes) there is always room for improvement. I have been looking at cloud solution, but they didn’t always deliver what was expected.

A couple of my customers are using Cisco Email Security Appliance (previously known as IronPort) solutions on-premises and are happy with it, so time to start testing a Cisco Email Security Appliance (ESA) in my own environment.

Setting up the ESA

The ESA is available as a physical device, but also as a virtual device. Unfortunately (says the Microsoft consultant in me) only as a vmware VM, but since I also have a vmware server running that’s not a problem. Deploying the virtual machine is because of the OVF solution a piece of cake. Connect three NICs (management, internal and external) and you’re good. My Cisco contact told me it’s a common practice to use a one arm deployment, but I did a two arm deployment, resulting in a configuration like this:

image

The management network is (by default) 192.168.42.42, the internal network is configured to use my 10.38.96.0/24 network while the external network is configured with the 176.62.196.244 address. The FQDN of the device is smtphost.exchangelabs.nl, in DNS this points to the above mentioned IP address and I have PTR records configured for this IP address.

Afther the initial setup (where entering the license file via PUTTY in the CLI was the most challenging) you can continue with the System Settings in the GUI. The default domain is exchangelabs.nl, enter a reporting address and a time server as shown in the following screenshot:

image

After entering the system settings you can continue with the network configuration. Enter a default gateway (on the public interface), a public and internal IP address and enter a forwarding address for inbound messages (i.e. the Exchange 2013 and Exchange 2016 servers) as shown in the following screenshot:

image

The last step is to configure the message security options like enabling the Reputation filtering, IronPort anti-spam, Sophos anti-virus and outbreak filters as shown in the following screenshot:

image

After reviewing the configuration you’re good to go and once accepted the device configures itself. This took only a few minutes, and I immediately started receiving configuration messages from the ESA. Some of these made no sense, it looks like the device starts sending out messages before the device is fully configured, but at least it worked well.

image

The next step is to create a Send Connector in the Exchange configuration, using the ESA as a smarthost. All outbound mail is routed through the ESA, since this has a proper FQDN, IP address and PTR record other servers on the Internet will accept my email messages.

Adding additional domains

Besides my test domain I want my jaapwesselius.com and jaapwesselius.nl domains to use the ESA. First is to add the domains in the Recipients Address Table (RAT) so that the ESA will accept messages for this domain. The second (and last) step is to configure an SMTP route for these domains so the messages will be delivered on my Exchange servers. Note to self: use the commit changes button after entering the domain configuration data 🙂

When checking inbound email on my platform (using Remote Connectivity Analyzer) you can clearly see that mail is going through the ESA:

image

Message Hygiene

To understand message hygiene in the ESA we have to take a closer look at the mail flow in the ESA. This is called the Email pipeline and consists of three parts:

  • Receipt – this is the initial step in receiving a message, i.e. accepting the connection, check for policies and validate the recipient
  • Work queue – when connected the message enter the work queue where filtering, safelist/blocklist, anti-spam/anti-virus, outbreak filtering and quarantining takes place.
  • Delivery – responsible for delivering the incoming message to the Exchange server.

One of the most important parts on any email security solution is connection filtering, you want to make sure that you only accept messages from trusted sources. Messages coming from sources known as spammers should not be accepted. On an Exchange Edge Transport server you can configure Realtime Block Lists (RBL) and I often use SpamHaus for this, as explained in my blogpost about configuring the Edge Transport Server.

The ESA is using Sender Reputation Filtering to achieve this. Sender Reputation Filtering is using SenderBase Reputation Service which is using the SenderBase affiliate network. The SenderBase Reputation Service assigns a SenderBase Reputation Score (SBRS) to email senders based on complaint rates, message volume statistics, and data from public blacklists and open proxy lists. The SenderBase Reputation Score helps to differentiate legitimate senders from spam sources. The SBRS can vary between -10 (most likely to be spam) to +10 (most likely to be legitimate).

image

After working with this for a couple of week now (in default configuration) I have to admit it is working extremely well. I haven’t received any spam messages so far, the only message I don’t want to receive are legitimate messages from primarily vendor sales departments.

TLS

By default, the ESA is configured with a self-signed certificate. This certificate is only used for accessing the ESA (for management purposes). To use TLS on this server you have to request (or import) a 3rd party SSL certificate.

After importing the SSL certificate you have to configure the listener and the mail flow policies on the Host Address Table (HAT). When configured you can use the http://checktls.com site to check if your site is TLS ready:

image

If you check the incoming TLS messages (the next day) you can see how many TLS encrypted messages are received. Personally I am a bit surprised that I still received 21 unencrypted messages yesterday (an average Tuesday).

image

Reporting

Reporting is always interesting, and the ESA sends out reports (in PDF) every day. One of these reports is about incoming messages. As can be seen in the following screenshot a lot of messages are blocked by the Sender Reputation Filtering:

image

In the same PDF you can see an overview of the sending (threat message) domains:

image

Outbound messages

The ESA can (should?) also be used for outbound messages. The mailflow in my environment is changed, the Edge Transport servers are no longer used. Instead, a Send Connector is created and the ESA is configured as a smart host on the Send Conector.

During installation, the 2nd network interface was configured to relay outbound messages, and the FQDN of outbound messages is configured to use smtphost.exchangelabs.nl so outbound messages appear to come from smtphost.exchangelabs.nl.

I have configured a reverse DNS entry, so when using http://misk.com/tools I can check if this is correct:

image

So, when a receiving SMTP host does a reverse lookup on my sending IP address they will see that there’s a match with my FQDN. So far I haven’t heard any complaints about messages not being delivered 🙂

Summary

The Exchange Edge Transport servers can be used for connection filtering and content filtering using keywords can be configured as well. Besides this the Edge Transport servers are not really an anti-spam solution. Most of the times I see Edge Transport servers they are used as a mail relay server sitting in the network’s DMZ.

For message hygiene there are several solutions, and the last couple of weeks I have been playing with the Cisco Email Security Appliance (running on Vmware). I must say, I am VERY IMPRESSED by the appliance. Installation was very easy to do, and using the ESA User Manual (consisting of 1216 pages 🙂 ) you can do a real deep dive into message hygiene.

At this moment I have an almost default configuration for inbound and outbound mailflow (including multiple domains and mailboxes) and I haven’t seen any spam since the installation.

In the upcoming blog posts I will go more into details about for example configuring anti-spam, DKIM signing, DMARC etc. using the Cisco Email Security Appliance.

Exchange 2016, backup and recovery databases

Last week I got a request from a customer. A long time ago I posted a blogpost on Exchange 2010 recovery databases, but after the customer migrated to Exchange 2016 his procedure around recovery databases didn’t work anymore. His request was basically to rewrite my blogpost.

For this blogpost I have a pretty simple Exchange 2016 Mailbox server, configured with one Mailbox database which is stored on a dedicated disk, and I’m using Windows Server Backup to backup the entire Mailbox database disk (VSS full backup).

Don’t pay too much attention to the naming of my Exchange server and the Mailbox database I’m using here. In fact, this is an Exchange 2016 hybrid server I’m misusing for the purpose of this blog Smile

Recovery database

You can restore a mailbox database to its original location and mount again, but you can also use a Recovery Database to restore and recover your data. A recovery database is a mailbox database that can be mounted on your Exchange server, but it is not visible for regular users but only for the Exchange administrator. The Exchange administrator can access this recovery database and recover data, for example create a PST of a particular mailbox in this database.

When restoring a database from backup, select the restore option and follow the wizard. When you reach the Select Recovery Type window select Applications as shown in the following screenshot.

image

Continue reading Exchange 2016, backup and recovery databases

Windows Server 2016 Hyper-V quickly fills up system disk

Recently I had to replace my two lab servers, so I bought two brand new HP DL360-Gen9 servers. Lots of memory and a number of disks and processor capacity. Two weeks after installing Windows 2016 Hyper-V I noticed that my system disk (C:\ drive, approx. 185 GB) was filling up rapidly.

image

Initially I thougt it was the paging file (with 192 GB internal memory this can be an issue) but this was not the case since the paging file was located on drive D:\

Further investigation revealed that most data was located in the directory C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines, where all VM related files are located (except the virtual hard disks which were located in D:\Hyper-V\Virtual Hard Disks). It turned out to be very dynamic data located in .VMRS file. When a VM was turned off the VMRS file was gone, as soon as the VM was turned on again dir VMRS file was allocated again, and the size of the file was identical to the amount of memory of the Virtual Machine as can be seen in the following screenshot:

image

Next I’ve been looking at the smart paging option in Hyper-V, but this only makes sense when using dynamic memory, which was not the case in my environment (VMs were running Exchange 2013/2016).

Production snapshots are new in Windows 2016 Hyper-V. Production snapshots use VSS to create a snapshot (where the traditional snapshots create a system state using .VSV and .BIN files) so that would make sense in my scenario. But disabling snapshots at all on a VM basis didn’t make any difference, and the .VMRS files were still created.

The last option I had was the Automatic Stop Action option in Hyper-V (on a per VM basis). Using this option you can control what happens when the host shuts down. By default it is set to Save the virtual machine state, so when the Hyper-V host shuts down the entire VM is saved at that particular moment. To achieve this, space on disk is reserved equal to the amount of memory used by VM. Other options here are Turn off the virtual machine and Shut down the guest operating system.

image

Bingo, this was my issue. Save state will certainly have performance benefits, but I prefer to use the shut down option in my lab environment. After changing this on (most of) my VMs I have plenty of free space on my system disk Glimlach