Windows Server 2016 Hyper-V quickly fills up system disk

Recently I had to replace my two lab servers, so I bought two brand new HP DL360-Gen9 servers. Lots of memory and a number of disks and processor capacity. Two weeks after installing Windows 2016 Hyper-V I noticed that my system disk (C:\ drive, approx. 185 GB) was filling up rapidly.

image

Initially I thougt it was the paging file (with 192 GB internal memory this can be an issue) but this was not the case since the paging file was located on drive D:\

Further investigation revealed that most data was located in the directory C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines, where all VM related files are located (except the virtual hard disks which were located in D:\Hyper-V\Virtual Hard Disks). It turned out to be very dynamic data located in .VMRS file. When a VM was turned off the VMRS file was gone, as soon as the VM was turned on again dir VMRS file was allocated again, and the size of the file was identical to the amount of memory of the Virtual Machine as can be seen in the following screenshot:

image

Next I’ve been looking at the smart paging option in Hyper-V, but this only makes sense when using dynamic memory, which was not the case in my environment (VMs were running Exchange 2013/2016).

Production snapshots are new in Windows 2016 Hyper-V. Production snapshots use VSS to create a snapshot (where the traditional snapshots create a system state using .VSV and .BIN files) so that would make sense in my scenario. But disabling snapshots at all on a VM basis didn’t make any difference, and the .VMRS files were still created.

The last option I had was the Automatic Stop Action option in Hyper-V (on a per VM basis). Using this option you can control what happens when the host shuts down. By default it is set to Save the virtual machine state, so when the Hyper-V host shuts down the entire VM is saved at that particular moment. To achieve this, space on disk is reserved equal to the amount of memory used by VM. Other options here are Turn off the virtual machine and Shut down the guest operating system.

image

Bingo, this was my issue. Save state will certainly have performance benefits, but I prefer to use the shut down option in my lab environment. After changing this on (most of) my VMs I have plenty of free space on my system disk Glimlach

This mailbox database contains one or more mailboxes

After moving all mailboxes from a mailbox database to other mailbox databases (both in Exchange 2016), including archive and arbitration mailboxes the mailbox database still cannot be deleted. Trying to remove the mailbox database still generates the following error message:

This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, public folder mailboxes or arbitration mailboxes, Audit mailboxes. To get a list of all mailboxes in this database, run the command Get-Mailbox -Database <Database ID>. To get a list of all mailbox plans in this database, run the command Get-MailboxPlan.

image

Get-Mailbox -Arbitration or Get-Mailbox -Archive doesn’t return anything.

Exchange 2016 introduced a new type of mailbox, the auditlog mailbox. This can be returned by the Get-Mailbox -Auditlog command:

image

Move this auditlog mailbox to another mailbox database and the old mailbox is now fully empty and can be deleted.

REGISTRY_FILTER_DRIVER_EXCEPTION (swin1.sys)

A blue screen on your Windows box, is there anyone that has never seen it? One place you don’t want to see this is on your Exchange server.

While upgrading 3 Exchange 2013 CU12 servers for a customer to Exchange 2013 CU15 I experienced a blue screen while updating the UM components, resulting in the following screen:

image

A quick search on Google (search on “swin1.sys”) revealed that McAfee was the culprit. I’m not a fan of installing (file level) anti-virus software on an Exchange server, and IMHO it’s not needed when you have a properly secured environment.

I was successful installing CU15 on the Edge Transport servers earlier, but it turned out that these servers were not running McAfee software.

Uninstalling the McAfee agent is not a big deal, just a matter of deselecting the server in the EPO console. The McAfee Solidifier didn’t want to uninstall, not via the EPO console nor via Add/Remove Programs.

According to the McAfee knowledgebase article KB75902 you can uninstall this software using a command line with the following commands:

sc stop scsrvc
sc delete scsrvc
sc delete swin
"\Program Files (x86)\McAfee\Common Framework\Mctray.exe" unloadplugin=scormcpl.dll
del /Q "C:\WINDOWS\system32\drivers\swin.sys"
rmdir /S /Q C:\Solidcore
rmdir /S /Q "C:\Program Files\McAfee\Solidcore"
rmdir /S /Q "C:\ProgramData\McAfee\Solidcore"
reg delete "HKLM\SOFTWARE\Network Associates\ePolicy Orchestrator\Application Plugins\SOLIDCOR5000_WIN" /f
reg delete "HKLM\SOFTWARE\Classes\Installer\Features\4E9BD2348836F234A9BD168E87F25439" /f
reg delete "HKLM\SOFTWARE\Classes\Installer\Products\4E9BD2348836F234A9BD168E87F25439" /f
reg delete "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{432DB9E4-6388-432F-9ADB-61E8782F4593}" /f

After running these commands and a reboot I was able to finalize the installation of Exchange 2013 CU15.

Updated Exchange Server Role Requirements Calculator

The Exchange Server Role Requirements Calculator which you need for designing a proper Exchange 2013 or Exchange 2016 on-premises environment has been updates, currently the calculator is at version 8.4.

Some new functionality is added to this version:

  • Added support for ReplayLagMaxDelay
  • Added support for SafetyNetHoldTime in CreateDAG.ps1

And seven bug fixes in this version:

  • Improved the DAG auto-calculation results display to highlight deployment configuration in both datacenters
  • Fixed an issue that prevented DAG auto-calculation in single site DAG deployments
  • Fixed a SPECInt2006 validation issue with DAG auto-calculation
  • Fixed a bug with the DAG auto-calculation with Active/Passive deployments
  • Fixed conditional formatting issues with the transaction log table
  • Removed data validation from certain unused cells on the Input tab
  • Fixed bug in calcNumActiveDBsSF formula

The requirements calculator can be downloaded here: https://gallery.technet.microsoft.com/Exchange-2013-Server-Role-f8a61780

More information:

Exchange 2016 CU4

On Tuesday december 13, 2016 Microsoft has released its quaterly updates:

and Update Rollups:

Looking at Exchange 2016, what does this CU bring us?

There are minor changes to the user interface of Outlook on the Web or Outlook Web App, whatever you may call it. It’s not that dramatically, the formatting controls have been moved to the bottom of the frame containing the editing pane, as can be seen on the following screenshot.

image.png

And finally, there’s support for the .NET Framework 4.6.2. Using .NET Framework 4.6.2 is still optional (but recommended), but the upcoming release in March 2017 (Exchange 2016 CU5) will require the use of .NET Framework 4.6.2.

As you might have noticed, Exchange 2016 CU3 (the previous release) introduced support for Windows Server 2016. This was also announced at the Ignite 2016 in Atlanta. Unfortunately there was a major flaw in Windows 2016 clustering that caused issues with Exchange 2016 in a Database Availability Group configuration. This has now been fixed by the Windows team (KB3206632), and Exchange 2016 again fully supports Windows 2016. The hotfix is mentioned is mandatory, and the setup application does a check for this hotfix.

Exchange 2016 does not introduce any new schema changes, but you may execute setup.exe /prepareAD /IAcceptExchangeServerLicenseTerms to make sure any changes in the configuration partition are applied successfully.

As usual, and especially after the latest issues with Exchange 2016 CU3 and Windows 2016 I strongly encourage everybody to thoroughly test Exchange 2016 CU4 (or any other update that’s needed of course) in your lab environment before bringing it into production!