Last week we had a major outage in our Exchange 2010 environment (28 multi-role servers in 2 DAGs). The provisioning system (based on Quest software) did some unexpected things after a restore of the provisioning database, resulting in (lots of) security groups in Active Directory being deleted. We were relatively lucky since the default groups (Domain Admins, Enterprise Admins etc.) were not deleted, but all Exchange Security Groups (in OU=Microsoft Exchange Security Group) were deleted.

These Exchange Security Groups can be recreated using the Setup.com /PrepareAD and Setup.com /PrepareDomain commands.

All seems to be running fine, but when executing PowerShell commands against a remote server (i.e. not the server being logged on to) would result in error message. For example, it was not possible to move an active Mailbox database from server1 to server2 in a DAG using the Move-ActiveMailboxDatabase command. When executing this command it would return the following error:

The Microsoft Exchange Replication service does not appear to be running on “computername”. Make sure the server is operating, and that the services can be queried remotely.

As you can see in the screenshot the Microsoft Exchange Replication service (MSExchangeRepl) service was running fine on the remote server (on all servers actually). At first it looked like a firewall issue, but this was not the case. All firewall exceptions were in place, and temporarily disabling the firewall did not solve the problem.

It turned out to be a security issue. Since the security groups were recreated, the Exchange Trusted Subsystem was no longer a member of the local Administrators security group of the Exchange 2010 servers (including the Witness server) as can be seen in the following screenshot:

Adding the Exchange Trusted Subsystem security group to the local Administrator security group on all Exchange 2010 servers and the Witness server solved the problem and all functionality was available again.