Tag Archives: Active Directory

Transfer FSMO Role Win32 error returned is 0x20af

For a client I had to test some domain controller scenarios and issues. There are two domain controllers (DC01 and DC02) and a new one (NewDC) is added. FSMO roles are moved to NewDC and DC01 must be decommissioned. No big deal you would say….

Domain FSMO roles (RID Master, PDC and Infrastructure master) moved in an instant, but moving the forest FSMO roles (Schema master and Domain Naming master) failed with the following error, both in the GUI as well as NTDSUTIL:

Win32 error returned is 0x20af(The requested FSMO operation failed. The current FSMO holder could not be contacted.)
Depending on the error code this may indicate a connection, ldap, or role transfer error.

Using Netdom query FSMO I could see the roles.

This happened when logged on the AD01 as well as NewDC, connectivity was not an issue and I did not notice any strange issues with Exchange and Outlook or OWA clients.

Using Google I found several other similar issues, and the overall recommendation was to seize the FSMO roles. Seizing FSMO roles is an option when the old Domain Controller is no longer available, but in my scenario I had to keep the old Domain Controller for testing.

But instead of using Google I had to look in the eventlog in the first place. The first entry I found was a confirmation that the roles could not be transferred, but Additional Data Error value: 4 was not very helpful:

An attempt to transfer the operations master role represented by the following object failed.

Object:
CN=Partitions,CN=Configuration,DC=msexchangebook,DC=com
Current operations master role:
CN=NTDS Settings,CN=DC01,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=msexchangebook,DC=com
Proposed operations master role:
CN=NTDS Settings,CN=NIEUWEDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=msexchangebook,DC=com

Additional Data
Error value:
4

Subsequent entries were more helpful, Event 2019, ActiveDirectory_DomainService indicated replication issues:

Or in tekst for Search Optimization:

This server is the owner of the following FSMO role, but does not consider it valid. For the partition which contains the FSMO, this server has not replicated successfully with any of its partners since this server has been restarted. Replication errors are preventing validation of this role.

Operations which require contacting a FSMO operation master will fail until this condition is corrected.

FSMO Role: CN=Partitions,CN=Configuration,DC=msexchangebook,DC=com

User Action:

  1. Initial synchronization is the first early replications done by a system as it is starting. A failure to initially synchronize may explain why a FSMO role cannot be validated. This process is explained in KB article 305476.
  2. This server has one or more replication partners, and replication is failing for all of these partners. Use the command repadmin /showrepl to display the replication errors. Correct the error in question. For example there maybe problems with IP connectivity, DNS name resolution, or security authentication that are preventing successful replication.
  3. In the rare event that all replication partners are expected to be offline (for example, because of maintenance or disaster recovery), you can force the role to be validated. This can be done by using NTDSUTIL.EXE to seize the role to the same server. This may be done using the steps provided in KB articles 255504 and 324801 on http://support.microsoft.com.

The following operations may be impacted:
Schema: You will no longer be able to modify the schema for this forest.
Domain Naming: You will no longer be able to add or remove domains from this forest.
PDC: You will no longer be able to perform primary domain controller operations, such as Group Policy updates and password resets for non-Active Directory Domain Services accounts.
RID: You will not be able to allocation new security identifiers for new user accounts, computer accounts or security groups.
Infrastructure: Cross-domain name references, such as universal group memberships, will not be updated properly if their target object is moved or renamed.

In the end it turned out that DNS was not configured correctly, hence the replication issues. Fixed the DNS settings on the Domain Controllers and after a short while the Domain Controllers resumed replication and I was able to move the Forest FSMO roles to the new Domain Controller.

So in short, always check eventlog, replication and DNS when troubleshooting Domain Controller issues 😉

Exchange 2010 – This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, or arbitration mailboxes

After migrating all mailboxes to Exchange 2016 and Exchange Online it is time to decommission the old Exchange 2010 servers. One of the servers could not be removed and the dreaded “This mailbox database contains one or more mailboxes, mailbox plans, archive mailboxes, or arbitration mailboxes” was shown:

This typically means that there is some sort of mailbox (arbitration or archive) is still available in the database, causing this issue. Unfortunately using the -Verbose switch did not reveal any other useful information.

Also, when trying to use Remote PowerShell directly (Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010, which bypasses RBAC and sometimes help) dit not help either this time.

Note. I was able to find dozens of similar threads on the Internet about this warning, but not a single one applied to my scenario. Ok, I found one where they removed the mailbox database using ADSI Edit, but on the long term that will cause other issues I’m afraid. You can do this, but there are still references to this mailbox database in Active Directory, so stuff will be lingering if you do this. Also, a lot of the threads are about Exchange 2013 and higher, but we were decommissioning Exchange 2010.

Last resort is to look directly into the mailbox database properties in Active Directory using LDP.exe. In LDP, navigate to the mailbox database and open the details (or dump it to a text file). Using the dump a user account was found that still referenced this mailbox database for an archive mailbox:

The user account was fixed and we were able to remove the mailbox database.

Configure OAuth authentication in Exchange 2016

As long as I can remember the Hybrid Configuration Wizard finishes successfully, and itgenerates the error about the OAuth portion of the hybrid configuration.

Configure Intra-Organization Connector

HCW8064 – The HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration. If you need features that rely on OAuth, you can try running the HCW again or manually configure OAuth using these manual steps.

The Learn more option redirects to the Microsoft page Configure OAuth authentication between Exchange and Exchange Online organizations. I used that article for the PowerShell commands in this blogpost.

OAuth is used cross-premises to logon to other services, on behalf of the user. So, if you are logged on to some Microsoft service, this service can use OAuth to access services in Exchange on-premises and vice versa.

Example of these cross-premises services are:

  • Message Records Management (MRM).
  • Exchange in-place eDiscovery.
  • Exchange in-place Archiving.
  • Teams calendaring.

The HCW can configure Azure Active Directory for OAuth authentication, it can create the IntraOrganizationConnectors, but it cannot export and import the (self-signed) certificate on the Exchange server, nor can it (or does it) create the authorization server objects in Active Directory. So, time to test, guided by the Microsoft article and write down my experiences.

Note. This only works for Exchange 2013 and higher, I have been working on this in a mixed Exchange 2016 and Exchange 2019 environment.

Configuring OAuth between Office 365 and Exchange Online involve a number of steps.

Create Authorization server objects in Exchange on-premises

To create the authorization server objects in your on-premises environment enter the following commands in the Exchange Management Shell.

New-AuthServer -Name "WindowsAzureACS" -AuthMetadataUrl "https://accounts.accesscontrol.windows.net/contoso.com/metadata/json/1"
New-AuthServer -Name "evoSTS" -Type AzureAD -AuthMetadataUrl https://login.windows.net/contoso.com/federationmetadata/2007-06/federationmetadata.xml

Your verified domain contoso.com (in the command) should be something like exchangelabs.nl, and not <your tenant name> as outlined in the Microsoft article.

New-AuthServer

Enable the partner application for use with Exchange Online

The partner application was created in the previous step (the first command) and this should be enabled. Do this using the following command in Exchange Management Shell (on-premises):

Get-PartnerApplication | ?{$_.ApplicationIdentifier -eq "00000002-0000-0ff1-ce00-000000000000" -and $_.Realm -eq ""} | Set-PartnerApplication -Enabled $true

Export the Exchange authorization certificate

Authentication cross-premises is using certificates, so the on-premises certificate needs to be exported to Azure Active Directory. In case you were wondering where the CN=Microsoft Exchange Server Auth Certificate certificate was coming from when running the Get-ExchangeCertificate command in Exchange Management Shell, here you go.

Use the following PowerShell commands and store them in a PowerShell script called ExportAuthCert.ps1 or something and run it. This should export the OAuth certificate to a file called OAuthCert.cer.

$ThumbPrint = (Get-AuthConfig).CurrentCertificateThumbprint
If((Test-Path $ENV:SYSTEMDRIVE\OAuthConfig) -eq $false)
{
md $ENV:SYSTEMDRIVE\OAuthConfig
}
CD $ENV:SYSTEMDRIVE\OAuthConfig
$oAuthCert = (dir Cert:\LocalMachine\My) | ?{$_.ThumbPrint -Match $ThumbPrint}
$CertType = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert
$CertBytes = $oAuthCert.Export($CertType)
$CertFile = "$env:SYSTEMDRIVE\OAuthConfig\OAuthCert.cer"
[System.IO.File]::WriteAllBytes($CertFile, $CertBytes)

ExportAuthCert

Note. The Export-ExchangeCertificate command doesn’t work in this scenario since the self-signed certificate isn’t exportable.

Import the Exchange authorization certificate into Azure AD

The next step is to import the OAuthCert.cer certificate into Azure AD. Connect to the Microsoft Online service (Connect-MSOLService, if you don’t have this installed you can use the Install-Module MSOnline command) and run the following commands when connected:

$Cred = Get-Credential
Connect-MSOLService -Credential $Cred
$CertFile = "$ENV:SYSTEMDRIVE\OAuthConfig\OAuthCert.cer"
$objFSO = New-Object -ComObject Scripting.FileSystemObject
$CertFile = $objFSO.GetAbsolutePathName($CertFile)
$CER = New-Object System.Security.Cryptography.X509Certificates.X509Certificate
$CER.Import($CertFile)
$binCert = $cer.GetRawCertData()
$CredValue = [System.Convert]::ToBase64String($binCert)
$ServiceName = "00000002-0000-0ff1-ce00-000000000000"
$P = Get-MsolServicePrincipal -ServicePrincipalName $ServiceName
New-MsolServicePrincipalCredential -AppPrincipalId $P.AppPrincipalId -Type asymmetric -Usage Verify -Value $credValue

This will import the self-signed certificate from the Exchange server into Azure AD so it can be used for mutual authentication.

I did not run the commands mentioned above on my Exchange server but on my Azure AD Connect server since the MSOL module was loaded on that server. For importing the certificate file I had to use the following command accessing the certificate file (instead of the $ENV:System variable):

$CertFile = "\\AMS-EXCH01\C$\OAuthConfig\OAuthCert.cer"

Register endpoints in Azure Active Directory

The last step is to register the endpoints of your on-premises Exchange environment into Azure Active Directory. You can use the following commands to register the endpoints:

$ServiceName = "00000002-0000-0ff1-ce00-000000000000";
$x = Get-MsolServicePrincipal -AppPrincipalId $ServiceName;
$x.ServicePrincipalnames.Add("https://webmail.exchangeserver.com/");
$x.ServicePrincipalnames.Add("https://autodiscover.exchangeserver.com/");
Set-MSOLServicePrincipal -AppPrincipalId $ServiceName -ServicePrincipalNames $x.ServicePrincipalNames;

Instead of webmail.exchangeserver.com and Autodiscover.exchangeserver.com you must use your own local FQDNs of the Exchange server (shown below in the verification screenshot).

You can use the the following command to check if this was configured correctly.

Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 | select -ExpandProperty ServicePrincipalNames

As shown in the following screenshot:

Register Endpoints

Note. Over the years I have run the HCW several times. Domains have been added, but not (automatically) deleted in Azure Active Directory.

IntraOrganizationConnectors and AvailabilityAddressSpace

The Hybrid Configuration Wizard created the IntraOrganizationConnectors (both in Exchange 2016 as well as Exchange Online) and configured the AvailabilityAddressSpace. There’s no need to create these, but you have to check them using the Get-IntraOrganizationConnector and the Get-AvailabilityAddressSpace commands)

Verify the OAuth configuration

To verify the OAuth configuration you can use the Test-OAuthConnectivity command. You must do this on the on-premises Exchange server and in Exchange Online.

On the on-premises Exchange server use the Exchange Online Uri and a mailbox on-premises:

Test-OAuthConnectivity -Service EWS -TargetUri https://outlook.office365.com/ews/exchange.asmx -Mailbox onprem-user@exchangeserver.com -Verbose | Format-List

In Exchange Online, use the Exchange on-premises Uri with a mailbox in Exchange Online:

Test-OAuthConnectivity -Service EWS -TargetUri https://webmail.exchangeserver.com/metadata/json/1 -Mailbox Online-User@exchangeserver.com -Verbose | Format-List

The output is an extended list, but in the end you should see ResultType: Success and IsValid:True on your console:

Test-OAuthConnectivity

You have now configured the OAuth between Exchange Online and Exchange On-Premises.

 

on-Premises Azure Active Directory Password Protection

Last year I wrote a blogpost on password in Azure Active Directory (Choose a password that’s harder for people to guess – https://jaapwesselius.com/2018/10/15/choose-a-password-thats-harder-for-people-to-guess/) in which I mentioned the banned password lists and the Azure AD Password Protect feature. Back then this was only for Azure AD, but right now it is also available for on-premises Domain Controller as well (for some time already). It is possible for on-premises Domain Controllers to use the password protect functionality in Azure AD and thus block the possibility to use weak passwords in your on-premises environment. Let’s see how it works.

The password protection feature on-premises uses a Password Protection Agent that’s running on the on-premises Domain Controllers. When a user initiates a password change, the new password is validated by the Azure AD Password Protection agent, which request a password policy from the Azure AD Password Protection proxy service. This Password Protection service requests a password policy from Azure AD. The new password is never sent to Azure AD. This is shown in the following picture (borrowed from the Microsoft website):

azure-ad-password-protection

After receiving the password policy, the agent returns pass or fail for the new password. In case of fail the user must try it again.

Installation of the password protect consists of two steps:

  • The Azure AD Password Protection Proxy service using the AzureADPasswordProtectionProxySetup.exe software installer. This is installed on a domain joined computer that has access to the Internet and proxies the password policy request to Azure Active Directory.
  • The DC Agent service for password protection by using the AzureADPasswordProtectionDCAgentSetup.msi package. This runs on the Domain Controllers and send the password policy requests to the server running the proxy service.

Both can be downloaded from the Microsoft download center on https://www.microsoft.com/en-us/download/details.aspx?id=57071

Password Protection Proxy Installation

The first step is to install the password protection service. This server should be able to access Azure AD and since the Domain Controller does not have an internet connection this should be installed on a separate server. In my lab environment I have installed the password protection service on the Azure AD Connect server.

Installation of the password protection proxy is straightforward; you can use the GUI or the command line setup with the /quit switch for unattended install (and Server Core). After installation use PowerShell to register the proxy in Azure AD by using the following commands:

[PS] C:\> Import-Module AzureADPasswordProtection
[PS] C:\> Register-AzureADPasswordProtectionProxy -AccountUpn 'administrator@tenant.onmicrosoft.com'

This command can work when you have MFA enabled for admin accounts, if you don’t require MFA on your admin accounts (which is a bad practice IMHO) you can use the following command:

[PS] C:\> $globalAdminCredentials = Get-Credential
[PS] C:\> Register-AzureADPasswordProtectionProxy -AzureCredential $globalAdminCredentials

The last step is to register the forest in Azure Active Directory. This is very similar to the registration process of the proxy service. You can use the following PowerShell commands to register the forest:

Register-AzureADPasswordProtectionForest

[PS] C:\> Register-AzureADPasswordProtectionForest -AccountUpn ‘yourglobaladmin@yourtenant.onmicrosoft.com’

Again, when MFA is not enabled you can use the following command to register your forest in Azure AD:

[PS] C:\> $globalAdminCredentials = Get-Credential 'yourglobaladmin@yourtenant.onmicrosoft.com'
[PS] C:\> Register-AzureADPasswordProtectionForest -AzureCredential $globalAdminCredentials

Note. A multi-forest scenario is supported for the Password Protection service, you can install multiple forest using these commands. Multiple domains against one tenant is supported, one domain against multiple tenants is a not-supported scenario.

Some remarks:

  • The server where the password proxy agent server is installed should have .NET Framework 4.7 or higher installed.
  • For high availability it is recommended to install the password protection agents on multiple servers
  • The password protection proxy supports an in-place upgrade, so a newer version can be installed without uninstalling the previous version.

So how does this work, and how does the password protection service find the proxy server (or servers)?

When the Password Protection Proxy is installed it is registered in Active Directory with a well-know GUID. The Password Protection Agent checks Active Directory for this well-know GUID and finds the server where the Password Protection Agent is installed.

You can use the following PowerShell commands to find the Password Protection Proxy:

$SCP = "serviceConnectionPoint"
$Keywords = "{ebefb703-6113-413d-9167-9f8dd4d24468}*"
Get-ADObject -SearchScope Subtree -Filter {objectClass -eq $SCP -and keywords -like $Keywords }

It returns the server, and you can use ADSIEdit to inspect the computer:

Azure AD Password Protection Proxy SCP

This is much like how domain-joined Outlook clients find the Autodiscover SCP in Active Directory.

Installing the DC agent service

When the proxy service is installed and registered the Domain Controller agent service can be installed. It is just an MSI package that can be installed (using the GUI, accept license agreement and click install) or you can install it on the command line using the following command (use elevated privileges):

C:\> msiexec.exe /i AzureADPasswordProtectionDCAgentSetup.msi /quiet /qn

Note. Installation of the DC agent requires a restart, but you can use the /norestart switch to reboot at a more convenient time.

After rebooting the Domain Controller the password protection service is ready for use.

Some remarks:

  • Azure AD Password protection service requires an Azure AD Premium P1 or P2 license.
  • Domain Controllers should be Windows 2012 or higher.
  • Domain Controllers should have .NET Framework 4.5 or higher installed.
  • You never know which Domain Controller is going to process a password change. Therefore, the Password Protection service need to be installed on all Domain Controllers. For a straightforward environment this should not be a problem, but for large enterprises with lots of DC’s it can be an issue (I deliberately do not that about security officers at this point :-))
  • Both the proxy service and the DC agent support an in-place upgrade, so a newer version can be installed without uninstalling the old version.

Testing the Azure AD Password Protection service

So, after installing the Password Protection Proxy and the DC agent it’s time to test which is relatively simple. Logon to a domain-joined workstation, use CTRL-ALT-DELETE to change the password. When using a simple password like “Summer2019” or something it fails with the following error message.

Unable to update the password

From this moment on it is no longer possible to use weak passwords, locally enforced by Azure Active Directory and again a step closer to a safer environment.

Ignite 2018 – Azure AD and security sessions

A little later than originally planned because of an unexpected visit of the Massachusetts General Hospital in Boston on my way…. In my previous postings I blogged about the start of the conference and some of the Exchange sessions I attended in the first two days. Now how much I do love Exchange, most of my clients are moving towards Office 365 and Exchange Online, so what else is important here?

Yes, authentication! Azure Active Directory, Identity and Access Management and security around these solutions. And this happens to be important for Exchange and Exchange Online as well, so….

Secure access to Office 365/Azure Active Directory with new features in AD FS in Windows Server 2019 and Azure AD Password Protection

Sessions “BRK3226 – Secure access to Office 365/Azure Active Directory with new features in AD FS in Windows Server 2019 and Azure AD Password Protection” is all about authentication in Azure AD. It explains the traditional password hash sync as well as the ADFS options (more that 71 million users are actively using ADFS). But there are also 1.29 billion authentications blocked in August 2018 and 81% of all security breaches are because of weak, default or stolen passwords.

securing-resources

Common passwords used in (Azure) AD are Password, Spring, Summer, Autumn, Winter, 2018, 1234, your favorite football team etc. And these in turn are used in password spray attacks! Also vulnerable are passwords where number and letters are changed, for example “I” becomes “!”, “O” becomes “0” etc. And now you wonder, how many of my users are doing this? Password protection in Azure AD also includes normalization of the password, so these changes are automatically blocked. The good thing is, Azure AD password protection is coming to on-premises AD as well!

You can find the presentation on Youtube https://youtu.be/DC4cyF_JEgw and the presentation can be found here https://mediusprodstatic.studios.ms/presentations/Ignite2018/BRK3226.pptx

Azure Active Directory best practices from around the world

The title of the session was renamed to “Azure AD: Do’s and Don’ts”, but this is a more ‘notes from the field’ session with a lot of practical information around Azure AD, legacy authentication, modern authentication, Hybrid Azure AD Joine (HAADJ, I hate 3 letter acronyms, let alone 5 letter versions 😊) and what to do to get a better and more safe authentication experience.

legacy-authentication

Interesting in this presentation is that is also discusses what step you need to take to move from legacy authentication to modern authentication, and also the pitfalls you might encounter, including links to more information (found in the presentation).

associating-devices

You can see the presentation on Youtube https://youtu.be/wGk0J4z90GI and you can find the presentation here https://mediusprodstatic.studios.ms/presentations/Ignite2018/BRK3408.pptx

Scott Schnoll’s Exchange and Office 365 tips and tricks

I don’t know how many times Scott Schnoll has delivered this session, but it still is an awesome session and contains so much practical information around Exchange and nowadays Exchange Online.

scott-schnoll

I tried to make some pictures with Office Lens, but I think the color of the slides and text are not identified correctly so they are horrible. The slides aren’t available (yet), so you have to check the presentation on Youtube: https://youtu.be/0WNMX8EKYZk

Topics include anti-virus exclusions, DMARC enhancements, decommission on-premises Exchange in (or after) hybrid, changes to EOP IP ranges, migrating DL’s to Office 365 (including a script to do so), a license administrator in Office 365 (preview), DLP and credit card numbers and Mail Flow Insights, a new tool/dashboard that is currently being developed. Scott is doing a demo on this at the end of his presentation. Very cool, very promising, very useful!

Summary

So, after 5 days (well, four and a half days) we can say it was a very successful event. It is so huge, approx. 30,000 attendees from 5,000 organizations. So many sessions, break-out, theatre, workshop, hands-on, almost too much. And the sheer size of the location, I guess one can walk between 6 and 7 miles every day between the various locations. Would I go again? Sure, next year, again in Orlando, November 4-8. Hope to see you there!

More information/sessions

And some more interesting sessions to view online….

BRK2407 – Windows 10 and Office 365 ProPlus lifecycle and servicing update (CONDENSED)

https://mediusprodstatic.studios.ms/presentations/Ignite2018/BRK2407.pptx  https://youtu.be/t9Bs55czc1E

BRK3234 – An IT pros guide to Open ID Connect, OAuth 2.0 with the V1 and V2 Azure Active Directory endpoints (very informative, but not available online yet I’m afraid)

BRK3397 – Protect and control your sensitive emails with Office 365 Message Encryption

https://mediusprodstatic.studios.ms/presentations/Ignite2018/BRK3397.pptx

https://youtu.be/Ld4b4pFua0g

BRK3408 – Azure Active Directory best practices from around the world

https://mediusprodstatic.studios.ms/presentations/Ignite2018/BRK3226.pptx

https://youtu.be/wGk0J4z90GI

BRK3146 – What’s amazing and new in calendaring in Outlook!

https://youtu.be/-ZrNTylawOA

THR3024 – How to add MFA to your Exchange Online/on-premises mailboxes in 20 minutes or less

https://youtu.be/7hoEmEwV8Rk

BRK3081 – Implementing a modern network architecture to get the most out of Office 365

https://youtu.be/FGMzS_MjuPY

BRK3145 – Deploying Outlook mobile securely in the enterprise

https://youtu.be/4mHlxdJMh1Q

THR3036 – Azure Active Directory hybrid identity and banned password detection

https://youtu.be/kuVkfIiapI4