Recently I was implementing an Exchange 2010 UM server that was not willing to deliver any voicemail messages to the user’s inbox. On the UM server I was several EventID 1423 UMCore error messages in the application eventlog:

Log Name: Application
Source: MSExchange Unified Messaging
Date: 4-2-2013 14:53:25
Event ID: 1423
Task Category: UMCore
Level: Error
Keywords: Classic
User: N/A
Computer: EXUM02.contoso.com
Description:
The Unified Messaging server encountered an error while trying to process the message with header file "C:\Program Files\Microsoft\Exchange Server\V14\UnifiedMessaging\voicemail\2d831f7a-2a85-40f2-864c-70b4680a118f.txt". Error details: "Microsoft.Exchange.Net.ExSmtpClient.TlsApiFailureException: A TLS API failure occurred. Error = 0x80090301

At the same time I saw lots of EventID 36885 Schannel error messages in the system eventlog of the Hub Transport Server:

Log Name: System
Source: Schannel
Date: 4-2-2013 12:32:56
Event ID: 36885
Task Category: None
Level: Warning
Keywords:
User: SYSTEM
Computer: exhub01.contoso.local
Description:
When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

Note. As you may know the UM server records the voicemail message and the voicemail message is sent to the user’s mailbox using the Transport Server.

When looking at the Trusted Root Certification Authorities on the Hub Transport Server it turned out that there were 355 certificates stored here.

This is where things are breaking. The UM server is using TLS to communicate with the Hub Transport Server and during the handshake between the servers the list of root certs is sent. The maximum size of the package being sent by Schannel is only 16KB and the 355 root certificates never fit in these 16KB. Schannel fails, the list of certificates is truncated, resulting in EventID 36885 and the UM server only sees an invalid handshake with a truncated list of certificates and does not want to communicate.

So, the initial entry in the eventlog on the UM server is a result of a TLS issue and important to note, this is not an Exchange problem!

The way to solve this is delete a large number of root certificates from the Trusted Root Certification Authorities on all Hub Transport server. In my environment I reduced the number of root certs to about 85 which is sufficient. When the certificates are deleted the TLS handshake succeeds and the UM server starts sending the voicemails.

The question is how these 355 root certificates ended up in the trusted root store, a newly installed Windows 2008 R2 server in my test environment only reveals 11 certificates in the trusted root store.

Most likely the rootsupd.exe tool has been run sometime ago which updates the list of root certificates on the computer to the list that is accepted by Microsoft as part of the Microsoft Root Certificate Program. Now this is fine on a laptop or workstation, but you don’t want this to happen on your server because it can lead to unpredictable results.

This package however was released to Windows Update and WSUS on December 11, 2012 and was intended for client OS’es only. It also affected servers however and after customer reports the package was set to expired in Windows Update and WSUS.

For more information please check the following knowledge base articles.

• http://support.microsoft.com/kb/931125 – Windows root certificate program members
• http://support.microsoft.com/kb/2801679 – SSL/TLS communication problems after you install KB 931125
• http://www.microsoft.com/en-us/download/details.aspx?id=3867 – Update for Root Certificates [May 2010] (KB931125) <– note. Don’t run this on your server 😉