Mysterious Issue Threatens to Derail Windows Server Upgrade

It’s not often our Microsoft team gets thrown by an issue they encounter at a customer’s site. When it does happen, no white flags are waved and no hands are thrown in the air. Instead, they embrace the challenge and get right to work learning everything they can about the issue so they can solve it.

Case in point: While he was working on upgrading a client’s Active Directory from Windows Server 2003 to 2012 R2, one of our Microsoft engineers ran into an issue after he promoted the first 2012 R2 domain controller. That domain controller refused to replicate properly with the existing 2003 domain controllers. The ADPrep commands had all run without issue and the domain controller had promoted properly, but replication wouldn’t occur.

The error messages he received included:

• Active Directory Domain Services did not perform an authenticated remote procedure call (RPC) to another directory server because the desired service principal name (SPN) for the destination directory server is not registered on the Key Distribution Center (KDC) domain controller that resolves the SPN.
• The attempt to establish a replication link for the following writable directory partition failed.

Upon further investigation, the following error was found in the system event log: The Security System could not establish a secured connection with the server LDAP/ServerName.Domain.com/Domain.com@Domain.com. No authentication protocol was available.

Essentially, when the new 2012 R2 domain controller was promoted, there was very minimal communication between it and the preexisting 2003 domain controllers. In a normal upgrade, information would flow easily between these two types of domain controllers. Since it wasn’t, it meant the new domain controller wasn’t properly replicating changes happening in the environment.

This combination of communication issues and error messages was one the engineer hadn’t come across before. After consulting with other members of our Microsoft team, who were also unfamiliar with it, he began to do some sleuthing.

Correcting the domain controller issues

During his search, our engineer found a few Microsoft knowledge base articles online detailing similar error messages. However, these articles weren’t dealing with replication issues. While the error messages may have been similar, the bigger picture problems were things like not being able to remote desktop into a server. The fact that the error messages were so similar led him to look more closely at what the articles had to say.

The posts pointed to an issue with the Active Directory KRBTGT account stemming from an authoritative restoration on the Users container. The KRBTGT account is a service account used by the Kerberos Key Distribution Center (KDC) for encryption purposes. Problems can occur when the version number of the KRBTGT account increases following an authoritative restoration. If a version number reaches a certain level, it can cause miscommunication between domain controllers.

To test the theory, the tech ran a command in the customer’s test environment to see the version number of the KRBTGT account of the 2003 domain controllers. Sure enough, the version number on the KRGTBT account was extremely high. This indicated that an authoritative restore, likely even multiple authoritative restores, had been performed on the account.

Once the root cause of the issue was tracked down, the fix itself was actually fairly easy. A hotfix from Microsoft was installed on all of the 2003 domain controllers, which allowed the 2012 R2 domain controller to recognize the validity of the 2003s. Once complete, the new 2012 R2 domain controller was able to replicate successfully and there were no further problems with the upgrade.

The Arraya Promise

We take great pride in the depth of knowledge that our Microsoft practice has cultivated over the years. Our know-how hasn’t gone unnoticed either as we’re a two-time winner (2014, 2015) of Microsoft’s Mid-Atlantic Area “Rising Star” award. However, it’s inevitable that, on occasion, a previously-unknown issue is going to crop up. We don’t pretend like our Microsoft team has seen all there is to see. What we do promise is that, if they do encounter something new, they will work tirelessly to find a solution.

In this case, delivering our signature superhero service required us to do some detective work first. Once the pieces of the puzzle had been put together, we were able to get back to the mantra we’ve laid out for every customer engagement: Go above and beyond to help them succeed.

To learn more about how our Microsoft team can help your organization, please visit www.arrayasolutions.com/partnership/microsoft/. If you’re ready to strike up a conversation about Microsoft, or any one of our other specialty areas, click here.

Also, be sure to follow us on Twitter, @ArrayaSolutions, to keep up with all of our latest company news, special offers and insights.