Introduction

In this blog post, I’ll walk you through an interesting case we encountered where a financial transaction processing application, Postilion, failed after the customer applied security patches to their domain controllers. The problem was critical, affecting their entire environment, and the resolution required a deep understanding of authentication protocols and timely action.

Application and Impact

The customer reported that after patching their domain controllers, Postilion, a transactional processing software used by financial institutions, started failing. Postilion relies on Windows authentication to access its database, and the failures caused a major disruption in their environment.

The system administrators initially suspected authentication issues, and they had already spent significant time rolling back and reapplying patches to troubleshoot the issue. Compounding the problem was the fact that the domain controllers hadn’t been patched for several months, leaving the system vulnerable to security threats.

Evidence and Root Cause Analysis

A root cause analysis revealed several key clues. The security event log showed multiple 4625 events, which indicate failed logon attempts. These failures occurred when the system attempted to use the NTLM authentication protocol. The error pointed to a wrong password as the reason for the logon failure, which seemed unusual given that the password hadn’t been changed.

• Event ID: 4625
• Failure Reason: Unknown username or bad password
• Status: 0xC00006D
• Sub Status: 0xC000006A
• Authentication Package: NTLM

At the same time, the database logs showed a similar authentication error with event 17806, stating that the connection handshake failed due to an operating system error.

• Database Event: 17806
• Error: 0x8009030c (SEC_E_LOGON_DENIED)
• Message: The logon attempt failed.

Authentication Flow

To better understand the problem, let’s look at the authentication flow in this setup.

1. The servicespostilion service account runs on two application servers, which communicate with two dedicated SQL database servers.
2. The authentication method between Postilion and the database servers is Windows authentication using NTLM.
3. The service account sends jobs to the database server, which works smoothly until domain controllers are patched.

The typical NTLM authentication flow is as follows:

• Step A: The servicespostilion service sends the username to the database server.
• Step B: The database server sends back a challenge (nonce).
• Step C: The service responds with a challenge-response.
• Step D: The database server sends the username, challenge, and challenge-response to the domain controller for verification.

Action Plan

Given the impact on a production financial system, our priority was to resolve the issue without rolling back the patches and to address the security risks posed by unpatched domain controllers.

The customer scheduled their monthly maintenance window, and we prepared for a night of troubleshooting. I had a theory that the issue was related to NTLM compatibility level mismatches between the domain controllers and the application servers. If that was indeed the case, adjusting the settings should resolve the problem.

However, I also had a backup plan to switch to Kerberos if NTLM continued to fail.

In case you are not familiar with the NTLM protocol and its versions

NTLM (NT LAN Manager) is an authentication protocol used in Microsoft environments. While newer systems favor Kerberos, NTLM is still required in legacy scenarios. NTLM works via a challenge-response mechanism where a client sends a username, receives a challenge (nonce) from the server, and responds with a hashed password. The domain controller then verifies the response.

LMCompatibilityLevel and NTLM Versions

NTLM has two main versions: NTLMv1 (less secure) and NTLMv2 (more secure). The LMCompatibilityLevel setting determines which version is used. Higher values enforce stronger security. For example:

• 0: Accepts both LM and NTLM.
• 3: Uses NTLMv2 for authentication.
• 5: Uses only NTLMv2, rejecting older versions.

Configuring NTLM via Group Policy

You can manage NTLM through Group Policy:

1. Open Group Policy Management.
2. Go to Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options.
3. Set Network security: LAN Manager authentication level to 5 for NTLMv2.

Configuring NTLM via Registry

For smaller setups, use the Registry:

1. Open regedit and navigate to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa

2. Modify LMCompatibilityLevel (DWORD) and set it to 5.

NTLM version compatibility

NTLM versions have varying degrees of compatibility, which can lead to authentication failures if misconfigured:

• NTLMv1: This older version uses a less secure challenge-response mechanism. It is compatible with servers accepting NTLMv1 or LM responses (lower security).
• NTLMv2: A more secure version, required by higher security levels. It can authenticate with servers set to accept NTLMv2, but is incompatible with servers or clients set to use only NTLMv1 or LM.
• LM (LAN Manager): This is the oldest and least secure method. Modern systems reject it if they are set to enforce NTLMv2 only.

For example, if a domain controller is configured for NTLMv2 only (LMCompatibilityLevel 5) but a client is using NTLMv1 (LMCompatibilityLevel 1 or 2), authentication will fail. Both client and server need to be on compatible NTLM versions for successful authentication.

Resolution

Once the servers were patched, I joined the customer’s troubleshooting session and was able to reproduce the failure immediately. The NTLM compatibility level was the first thing I checked, and my suspicions were confirmed. The LMCompatibilityLevel setting was mismatched:

• Application servers (Postilion): NTLM compatibility level set to 1.
• Domain controllers: NTLM compatibility level set to 5.

This mismatch was causing the challenge-response mechanism to fail, resulting in the “bad password” error. Interestingly, the application worked fine before patching, but this discrepancy caused issues after the patches were applied.

We changed the compatibility level on the application servers to 3, aligning it with the domain controllers. The system immediately started functioning again, and the authentication failures were resolved within 30 minutes. This not only restored functionality but also enhanced the security of the application by using a more secure NTLM version.

Lessons Learned and Final Thoughts

While we always strive to understand every detail of why something works or doesn’t, some nuances remain unexplained. In this case, the fact that the application worked with unpatched domain controllers but failed after patching is still somewhat mysterious. However, in production environments, the immediate priorities are security and making the system work.

For this case, we succeeded in both: we resolved the problem without rolling back patches and strengthened security by using a higher NTLM version. That said, there is always room for improvement. In the future, switching to Kerberos, the industry-standard authentication protocol, would be a better long-term solution.

Conclusion

This case highlights the importance of proper authentication configurations and the potential impact of security patching on enterprise applications. By aligning the NTLM settings across systems, we were able to quickly resolve the issue and ensure the continued operation of a critical financial application. It serves as a reminder to review and understand the authentication mechanisms in your environment, especially when applying patches to domain controllers.

If you’re running applications that rely on NTLM authentication, now might be a good time to review your LMCompatibilityLevel settings across your environment or even start thinking about moving them to Kerberos. Have you experienced similar issues post-patching? Drop a comment below or share your story

References

• Network security: LAN Manager authentication level.
• NTLM deprecation

A few moons back, I found myself wrestling with an interesting case that gave me a fresh lesson on SSL and security providers. The customer I was working with had a vulnerability scanner (Rapid7, in this case) that flagged their server as vulnerable to Sweet32. If you’ve ever had to deal with vulnerability scans, this might sound familiar!

Let me walk you through the scenario, the fix, and the curveball that a thrid party process threw at us.

The Vulnerability Report

Our story begins with a standard security scan, which operates by sending multiple “client hello” messages to the server with different cipher suites. The server responds with a “server hello” for every cipher it accepts—secure or not. The scanner then flags anything it sees as insecure. Enter Sweet32.

What is Sweet32?
Sweet32, or the birthday attack, targets encryption algorithms like 3DES or Blowfish that use 64-bit block sizes. It’s kind of like having a lock on your door, but after enough time, the key pattern starts repeating itself. When a cipher’s encrypted blocks start repeating, an attacker can piece together sensitive data. In high-traffic environments, this is a big no-no. For modern security, we need to switch to stronger encryption algorithms like AES that use larger block sizes and avoid this vulnerability.

The First Fix: Disable 3DES Cipher Suite

The customer’s server was flagged for accepting the 3DES cipher, a known weak link in the encryption chain. So, the first logical step was to configure the server to stop allowing this cipher by tweaking both local and Group Policy-controlled registry settings.

We tweaked the following registry keys:

• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL
• HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\0010002
• HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\0010002

The server was rebooted, and we expected the issue to be resolved. But, spoiler alert—it wasn’t.

The Mystery Continues: False Positive or Missed Configuration?

After the reboot, the vulnerability scan still flagged the 3DES cipher as enabled. We thought, “Okay, it must be a false positive.” But network traces said otherwise. Despite our changes, the handshake process was still completing using the 3DES cipher.

So, we went back to square one—double-checking policies, reviewing registries, running the scan again. Everything seemed in place for the cipher to be disabled. Yet the vulnerability persisted.

The Sherlock Moment: Different Port, Different Story

Eventually, we started noticing a pattern. While port 443 (used for HTTPS connections) rejected the 3DES cipher as expected, port 5989—now that was a different story. The cipher was still being accepted there!

A quick internet search didn’t turn up anything definitive about port 5989, but we were determined to solve the mystery. So, we took a deeper dive using the trusty Netstat -anob and Tasklist commands. And guess what? We found the culprit: WMIServer.exe, owned by HPE at the time.

The Real Lesson: More Than Just Schannel

Here’s the big takeaway: Windows servers can have more SSL credential providers than just Schannel. In this case, WMIServer.exe was accepting the weak 3DES cipher, and it had its own SSL settings independent of Schannel.

So, when you’re dealing with SSL vulnerability scans like those from Nessus, QualysGuard, or OpenVAS, don’t forget to check all the SSL credential providers, not just Schannel. There might be other services on your server accepting those insecure ciphers without you even realizing it.

How to Identify the Process Listening on a Port

In case you ever find yourself in this situation, here’s a quick guide:

1. Collect a Wireshark Network trace while you are running the scanner
2. After the scanner report is done, start analyzing the network trace, you can use the filter “tls.handshake.type == 2” to show server hello only packets
3. Expand the transport layer security and find the cipher suite included, look for any that you would like to eliminate
4. Take note of the port in your local server, that port number is being used by a process that is accepting the vulnerable connection
5. Run netstat -anob to see all active connections, the associated process IDs, and the listening ports.
6. Use tasklist to match the process ID to the name of the running process.
7. Optionally, use Process Monitor (ProcMon) or Windows Performance Recorder (WPR) with Windows Performance Analizer (WPA) to get the name of the company that owns the process (but usually you don’t need to go this far).

Conclusion & Call to Action

The moral of the story? Never assume Schannel is the only SSL provider at play. Windows can have more lurking in the background, especially with third-party software involved.

If you’re facing a similar vulnerability scan issue and are banging your head trying to figure out why your configurations aren’t sticking, consider checking for other SSL providers on your system. The solution might be right there in front of you!

Have you encountered something similar with SSL configurations or vulnerability scans? Drop your thoughts in the comments, and let’s discuss!

References

• Sweet32 info
• Speaking about ciphers

Welcome back to Master in Progress! Today, we’re diving into a topic that might feel like your machines have a bit of a rebellious streak: Group Policy Object (GPO) registry tattooing. Yes, it’s real, and no, it’s not as cool as it sounds. So, what’s the deal with these “tattoos” your GPOs are leaving behind, and why won’t they just go away?

What is GPO Registry Tattooing?

Let’s start with the basics.

GPO (Group Policy Object) registry tattooing occurs when a GPO applies specific settings to a machine’s registry, and when that GPO is removed, the registry settings aren’t automatically reset to their default values. The settings are “tattooed” onto the system, sticking around like a bad tattoo from your rebellious teenage years.

This can be especially persistent when it comes to security settings, as these settings are stored in a local database on your computer. Whenever a security setting is applied via GPO or local policy, it gets saved in this database. Even if the policy is later removed or unlinked, the security settings don’t automatically revert to previous values unless they were explicitly defined beforehand. If there’s no previous value in the database, the setting remains stuck with its last defined value, a behavior commonly referred to as “tattooing.”

Why Does This Happen?

Normally, when a GPO is removed, you expect the system to revert to its pre-GPO state. However, certain registry keys modified by the GPO remain unchanged unless specifically instructed otherwise.

Here’s why:

• One-Way Ticket: Some GPO settings are designed to apply a value but not revert it. Once the setting is tattooed into the registry, the GPO doesn’t include instructions for what happens when it’s removed.
• No Cleanup Logic: In some cases, the GPO doesn’t have logic to revert or reset the change after it’s no longer applied. The settings are left behind, sitting in the registry like an awkward guest who didn’t realize the party was over.

How to Avoid or Mitigate GPO Tattooing

Just like avoiding a bad tattoo, you can avoid GPO registry tattoos with careful planning. Here’s how to prevent or fix tattooed settings:

1. Set the GPO to the Desired Value Before Removal
   Instead of simply deleting or disabling a GPO, set the policy to the value you actually want. By explicitly setting the correct value before unlinking or removing the GPO, you ensure the desired state is applied. This avoids leaving an undesired setting stuck in place.
2. Run Cleanup Scripts
   If tattooing has already occurred, scripts can be a lifesaver. Use PowerShell scripts to manually reset the registry keys that were modified by the GPO. This is the manual “tattoo removal” you need to bring your systems back to a clean state. Example PowerShell snippet to reset a specific registry key:

   Set-ItemProperty -Path 'HKLM:\Software\Policies\' -Name 'YourKeyName' -Value 0

3. Leverage GPO Removal Features (Where Available)
   For certain policies, there may be an option to specify behavior on removal. If available, use this feature to automatically undo the settings when the GPO is no longer applied.

How to Identify Tattooed Registry Settings

Now that you’ve removed a GPO but suspect a lingering setting, how can you find the “tattooed” registry key? Here’s a practical approach:

1. Identify the Misbehaving Behavior
   First, pinpoint the behavior or setting that seems to be stuck. Let’s use an example where your system is still enforcing a custom SSL cipher suite order, even though you’ve removed the GPO that configured it. This could cause issues with SSL/TLS negotiations that are out of sync with your current security requirements.
2. Use Websites Like ADMX Help to Find Corresponding Registry Keys
   Now that you’ve identified the misbehaving setting, the next step is to track down the registry key associated with the GPO. ADMX Help is a great resource for this task. Let’s walk through how you can find the relevant registry key for the SSL Cipher Suite Order policy. Example: Tracking Down the Registry Key for SSL Cipher Suite Order Policy

• Step A: Navigate to ADMX Help.
  Head over to ADMX Help in your browser. This site provides a detailed breakdown of ADMX policies, including the registry paths and keys that they modify.
• Step B: Search for the GPO setting.
  In this case, you want to look up the SSL Cipher Suite Order policy. In the search bar, type “SSL Cipher Suite Order” or directly go to this link: ADMX Help – SSL Cipher Suite Order.
• Step C: Review the Policy Details.
  Once you land on the page for the SSL Cipher Suite Order policy, you’ll find all the critical information:
  • The policy name as it appears in the Group Policy Management Editor: “SSL Cipher Suite Order.”The ADMX file where this policy is defined: cipherstrength.admx.The registry path and key:
  Registry Key: HKLM\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002 Value Name: Functions
  • The value this policy sets: a long string that specifies the order of SSL ciphers (for example, “TLS_RSA_WITH_AES_128_CBC_SHA256, …”).
• Step D: Check the Registry.
  With this information, open the Registry Editor (regedit.exe) on the machine where the GPO was previously applied. Navigate to the following path:
  HKLM\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002
  Inside, look for the Functions value. If it’s still present with the custom cipher order, even though you’ve removed the GPO, then this is your tattooed setting. The machine is still enforcing this order because the registry key wasn’t reverted when the policy was removed.

3. Modify or Remove the Registry Entry
   Now that you’ve found the tattooed setting, you can manually modify the Functions value to reflect the default or desired cipher suite order. Alternatively, if you no longer want any custom order applied, you can simply delete the key or reset it to a neutral value. Example of removing the setting via PowerShell:

   Remove-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002' -Name 'Functions'

4. PowerShell for Quick Queries
   To check for this tattooed setting on multiple machines, you can use PowerShell to quickly query the registry key. Here’s an example:

   Get-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002' -Name 'Functions'

This will return the current value of the SSL Cipher Suite Order if it’s still applied. You can then decide whether to reset, modify, or remove it based on your requirements.

Conclusion

GPO registry tattooing might sound intimidating, but with tools like ADMX Help, identifying lingering registry keys becomes much easier. By taking the time to properly remove or reset these settings, you can ensure your systems aren’t following outdated rules.

Remember, always check the registry for tattooed settings and use resources like ADMX Help or Group Policy Search to track down the corresponding keys. For persistent policies, like the SSL Cipher Suite Order, manually resetting the registry key is key to keeping your systems secure and up-to-date.

Has GPO registry tattooing caused you trouble in the past? Share your experiences in the comments below, check other posts in Master in Progress for more tips, tricks, and insights into mastering Active Directory and beyond!

References

• Security policies

If you’re considering adding a load balancer between your clients and domain controllers (DCs), stop right there! While load balancers can work wonders in many networking scenarios, Active Directory (AD) isn’t one of them. Here’s why this setup is more trouble than it’s worth. We’ll dive into five detailed reasons why you should let AD do its job without getting a load balancer involved.

1. Active Directory’s Built-In Load Balancing Is All You Need

Active Directory already comes with robust tools that handle load balancing across domain controllers—so why complicate things with a load balancer? Let’s break this down:

• Sites and Services Snap-in: AD allows you to set up sites and assign DCs to each one. This lets you group your DCs based on geographical regions or network topology. You decide which DCs are responsible for which site, ensuring that users connect to the closest one for optimal performance. It’s like setting up traffic signs that direct users to the nearest gas station.
• Subnets Linked to Sites: AD lets you map subnets to sites. This way, a client in New York won’t accidentally connect to a DC in California—unless you want it to. This is a seamless, automatic process that reduces latency and improves login speed for users.
• DNS Priorities and Weights: When you have multiple DCs within a site, AD uses DNS records to balance the load. It assigns priorities and weights to each DC, distributing client connections evenly. It’s like having multiple checkout lines at a grocery store, and a store manager directs people to the least busy line. There’s no overcrowding, and everyone gets served efficiently.

Why add a load balancer? AD already handles the heavy lifting here. Adding an external load balancer just adds complexity and expense with no real benefit.

2. TCP Communication: Don’t Mess with the Three-Way Handshake

When it comes to TCP connections, load balancers can wreak havoc. Every TCP connection begins with a “three-way handshake,” and if the load balancer decides to switch servers mid-communication, things fall apart fast.

• TCP Handshake Breakdown: TCP requires this handshake to establish a reliable connection. If your load balancer moves the client to a different DC halfway through, the new DC will not recognize the connection because it wasn’t part of the original handshake. This leads to dropped packets, timeouts, and user frustration. Imagine you’re ordering a pizza, and halfway through the call, your phone suddenly connects you to a completely different restaurant. You can’t expect them to know your order—they weren’t part of the conversation from the start!
• TCP Reset Floods: Additionally, if the load balancer aggressively switches DCs, it can lead to TCP reset storms where one DC keeps closing connections that it didn’t start. This can overwhelm your DCs and affect network performance.

Takeaway: AD’s native site-based load balancing works with TCP’s needs, while a load balancer could disrupt the connection flow, causing delays and dropped communication.

3. Kerberos Doesn’t Like Load Balancers Either

Kerberos, AD’s go-to authentication protocol, is a sensitive beast. It doesn’t play well with load balancers for one simple reason: tickets. When a client authenticates, Kerberos gives it a ticket, which is essentially a “permission slip” to access resources via a specific DC. But if your load balancer reroutes the client to a different DC, that ticket becomes invalid.

• Ticket Mismatch: Kerberos tickets are encrypted with the original DC’s key. When a client presents a ticket to a different DC, that DC will reject it because it can’t decrypt the ticket. It’s like showing up to a concert with a ticket for a completely different venue—the bouncer won’t let you in.
• Sticky Sessions?: You could configure Sticky Sessions on the load balancer to ensure the client sticks to the same DC for the entire session. But this workaround only adds more complexity. If the sticky session fails, you’ll still have ticket decryption issues.
• Performance Hits: Every time a Kerberos ticket is rejected due to mismatched DCs, the client has to request a new ticket from the Key Distribution Center (KDC). This adds overhead to both the client and DC, ultimately affecting performance.

Takeaway: Kerberos doesn’t like surprises. When you throw a load balancer into the mix, it becomes a guessing game of which DC holds the right ticket.

4. SSL/TLS Handshake for LDAPS: The Certificate Nightmare

If you’re using LDAPS (LDAP over SSL/TLS), things get even messier with a load balancer. Why? Because SSL/TLS handshakes depend on certificate matching, and your load balancer can break this chain of trust.

• Certificate Subject Name Mismatch: SSL/TLS requires that the server’s certificate matches the hostname the client is trying to connect to. So, if your client is trying to connect to dc1.domain.com but your load balancer routes it to dc2.domain.com, the client will receive a certificate mismatch warning. The SSL handshake will fail, and the connection won’t be established.
• Managing SAN Certificates: One workaround is to create a Subject Alternative Name (SAN) certificate, which lists all your DCs in the certificate’s SAN field. This works, but if you have many DCs, your certificate could end up with a dozen or more entries. Keeping this updated as DCs come and go can become a headache. It’s like trying to maintain a guest list for a wedding when people keep RSVPing at the last minute.
• Wildcard Certificates: Not a Great Solution: Alternatively, you might consider using a wildcard certificate (*.domain.com), but this is a security risk. If that wildcard certificate is ever compromised, it could be used to impersonate any server in your domain—not just your DCs.

Takeaway: SSL/TLS handshakes require exact matching between the certificate and the hostname. A load balancer just complicates this, and the alternatives aren’t much better.

5. LDAP Channel Binding: Don’t Let Your Load Balancer Break the Chain

LDAP channel binding is a security feature that was designed to prevent man-in-the-middle attacks, and guess what? Your load balancer can mess this up too, especially if it’s doing SSL offloading.

• SSL Offloading Breaks Binding: With SSL offloading, the load balancer terminates the SSL connection and then re-establishes it with the DC. While this might improve performance, it also breaks the integrity of the SSL session, effectively negating channel binding. The client and DC are no longer directly communicating, which opens up the possibility for tampering.
• Security Risk: The whole point of LDAP channel binding is to ensure that the client and DC have a direct, secure connection. By offloading SSL, the load balancer inserts itself in the middle, which is exactly what channel binding is supposed to prevent.
• Troubleshooting Pain: Once SSL offloading breaks channel binding, tracking down the issue can be a nightmare. You might be scratching your head wondering why perfectly valid SSL certificates are failing, only to realize that your load balancer is breaking the chain.

Takeaway: SSL offloading may seem like a good idea to reduce load on your DCs, but it breaks a crucial security mechanism—LDAP channel binding.

6. Account Lockouts: The Detective’s Nightmare

One of the biggest challenges with load balancers is account lockouts. Specifically, troubleshooting them becomes incredibly difficult when your load balancer doesn’t preserve the client’s IP address.

• No IP Preservation, No Clues: Without IP preservation, all your DCs see is the load balancer’s IP address. So, when a client is repeatedly sending bad login attempts, you have no idea where those attempts are really coming from. It’s like trying to solve a crime when all the witnesses give you the wrong description of the suspect.
• Bad Logon Attempts: Imagine a user typing the wrong password several times and getting locked out. With no IP preservation, all you see is the load balancer’s IP, so you’re stuck playing detective to figure out which user is at fault. This can be especially tricky if the bad logon attempts are coming from multiple devices or locations.
• Best Practice: If you must use a load balancer, at least configure IP preservation. This ensures that each DC sees the client’s actual IP address, making troubleshooting much easier.

Takeaway: Account lockouts are hard enough to troubleshoot without a load balancer getting in the way. If you’re not preserving IP addresses, you’re adding unnecessary complexity to an already frustrating process.

7. If Your Application Needs a Load Balancer to Communicate with Active Directory, It Wasn’t Built for AD

Let’s be real—if your application requires a load balancer to talk to Active Directory, then it probably wasn’t designed with AD in mind. And that’s a red flag! Here’s why:

• AD-Aware Applications Don’t Need Load Balancers: Any well-designed application that integrates with Active Directory should already know how to leverage AD’s built-in load balancing mechanisms. AD’s DCLocator function does this heavy lifting automatically. DCLocator ensures that your application finds the most appropriate domain controller based on the client’s location and network configuration. It’s like having a built-in GPS that always directs you to the nearest and least busy DC.
• Developer Oversight: If the application’s developers didn’t consider DCLocator when building the app, then it’s likely they also missed other important Active Directory best practices. This could lead to more issues down the road, such as poor performance, security vulnerabilities, or broken authentication flows.
• Microsoft’s Stance: Here’s the kicker—Microsoft doesn’t officially support load balancers between clients and domain controllers. This should be a clear sign that it’s not a recommended practice. When something breaks (and it will), you might find yourself in a support dead-end, as Microsoft won’t offer assistance for a setup that includes load balancers in the middle of AD traffic.

Takeaway: If your application relies on a load balancer to communicate with AD, it wasn’t built with AD in mind. You’re better off finding or developing applications that natively support Active Directory’s load balancing and fault tolerance features.

Final Thoughts: Let Active Directory Do Its Thing

In conclusion, while it may be tempting to introduce a load balancer into your AD infrastructure, it’s often more trouble than it’s worth. AD has built-in mechanisms like Sites and Services, DNS priorities, and the DCLocator function to handle traffic distribution and fault tolerance. Introducing a load balancer can disrupt TCP communication, Kerberos authentication, SSL/TLS handshakes, and LDAP channel binding, and make troubleshooting account lockouts a nightmare. And let’s not forget—if your app needs a load balancer to talk to AD, it’s probably not built with AD best practices in mind.

Call to Action: Have any stories of load balancers causing havoc in your AD environment? Drop them in the comments below, or feel free to reach out for help optimizing your AD infrastructure. Let’s keep it simple—let AD do the job it was designed to do!

References

• Microsoft statement about Load balancers and DCs
• “Simple” load balancer configuration

Picture this: You’re migrating a user from one Active Directory (AD) domain to another. All seems to be going well until you hit a roadblock: the dreaded error message “The maximum size of an object has been exceeded.” Let’s break down how to solve this, step by step, with a dash of humor and a ton of technical insight.

Identifying the Root Cause: SID History Strikes Again!

When dealing with Active Directory migrations, there’s one attribute that loves to throw a wrench in your well-oiled machine: SIDHistory. In this case, the error was triggered because the SIDHistory attribute had grown way too large, thanks to multiple migrations over the years. Think of it like your email inbox after 10 years without clearing out spam—it gets bloated!

• SIDHistory: This attribute stores the Security Identifiers (SIDs) of a user from their previous domains. Handy for keeping access during a migration but can quickly become oversized if left unchecked.
• The problem: Users with an excessively large SIDHistory attribute exceeded the maximum object size that AD could handle.

Access Denied: ADSIEdit and the Locked SIDHistory

Now, you might think, “No problem, I’ll just fire up ADSIEdit and clean up the SIDHistory manually.” But when you try, you get another lovely error:

“Operation failed. Error code: 0x5 Access is denied. PROBLEM 4003 INSUFF_ACCESS_RIGHTS”

Why? Well, SIDHistory is a system-controlled attribute. Even with administrative rights, AD locks you out from directly modifying this attribute.

The Power of PowerShell: Cleaning Up SIDHistory

Here’s how I resolved the issue:

1. Set up the lab: Always a good practice when you’re testing something new or potentially destructive.
2. Run the PowerShell command: To remove all the SIDs from the SIDHistory attribute, I used the following command:

Get-ADUser migUser -Properties sidhistory | ForEach-Object {Set-ADUser $_ -Remove @{sidhistory = $_.sidhistory.value}}

1. This command retrieves the user (in this case, migUser) and removes all SIDHistory entries.
2. It works like a charm because it handles the system-controlled attribute in a way that AD approves of—no more access denied errors.

Mission Accomplished: Migrating Users with a Clean Slate

Once we removed the bloated SIDHistory, we were able to successfully migrate the user to the new domain without any issues. And since the customer wanted the users’ SIDs migrated to the new domain, we ensured that their SIDHistory was clear to start fresh.

Key Takeaways

• Watch out for large SIDHistory attributes: If a user has been migrated multiple times, this attribute can cause migration failures. There are other attributes that can also impact this, like mSMQSignCertificates, mSMQDigests, UserCertificates, etc.
• ADSIEdit isn’t always your friend: Don’t expect to edit system-controlled attributes like SIDHistory directly, even as an admin. There are other ways to edit system-controlled attributes that I will cover in a future blog post.
• PowerShell to the rescue: Use the right tools for the job. PowerShell is a powerful ally in dealing with AD challenges.

Next time you face an AD migration, remember to check the size of the SIDHistory attribute, especially for users with a complex migration history. Share your own war stories with Active Directory in the comments. Let’s learn together!

Happy migrating, fellow techies!

References

• System-Only attributes
• sidHistory attribute

Table of Contents

Ever had a situation where everything seems to be secure, but sneaky old passwords still work their magic? Recently, we dealt with a curious case from a customer who uses CyberArk to rotate admin credentials daily. Sounds like a solid plan, right? The admins would grab their shiny new password at the start of the day and use it for their daily tasks. Simple, clean, and secure. Or so it seemed.

The Problem: The Ghost of Passwords Past

The issue? Administrators found out they could still use yesterday’s password to elevate tasks on their local machines. They could open a terminal or run any app as admin, using not just the freshly-minted password but also the previous day’s credentials.

The customer was understandably not thrilled. They wanted admins to use only the new password—yesterday’s password needed to stay in the past. But why was the old password still working?

Digging into Event Logs: Sherlock Mode On

Cue our deep dive into the event logs. We found that when admins were using the old password to elevate their tasks, no call was being made to the domain controller. A local authentication was happening instead.

Here’s what we saw in Event ID 4624 (Successful Logon):

SubjectUserSid         S-1-5….
SubjectUserName        SomeUser
SubjectDomainName      SomeDomain
SubjectLogonId         0x3e7
TargetUserSid          S-1-12-1-….
TargetUserName         alex@contoso.com
TargetDomainName       SomeDomain
TargetLogonId          0x123456
LogonType              11
LogonProcessName       User32
AuthenticationPackage  Negotiate

The key here is LogonType 11: CachedInteractive, which means the user logged in using credentials that were stored locally, and no one bothered to check in with the domain controller.

The Root Cause: Cached Credentials Aren’t Playing Fair

After some research (and maybe a few cups of coffee), we discovered this is actually default behavior for elevated tasks. Cached credentials aren’t updated when you run an elevated task with a different account. Even if you change the admin’s group membership in the domain, the cached data on the machine remains, letting the old password through.

The Fix: Taming the Cached Credentials

Luckily, we found a solution! By tweaking the registry, we can force Windows to verify credentials with the domain controller before allowing elevation. Here’s the magic PowerShell script we used:

# Define the path to the registry subkey
$registryPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System"

# Check if the registry path exists, if not create it
if (-not (Test-Path $registryPath)) {
    New-Item -Path $registryPath
}

# Define the new registry key and set its value
$registryKey = "InteractiveLogonFirst"
New-ItemProperty -Path $registryPath -Name $registryKey -PropertyType DWORD -Value 1 -Force

# Confirm the value has been set
$registryValue = Get-ItemProperty -Path $registryPath -Name $registryKey
Write-Host "Registry key '$registryKey' set to value:" $registryValue.$registryKey

What’s Happening Here?

• $registryPath: Points to the right spot in the Windows registry.
• Test-Path: Ensures the registry path exists before creating anything.
• New-ItemProperty: Creates a new DWORD (32-bit) registry value called InteractiveLogonFirst and sets it to 1.
• Get-ItemProperty: Checks that the value was set correctly.

Setting InteractiveLogonFirst to 1 forces Windows to verify with the domain controller for any elevation task. If the domain controller is unavailable? Well, cached credentials are still used.

But Wait… The “Admin Workaround”

Administrators are clever creatures. If they disconnect from the network, they can still use cached credentials. You can counter this by configuring the machine to not use cached credentials at all. But do you really want to go to war with your admins? Remember, they are the ADMINISTRATORS.

Final Thoughts and Call to Action

This tweak could save you from a lot of headaches if you’re rotating admin passwords and want to avoid yesterday’s ghosts popping up where they’re not wanted. It’s a simple registry fix, but it can have a big impact on how credentials are handled in your environment.

Have you run into any strange caching behaviors in your environment? Or maybe you’ve got an interesting workaround? Share your experiences in the comments below!

Stay secure, stay smart, and don’t let yesterday’s passwords haunt you!

References

• Default behavior for cached credentials
• Logon types