My organization never really paid attention to the password of the BladeLogicRSCD account, which always seemed to use the same default whenever we'd install an agent. It never posed any problems until we upgraded our platform to BBSA 8.1, and then 8.3.
My organization supports multiple customers, which have different networks and domains, and different Windows versions, and from time to time, we noticed that the BladeLogicRSCD account of some Domain Controllers would get locked out due to too many invalid logon attempts. After troubleshooting these issues in depth, it was found that the source came from other member servers (on the same domain) that were executing commands via BladeLogic to fetch account information that would need to query the domain. For example: "net user someuser /domain", or simply fetching the localgroups which contained domain groups in them.
This happens because the member server launches a command as its local BladeLogicRSCD user, and there is also a BladeLogicRSCD account on the domain controller of the domain, but it has a different password. The command therefore attempts to authenticate on the domain controller using the local BladeLogicRSCD account credential of the member server (which fails) and causes the domain account to get locked out after too many attempts depending on how the account lockout policy is configured on the domain. This leads to the following error when trying to access any of the replicated domain controllers of the domain: Login not allowed for user.
Up until what seems to be version 8.1, the BladeLogicRSCD default password always seemed to be the same, however I have recently tried to recreate the account (stop the service, delete the account, restart the service) on two agents of the same version (8.3.02.332) and they both ended up with different passwords. Only the chapw command done against both servers was able to set the same password for both accounts.
The only way I have been able to properly fix this, was to use the chapw command to set the same password on both the member server and the domain controller's BladeLogicRSCD account. We thought about using automation principals, but that requires the app server going through a NSH Proxy which considerably slows down the performance of NSH Script Jobs, so it's a no-can-do for us.
Bill Robinson also suggested to use the following procedure to resolve a similar issue: Preparing an alternate user name for the agents in the Windows system, which will indeed prevent the account from being locked out again on the DC, but I'm not entirely convinced this will not cause another issue.
For example, this will not prevent hundreds of member servers on which we run inventory scripts on to generate thousands of "Logon failure" events on the domain controllers because they try to use BladeLogicRSCD to query the domain. An example of a script that will cause this kind of issue is the popular XCACLS.vbs script provided by Microsoft (http://support.microsoft.com/kb/825751). The only way to prevent this is to allow the member servers to either login with BladeLogicRSCD (which requires all to have the same password), or to use automation principals so all member servers use a valid domain account (which requires the NSH Proxy).
That said, I was wondering what was the best practice regarding the password of this account? Should we be changing it to something specific for each of our customers using the chapw command? What is BMC recommending on this?