This posting is a bit different from my first couple here on the Communities pages, but I promise, there is a purpose.


Digital certificates are rather ubiquitous these days.  We use them in so many online transactions and interactions they've become notable only in their absence when we expect them.  I will avoid providing examples for lapses organizations have made in these areas; however, there are many from which to choose.


Most people in IT are familiar with the need for digital certificates for providing security to web servers in the form of https:// URL prefixes.  But so many in IT are not tasked with requesting, obtaining, installing, and maintaining those certificates that we can forget their need for other areas critical to security.  This posting is intended to provide a practical discussion on some of that process; however, it will be restricted to some of the requirements for X.509 digital certificates with Remedy.


First, for those that may want to refresh themselves on some of the major concepts in more detail, these resources may give you a start on the terms, and their detailed explanations.  This posting is more "trade school" than "classroom". 


And to any cryptography/security experts:  because I am not going to cover some areas in detail, please forgive me for over-simplifying some topics.


Public Key Cryptography - What is it?:




NSS 4.0 Overview (rather technical):


Java Keytool



While I may be a bit paranoid whenever someone says "trust me", the truth is that we must all trust someone (to varying degrees.)  This is particularly critical with digital certificates.  With physical tokens, we trust issuers like a "Department of Motor Vehicles"  (for driver's licenses) or "United States Department of State" (for passports), relying on the difficulty in forging a token as one basis for the trust.  Recall that in historical times, a simple wax seal was used because the crests impressed in them were typically difficult to duplicate...well, most days. 


With digital certificates, which are all about trust and tokens, the requirement for both parties to trust common point, typically a certificate issuer or "certifying authority" (CA).  CA's may be public (e.g. Verisign/RSA/EMC), private to a sector (e.g. a government-sponsored defense organization will likely have their own issuing authority), private to a company, or even private to a single system (typically referred to as "self-signed" certificates.)  The construction of the certificates from each of these types of authorities may be TECHNICALLY valid, but does our usage allow us to trust some types more than others?  Often, the answer is yes.


Obtaining a new certificate

A certificate begins as a large random number, typically referred to as a private key.  This is the basis for everything related to the use of this certificate and should be protected.  VERY strongly in some cases (e.g. if someone compromised a well-regarded public CA's private key, falsified certificates could be issued to compromise security controls.)  That would be bad.


From the private key, a certificate request is generated.  Who requests the certificate?  A web servers requiring security, LDAP-enabled directory servers, applications, and people requiring encrypted e-mail are all frequent requesters. The request contains several pieces of information about the requesting entity in the request: who owns/operates the system, contact information, etc.  The private key is NOT contained in the request; but does sign the request.


The certificate request is relayed to the CA.  How is it relayed?  That depends.  Email, web forms, ftp, private courier in locked suitcases,...really doesn't matter in this example.  The CA takes the request and from that and the CA's private key, an encoded digital certificate is created and for the requester.  For servers, this is frequently a specially formatted text file that can be added to a variety of certificate stores.


The requester can take the response and add the issued certificate back, but only to the system which has the private key from which the request originated. 


For those keeping score, the requestor would now have (at a minimum)

  1. A partially built certificate keystore that contains a private key,
  2. A certificate request built from the private key,
  3. A digitally-signed certificate from the CA usable by the requestor

The CA would have:

  1. A certificate keystore that contains the CA's private key,
  2. A certificate request submitted by the requestor,
  3. A digitally-signed certificate from the CA usable by the requestor.

The private keys are never exchanged.  Only a public identifying component for each  of the entities is passed, called the public key.  Each participant in a secure negotiation will have their own public and private key.  Each will only share their PUBLIC key. 


These trust relationships are not just point-to-point.  Certificates can grow into multi-level hierarchies call chains of trust.  "SuperSafeServer" may trust only certificates issued by "PrimeCA", but the Joe Client's browser has trusts a certificate from "BlackPhone", whose root certificate was also issued by "PrimeCA", so "SuperSafeServer" and Joe Client may trust one another without stopping to ask a user if the certificate presented by "SuperSafeServer" can be trusted.




NSS Keystores (cert8.db/key3.db)

There are two primary types of keystores with which Remedy is primarily concerned.  The first type may be familiar to anyone who has configured AREA LDAP:  cert7.db and key3.db.  The cert7.db has been replaced by the cert8.db file, so we will refer to the cert8.db/key3.db file names.  These started with Netscape browsers and are managed with the CERTUTIL command as part NSS (Network Security Services) cryptographic libraries.  Obtaining the CERTUTIL package can be problematic for some, but it is commonly used on Linux platforms as a standard component (e.g. Debian "libnss3-tools" provides this tool, but the package name will vary by distribution.)  The commands coming up are done on a Red Hat Enterprise Linux machine, so if you wish to reproduce them on Windows, adjustment accordingly.


CERTUTIL does have some quirks.  It may want to create new files in a sub-directory of the user's "home" directory (Linux/Unix alias this as a tilde ~) in .netscape (note the directory name starts with a period.)


The two files contain the private key, and the list of trusted CAs.  So, to create an empty new file set and get started.  In this example, the PUBLIC key for the authority to be trusted has been obtained beforehand in the file root.crt.  The certificate from the directory with which we are attempting to set up AREA LDAP has been exported from the directory server to ldap_server.crt.


Microsoft Active Directory Server

  1. Run MMC (Microsoft Management Console)
  2. Add the CA Microsoft Management Console snap-in
  3. Right-click the server and select Properties
  4. General > View Certificate
  5. Export to file.


Update:  On my Windows 7 system, I exported the certificate using the following:

  1. Run MMC (Start > Run or WindowsKey+R mmc >Enter)
  2. File > Add/Remove Snap-in
  3. Select Certificates > Add
  4. Select Computer Account > Next
  5. Select Local Computer > OK > OK.
  6. Expand Certificates > Trusted Root Certification Authorities > Certificates
  7. Right-click your domain servers' root CA and select All Tasks > Export.
    1. A  .CER file is your best choice generally, but I wrote it as CRT in this blog posting.


#  Create certificate database directory

mkdir ~/.netscape


#  Create empty DB

certutil -N


#  Add the ROOT CA with the privileges for Trusted Peer (P),Trusted CA to

#  issue client certs (C), and Trusted CA to issue server certs (C) for SSL. 

#  Trusted Peer for S/MIME clients certs (2nd P), etc..

#   -t trustargs      Set the certificate trust attributes:

#      trustargs is of the form x,y,z where x is for SSL, y is for S/MIME,

#      and z is for code signing

#          p      valid peer

#          P      trusted peer (implies p)

#          c      valid CA

#          T      trusted CA to issue client certs (implies c)

#          C      trusted CA to issue server certs (implies c)

#          u      user cert

#          w      send warning

#          g      make step-up cert

certutil -A -n "PrimeCA" -a -i ~/pki/root.crt -t"PTC,PTC,P"


#  Add the certificate for the directory server

certutil -A -n "MyLDAP" -a -i ~/pki/ldap_server.crt -t"P,P,P"


#  Display the cert

certutil -L

#  Copy the cert8.db to where you need it.

cp ~/.netscape/cert8.db /opt/bmc/ARSystem/arealdap

cp ~/.netscape/key3.db /opt/bmc/ARSystem/arealdap

The AREA LDAP Certificate Database configuration would need to point to the DIRECTORY with the two files (/opt/bmc/ARSystem/arealdap) and have permissions to read the files to operate.


Java Keystores (JKS)

There other primary type of keystores with which Remedy is concerned are Java Keystores.  These are used (for example) in Data Management when attempting to configure an encrypted connection to an LDAP-enabled directory, like Microsoft Active Directory. (You see; I told you this was going somewhere.)


Please note that when working with Java Keystores for Atrium SSO, restrictions apply on the use of the keytool encryption types supported and that you should review the documentation of Atrium SSO for details on creating/managing those keystores.


Java keystores support a variety of different encryption types and can hold multiple CA certificates and chains, private keys, etc. where each has an alias name associated with the entry.  Java keystores are also a single file and are accessed only when you have the correct authentication token (password) to unlock the file and use the contents. 


For most applications requiring a JKS file, you need the location of the JKS and the password to open it.  Some configurations may allow you to specify which certificates inside the file for usage (by the alias), but may will only allow one certificate for usage.  That usage may be encrypting a service, identifying a client application to a service, digitally signing a code package, etc.  For our Data Management LDAP scenario, a Java Keystore is required when connecting to directory server over a secured LDAP (LDAP/S) connection.  This requires a JKS with a CA certificate (or chain) that allows the Data Management job to trust the certificates presented by the directory server.


To create an empty JKS file and load the trusted CA:


#  Create a new JKS file (myLDAP.jks) with an unlock password (mySecretPassword)

#  and make the private key good for ~10 years (3650 days)

keytool -genkeypair -v -alias myKey -keystore myLDAP.jks -storepass mySecretPassword  -validity 3650


What is your first and last name?

  [Unknown]:  Remedy Customer

What is the name of your organizational unit?

  [Unknown]:  Remedy On-Demand

What is the name of your organization?

  [Unknown]:  BMC Software, Inc.

What is the name of your City or Locality?

  [Unknown]:  San Jose

What is the name of your State or Province?

  [Unknown]:  California

What is the two-letter country code for this unit?

  [Unknown]:  US

Is CN=Remedy Customer, OU=Remedy On-Demand, O="BMC Software, Inc.", L=San Jose, ST=California, C=US correct?

  [no]:  yes


Generating 1,024 bit DSA key pair and self-signed certificate (SHA1withDSA) with a validity of 3,650 days

          for: CN=Remedy Customer, OU=Remedy On-Demand, O="BMC Software, Inc.", L=San Jose, ST=California, C=US

Enter key password for <myKey>

          (RETURN if same as keystore password):  [RETURN]

[Storing myLDAP.jks]


#  Import the trusted CA's certificate (root.crt) into the keystore

#  and with the alias as ldapCA

keytool -import -alias ldapCA -file root.crt -keystore myLDAP.jks -storepass mySecretPassword

This JKS file would now need to be put into place on the server with AR (e.g. /opt/bmc/customer/myLDAP.jks).  That location would then be used in configuring a Data Management job based on the Secure LDAP People template.  That template contains two variables for the trust store path (the JKS file, not directory) and the trust store password (mySecretPassword) when they are used in a job.



As an administrator, I never like trying to remember a file location and entering it over and over.  One trick I have used with the new Data Management with my custom LDAP jobs is this: add the JKS to the Spreadsheet Manager or as a file attachment to a custom job.


What?!?!  Why would you do that?


Because, I can now upload a JKS for each job I schedule or reuse the one in the Spreadsheet Manager and simply provide the correct password for the proper JKS file to unlock it. 


If my custom transformation requires a JKS, use the ${DMTATTACHPATH} variable to get the location of the keystore.  Then, whether the JKS is attached using Spreadsheet Manager or as an attachment uploaded to the job, it should be present when the job runs, and I don't have to synchronize file locations across servers.  It still requires the correct password to unlock the file (e.g. ${Trust_Store_Password}), and allows a DMT User to update their keystore without requiring access to the server file system.  It also means one less variable a user has to enter when configuring a job.


I hope this helps.


UPDATE #2 (2013-03-15):  I was reminded today of one other topic:  conversion of a certificate.  Here's a handy tip for those that have an NSS certificate for their AREA LDAP and want to create  JKS from it:


To move NSS certificate (cert8.db/key3.db) to Java Keystore (JKS)

###   Export certificate from the NSS in ~/.netscape to a file CERT.RFC in RFC format

certutil -L -a -n ldapCA -d ~/.netscape/ > cert.rfc


###   Import RFC formatted certificate to JKS

keytool  -import -noprompt -trustcacerts -keystore myLDAP.jks -storepass mySecretPassword -alias ldapCA -file cert.rfc


This will create a cert.rfc file using the certutil and use the Java keytool command to import it into the keystore.file (which should already exist).  Then use the JKS file (with whatever password was used to unlock it) where needed (e.g. Atrium SSO or UDM.)