7 Replies Latest reply on Feb 25, 2015 6:46 AM by Ben Vassie

    Failed to set trusted certificates file

    Ben Vassie

      Our NSH jobs are currently failing with the following message

       

      Issue Summary:

      Error   Feb 12, 2015 12:28:20 PM        cd: error in TLS protocol: //CCASCIBBSA01.ccdc.local/

       

      Issue Details:

      Error   Feb 12, 2015 12:28:20 PM        cd: error in TLS protocol: //CCASCIBBSA01.ccdc.local/

      Error   Feb 12, 2015 12:28:20 PM        Failed to set trusted certificates file "/opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem" ("/opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem") - Failure in SSL_CTX_load_verify_locations

      Error   Feb 12, 2015 12:28:20 PM        Network Shell can be used for local access

      Error   Feb 12, 2015 12:28:20 PM        Error in Initializing RBAC User and Role (SSO Proxy)

      Info    Feb 12, 2015 12:28:20 PM        Exit Code 1

      Error   Feb 12, 2015 12:28:20 PM        Failed to set trusted certificates file "/opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem" ("/opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem") - Failure in SSL_CTX_load_verify_locations

      Error   Feb 12, 2015 12:28:20 PM        Network Shell can be used for local access

      Error   Feb 12, 2015 12:28:20 PM        Error in Initializing RBAC User and Role (SSO Proxy)

      Error   Feb 12, 2015 12:28:20 PM        Failed to set trusted certificates file "/opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem" ("/opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem") - Failure in SSL_CTX_load_verify_locations

       

      I have this logged (ISS04427628) but not getting anywhere!


      Restarting the job instance resolves the issue for a few hours, so far I have had to restart each morning for around two weeks

        • 1. Re: Failed to set trusted certificates file
          Ben Vassie

          Can anyone help?

          I keep having to restart the job instance every day to get BSA working again

          • 2. Re: Failed to set trusted certificates file
            Bill Robinson

            why do you have to restart bsa ?  are the appservers offline?  are they not able to run jobs of any type ?

             

            how many job servers do you have?  how many nsh proxies?  what are the following set to:

            maxworkitemthreads

            maxjobexecutionconnections

            maxnshproxycontexts

            maxnshproxythreads

            nshproxymaxthreadidletime

            • 3. Re: Failed to set trusted certificates file
              Ben Vassie

              To get BSA working again

               

              Something is wiping out /opt/bmc/BladeLogic/8.1/NSH/br/deployments/ccascibbsa01_job1/trust_store.pem, once I notice this issue the pem file is 0k, if I copy it from ../default it works again

               

              Few hours later, the file is 0k, making it read only does not help

              • 4. Re: Failed to set trusted certificates file
                Bill Robinson

                the next time this happens, can you check the modify time on the trust_store.pem file - that might give an indication when this is happening.

                 

                were you able to make the changes to the proxy service ? there seem to be connectivity issues to the nsh proxy - and that could be a problem w/ the settings you have - if a bunch of nsh jobs spin up and consume the available proxy connections, then you'll get sso connect errors through the nsh proxy.

                 

                the other issue is your cleanup:

                   cleanupRepeater
                    cleanupFileServer
                    cleanupHistoricalData
                    executeRetentionPolicy
                    cleanupDatabase
                    performFullCleanupJob
                    cleanupAgent

                 

                this is kind of wrong - look here: https://docs.bmc.com/docs/display/bsa83/Cleaning+up+the+BMC+Server+Automation+database

                 

                the order should be:

                 

                executeRetentionPolicy

                cleanupHistoricalData

                cleanupDatabase

                hardDeleteAllSharedObjects

                cleanupFileServer

                 

                the cleanupRepeater, cleanupAgent are separate - they don't clean the db.  performFullCleanupJob just calls cleanupDatabase and cleanupFileServer.  when you run cleanupAgent, what type of nsh job is that (runscript or execute once and pass targets) ? if it's a runscript, what parallelism do you run it w/ ?

                • 5. Re: Failed to set trusted certificates file
                  Ben Vassie

                  -rw-r----- 1 bladmin bladmin 0 Feb 25 00:19 trust_store.pem

                   

                  There is nothing in BSA scheduled for this time

                   

                  Here is the cleanupAgent script, it is a type two script

                   

                  #!/bin/nsh

                  cd //@;disconnect

                  SERVER_LIST=$1

                  for SERVER in $SERVER_LIST

                  do

                      blcli_execute Delete cleanupAgent $SERVER 10

                      if [ $? -ne 0]

                      then

                          echo "\n Cleanup Failed."

                          exit 1

                      else

                          echo "\n Cleanup Completed."

                          exit 0

                      fi

                  done

                  • 6. Re: Failed to set trusted certificates file
                    Ben Vassie

                    Bill

                     

                    I've edited the clean up tasks following your reccomendations, ran it manually multiple times and the pem file is fine

                     

                    I have no idea what is happening to that file at 00:19

                    • 7. Re: Failed to set trusted certificates file
                      Ben Vassie

                      Something in this script is killing the pem file on the app server

                       

                      ##################################

                      #

                      # bl_transdir_cleanup.nsh

                      #

                      # An nsh script to cleanup the agent transactions directory. Files and folders older

                      # than the parameterized number of days will be removed.

                      #

                      # WARNING:     Once a transaction is deleted, "UNDO" will no longer function for the

                      #        corresponding job run in the Configuration Manager.

                      #

                      # Arguments:

                      # days_back        - The number of days of transactions to keep from today.

                      #

                      #

                      ###################################

                      #

                      # 1.0 09/20/2007 - Created by Stathy Touloumis

                      # 1.1 09/21/2007 - Modified by Justin Suissa

                      # 1.2 02/15/2008 - Modified by Craig Williams - replaced FOR with WHILE READ which allows for spaces in RSCD_DIR path

                      #

                      ###################################

                      #

                      #

                      ###################################

                      #

                      # Initialize Variables

                      #

                      ###################################

                       

                      days_back=$1

                      host=`echo $PWD | awk -F"/" '{print $3}'`

                       

                      ###################################

                      #

                      # Determine Location of Transactions Directory

                      #

                      ###################################

                       

                      trans_dir=`blcli Server printPropertyValue $host RSCD_DIR`

                      trans_folder="/Transactions"

                      trans_dir=${trans_dir}${trans_folder}

                      echo RSCD Transaction Directory for clean-up on server $host -- $trans_dir

                      trans_path=//${host}${trans_dir}

                       

                      ###################################

                      #

                      # Clean up Transactions Directory

                      #

                      ###################################

                       

                      # Perform a find, looking for files/folders older than $days_back from the execution of this command

                       

                      find "$trans_path" -maxdepth 1 -mtime +${days_back} -and ! -path "$trans_path" | while read cleanup

                      do

                          echo Removing -- $cleanup

                          rm -rf "$cleanup"

                      done