2 Replies Latest reply on Aug 8, 2014 10:37 AM by richard mcleod

    No such file or directory - Caused By: JNI mkdirs of

    richard mcleod

      Some strange behaviour of one of our batch jobs that is built of file deploys

       

      One of our users kicked off a master batch job against 4 servers, 3 servers succeeded without issue, 1 server experienced an issue with 2 of the file deploy jobs. Logs below show a successful job, failed job, successful job, failed job and a successful job in ascending order. Wanted to include to show that the user was able to successfully execute the file deploys against the target and account has perms to create in tmp on the fileserver.

       

      InfoJul 11, 2014 4:21:26 PMStarted running the job 'deploy_linuxqc' with priority 'NORMAL' on application server 'appserver02_job_1'(25)
      InfoJul 11, 2014 4:21:33 PMExecuting work item File Deploy Job:deploy_linuxqc; Server:target4482;  on application server: appserver02_job_5
      InfoJul 11, 2014 4:21:45 PMThe job 'deploy_linuxqc' has succeeded

      -----------------------------

      InfoJul 11, 2014 4:21:48 PMStarted running the job 'deploy_precheck_oracle_client_fix' with priority 'NORMAL' on application server 'appserver02_job_1'(25)
      ErrorJul 11, 2014 4:21:49 PMcom.bladelogic.om.infra.mfw.util.BlException: java.io.IOException: JNI mkdirs of '//blfs.domain.com/D/BL_DataStore/Storage/tmp/FileDeployJob/b3334710-3e27-4427-95a4-9f8e8d347b4d/' failed: : No such file or directory, Error: java.io.IOException: JNI mkdirs of '//blfs.domain.com/D/BL_DataStore/Storage/tmp/FileDeployJob/b3334710-3e27-4427-95a4-9f8e8d347b4d/' failed: : No such file or directory (Caused By: JNI mkdirs of '//blfs.domain.com/D/BL_DataStore/Storage/tmp/FileDeployJob/b3334710-3e27-4427-95a4-9f8e8d347b4d/' failed: : No such file or directory)
      ErrorJul 11, 2014 4:21:49 PMThe job 'deploy_precheck_oracle_client_fix' has failed

      -----------------------------

      InfoJul 11, 2014 4:21:49 PMStarted running the job 'deploy_show_pcislotnumber' with priority 'NORMAL' on application server 'appserver02_job_1'(25)
      InfoJul 11, 2014 4:21:58 PMExecuting work item File Deploy Job:deploy_show_pcislotnumber; Server:target4482;  on application server: appserver02_job_5
      InfoJul 11, 2014 4:22:14 PMThe job 'deploy_show_pcislotnumber' has succeeded

      -----------------------------

      InfoJul 11, 2014 4:22:14 PMStarted running the job 'deploy_install_bmc_agent' with priority 'NORMAL' on application server 'appserver02_job_1'(25)
      ErrorJul 11, 2014 4:22:15 PMcom.bladelogic.om.infra.mfw.util.BlException: java.io.IOException: JNI mkdirs of '//blfs.domain.com/D/BL_DataStore/Storage/tmp/FileDeployJob/e11b2886-6b6c-4676-aa9d-08593594096e/' failed: : No such file or directory, Error: java.io.IOException: JNI mkdirs of '//blfs.domain.com/D/BL_DataStore/Storage/tmp/FileDeployJob/e11b2886-6b6c-4676-aa9d-08593594096e/' failed: : No such file or directory (Caused By: JNI mkdirs of '//blfs.domain.com/D/BL_DataStore/Storage/tmp/FileDeployJob/e11b2886-6b6c-4676-aa9d-08593594096e/' failed: : No such file or directory)
      ErrorJul 11, 2014 4:22:15 PMThe job 'deploy_install_bmc_agent' has failed

      -----------------------------

      InfoJul 11, 2014 4:22:15 PMStarted running the job 'deploy_BaseLinuxQC.pm' with priority 'NORMAL' on application server 'appserver02_job_1'(25)
      InfoJul 11, 2014 4:22:22 PMExecuting work item File Deploy Job:deploy_BaseLinuxQC.pm; Server:target4482;  on application server: appserver02_job_5
      InfoJul 11, 2014 4:22:39 PMThe job 'deploy_BaseLinuxQC.pm' has succeeded

       

      I ran the two failed jobs against the host separately and they succeeded without issue. Wondering if this is possibly being caused by a collision of sorts.

       

      (since there were 3 other hosts running the exact same job, is it possible that the tmp/hash directory was created for a different file deploy job target but was also allocated for this target? and ended up being locked or deleted before this job was able to access it?)

       

      bsa ver: 8.3.3 on Windows

      blfs rscd: 8.3.3 on Windows

      target: 8.3.3 Linux

        • 1. Re: No such file or directory - Caused By: JNI mkdirs of
          Bill Robinson

          what's in the agent log on blfs when this happens ?

          • 2. Re: No such file or directory - Caused By: JNI mkdirs of
            richard mcleod

            Nothing abnormal...

             

            d4097ec06bdf1009ebdb 0000013033 08/07/14 19:49:07.523 INFO1    rscd -  10.124.136.26 15312 BladeLogicRSCD@blafs01->adminuser@blafs01:PrivilegeMapped (BLAdmins:_app_vm_user1): CM: > [FileDeploy] Deleting //bldmzfsdomain.com/E/BL_DataStore/Storage/tmp/FileDeployJob/182f7b39-d06f-4cd5-8d1a-84a20229286d/

            b21bf1ed3544dfc5073b 0000013034 08/07/14 19:49:10.237 INFO     rscd -  blafs01 12356 BladeLogicRSCD (Not_available): (Not_available): The operation completed successfully. 

            69edc609268fcd8a6b14 0000013035 08/07/14 19:49:10.268 INFO     rscd -  blafs01 12356 BladeLogicRSCD (Not_available): (Not_available): The operation completed successfully. 

            8ccc483d1ff9cd321cd8 0000013036 08/07/14 19:49:10.284 INFO     rscd -  blafs01 12356 BladeLogicRSCD (Not_available): (Not_available): The operation completed successfully. 

            567bf632c74d0510eaff 0000013037 08/07/14 19:49:14.933 INFO1    rscd -  10.124.8.26 14804 BladeLogicRSCD@blafs01->adminuser@blafs01:PrivilegeMapped (BLAdmins:_app_vm_user1): CM: > [Deploy] Copying '//bldmzfsdomain.com/E/BL_DataStore/Storage/blpackages/sshd_config-10890.1' to '//server02/var/tmp/stage/a056a4de7ffb33ccb3ae72eb0ef58c5a'

            b0db057dec765f9e156a 0000013038 08/07/14 19:49:22.124 INFO1    rscd -  10.124.136.26 7756 BladeLogicRSCD@blafs01->adminuser@blafs01:PrivilegeMapped (BLAdmins:_app_vm_user1): CM: > [FileDeploy] Deleting //bldmzfsdomain.com/E/BL_DataStore/Storage/tmp/FileDeployJob/bd45feed-6845-4d96-aed0-c27df15db598/

            1abf0ba1ab544882bbe7 0000013039 08/07/14 19:49:34.370 INFO1    rscd -  10.124.136.26 12956 BladeLogicRSCD@blafs01->adminuser@blafs01:PrivilegeMapped (BLAdmins:_app_vm_user1): CM: > [FileDeploy] Deleting //bldmzfsdomain.com/E/BL_DataStore/Storage/tmp/FileDeployJob/e1cf88f8-4482-449d-b36a-f031f8217716/

            ac857dfd765a40102e66 0000013040 08/07/14 19:49:42.108 INFO     rscd -  blafs01 11360 BladeLogicRSCD (Not_available): (Not_available): The operation completed successfully. 

            12992d5623103e14ce54 0000013041 08/07/14 19:49:42.108 INFO1    rscd -  10.124.8.26 11360 BladeLogicRSCD@blafs01->adminuser@blafs01:PrivilegeMapped (BLAdmins:_app_vm_user1): CM: > [Deploy] Copying '//bldmzfsdomain.com/E/BL_DataStore/Storage/blpackages/bbaa2936-878b-4276-a587-ff1f2a37b404' to '//server01/var/tmp/stage/47447fbd1fd634af8de2866d62bcc2a2'

            52cbf81e02aa20c58568 0000013042 08/07/14 19:49:43.481 INFO1    rscd -  10.124.136.27 9148 BladeLogicRSCD@blafs01->adminuser@blafs01:PrivilegeMapped (BLAdmins:_app_vm_user1): CM: > [Deploy] Copying '//bldmzfsdomain.com/E/BL_DataStore/Storage/blpackages/b5800d06-da66-44af-ab8e-9fcc95565141' to '//server03/var/tmp/stage/701bef76e24e31cfa6c3fca1f6d271b3'