8 Replies Latest reply on Feb 10, 2011 7:02 AM by Bharat Moghe

    cleanupFileServer fails

    Bharat Moghe

      The cleanupFileServer fails with following log:

       

       

      Info Feb 4, 2011 7:18:19 PM [Fri Feb 04 19:18:19 GMT 2011] Starting deletion of files from file server.

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:19 GMT 2011] Starting to delete files marked for deletion.

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:22 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClearPkg-2001497.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001511.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001527.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001531.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001536.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001538.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001542.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/scripts/ifs/wrap_cmdb_if.sh

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/RBAC_SupportSvcs_PROD-2001480.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001500.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001509.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001517.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001533.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001534.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/scripts/dep/test.sh

      Info Feb 4, 2011 7:18:24 PM Command execution failed.  java.io.IOException: JNI file deletion failed: File exists

      Info Feb 4, 2011 7:18:24 PM Exit Code 1

      Info Feb 4, 2011 7:18:19 PM [Fri Feb 04 19:18:19 GMT 2011] Starting deletion of files from file server.

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:19 GMT 2011] Starting to delete files marked for deletion.

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:22 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClearPkg-2001497.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001511.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001527.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001531.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001536.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001538.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001542.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/scripts/ifs/wrap_cmdb_if.sh

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/RBAC_SupportSvcs_PROD-2001480.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001500.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001509.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001517.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001533.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/blpackages/EquityClear_Application-2001534.1

      Info Feb 4, 2011 7:18:24 PM [Fri Feb 04 19:18:23 GMT 2011] File could not be deleted from file server.It was not found under the specified location : //blaats4.unix.lch.com/nsh/storage/scripts/dep/test.sh

      Info Feb 4, 2011 7:18:24 PM Command execution failed.  java.io.IOException: JNI file deletion failed: File exists

      Info Feb 4, 2011 7:18:24 PM Exit Code 1

       

      Any clues, why this fails?

       

      Thanks in advance.

       

      Bharat

        • 1. cleanupFileServer fails

          Hi,

           

          Is there any chance you are running a performFullCleanUp job before this job?  The jobs are similar and perhaps the files are already marked for deletion.

          • 2. cleanupFileServer fails
            Bharat Moghe

            No, i did not run performFullCleanUp job.

             

            It seems it was a permission issue, earlier I ran this job with my username and BLAdmin role, it failed with above error log.

             

            Now this time I run it as BLADmin user with BLAdmin role, its running, not yet finished.

            I will update this thread once I get the job result.

            • 3. Re: cleanupFileServer fails
              Bill Robinson

              Check the user mapping entries in the users and users.local file on your file server.

              • 4. cleanupFileServer fails
                Bharat Moghe

                The cleanupFileServer job runs for 3 hours and then fails with following error, this time its not the permissions issue as its been sorted out.

                 

                Error log from blcli.log

                ======================

                07 Feb 2011 17:16:37,838 INFO [MESSAGE] [Mon Feb 07 17:16:37 GMT 2011] Deleting empty directory : //blaats4.unix.lch.com/nsh/storage/build/EquityClear/ECL_19.0.1_OP632_INT_DROP2/database/upgrade/OP333_PROD [/MESSAGE]
                07 Feb 2011 17:16:37,838 DEBUG [MESSAGE] processed response [/MESSAGE]
                07 Feb 2011 17:16:37,838 DEBUG [MESSAGE] will wait for more resp [/MESSAGE]
                07 Feb 2011 17:16:44,224 DEBUG [MESSAGE]  [/MESSAGE]
                07 Feb 2011 17:16:45,232 DEBUG [MESSAGE] Connection returned [/MESSAGE]
                07 Feb 2011 17:16:45,730 INFO [MESSAGE] Command execution failed.  [/MESSAGE]
                com.bladelogic.app.remote.BlRemoteException: com.bladelogic.mfw.util.BlAppServerException
                at com.bladelogic.message.app.APIMessageRequest.sendAPIMessage(APIMessageRequest.java:45)
                at com.bladelogic.app.api.DeleteServiceClientStubImpl.cleanupFileServer(DeleteServiceClientStubImpl.java:587)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
                at java.lang.reflect.Method.invoke(Unknown Source)
                at com.bladelogic.cli.factory.APICommand.executeCommandOnSingleObject(APICommand.java:127)
                at com.bladelogic.cli.factory.APICommand.executeCommand(APICommand.java:106)
                at com.bladelogic.cli.factory.LoadCommand.executeCommand(LoadCommand.java:66)
                at com.bladelogic.cli.commands.utility.RunWithProgressCommand.executeCommand(RunWithProgressCommand.java:65)
                at com.bladelogic.cli.factory.ComplexCommand.executeCommand(ComplexCommand.java:72)
                at com.bladelogic.cli.CLIEngine.executeCommand(CLIEngine.java:80)
                at com.bladelogic.cli.CLIMain.runSingleCommand(CLIMain.java:374)
                at com.bladelogic.cli.CLIMain.main(CLIMain.java:176)
                Caused by: com.bladelogic.mfw.util.BlAppServerException
                at com.bladelogic.mfw.net.BlSessionClientConnection.pread(BlSessionClientConnection.java:359)
                at com.bladelogic.mfw.net.BaseClientConnection.read(BaseClientConnection.java:258)
                at com.bladelogic.mfw.net.BaseClientConnection.read(BaseClientConnection.java:243)
                at com.bladelogic.message.MuxRequest.sendRequest(MuxRequest.java:122)
                at com.bladelogic.message.MuxRequest.sendRequest(MuxRequest.java:81)
                at com.bladelogic.message.app.APIMessageRequest.sendAPIMessage(APIMessageRequest.java:43)
                ... 13 more
                07 Feb 2011 17:16:45,749 DEBUG [MESSAGE] Command execution failed. Command Result : Fail
                ------------------------------------
                Invoked command : cleanupFileServer
                Target Object : null
                Return value : null
                Errors : com.bladelogic.app.remote.BlRemoteException: com.bladelogic.mfw.util.BlAppServerException
                Comments :
                Added a failed sub result
                Stopped command execution due to failure of subcommand


                SubCommand Result : Fail
                ------------------------------------
                Invoked command : cleanupFileServer-api
                Target Object : com.bladelogic.app.api.DeleteServiceClientStubImpl@1b994de
                Return value : null
                Errors : com.bladelogic.app.remote.BlRemoteException: com.bladelogic.mfw.util.BlAppServerException
                Comments : No comments [/MESSAGE]
                07 Feb 2011 17:16:45,767 INFO [MESSAGE] Cancelling operation... [/MESSAGE]
                07 Feb 2011 17:16:45,833 INFO [MESSAGE] No request for thread : main [/MESSAGE]

                =====================================

                 

                Any suggestions?

                 

                Bharat

                • 5. cleanupFileServer fails
                  Bharat Moghe

                  After this job fails, it seems Appserver gets crashed, as it does not respond to any query.

                  Also unable to do any action from console and it returns TLS connection error.

                  • 6. cleanupFileServer fails

                    Hi Bharat, I am suspecting you will need to raise a support case for this.

                    • 7. cleanupFileServer fails
                      Bharat Moghe

                      Raised a issue with support y'day, waiting for reply

                      • 8. cleanupFileServer fails
                        Bharat Moghe

                        Hi All,

                         

                        No reply from support, but managed to fix the issue

                         

                        After reducing JVM heap size to 2GB from initial 3GB and run it with ulimit with unlimited virtual memory in nsh solved the problem.

                         

                        blcli Delete cleanupFileServer completed successfully.