12 Replies Latest reply on Sep 20, 2016 3:01 AM by Pabitra Subudhi

    Data management Job stuck in In Progress and it's Step remains in Queued status

    Pabitra Subudhi
      Share This:

      Hello,

       

      I need a quick help regarding DMT jobs which are stuck in In progress status and it's step stuck in Queued state. The purpose is to Bulk Upload CI data into one staging form using DMT Job (Remedy/ITSM/CMDB Version 9.1) where we created a Job then a Step and in the step we have mentioned one Atrium Integrator Job Name and attached an excel file.

       

      In the AI job we have a transformation which has 3 steps - 1. Excel Input, 2. Get variable and 3. AR OutPut to the staging form.

       

      So whenever we run the DMT (UDM) Job, it remains in In Progress and it's step is geting stuck at "Queued". We have built this new set servers from 8.1 old as a part of upgrade. In the Old system of 8.1 it works fine but not in new 9.1 system.

       

      Also few analysis:-

      1. Tried to clear out records from DMT:SYS:SequencingEngine with Parent_Job_GUID  = "DO NOT REMOVE" .

      2. ReBuild Sequence table from this form - DMT:SYS:StagingFormDependency .

      3. Found one one entries for each job in CAI:Events form where status was WARNING .

      4. Cleared the records from DLD:ThreadManager form.

      5. Observed Source and Target Server Name field values in CAI:Events form which holds app load balaner name, in oid system, I am able to ping and login using app LB name but not the case in new system.

       

      Please let me know if any more information is required, I would glad to share. Kindly help!

       

      Thanks,

      Pabitra

        • 1. Re: Data management Job stuck in In Progress and it's Step remains in Queued status

          -Make sure that escalations are enabled and running on the server on which you are creating/running the job.

          Any errors in arerror.log, arjavaplugin.log and arcarte.log files?

          • 2. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
            Pabitra Subudhi

            Thanks for your reply!

             

            Yes we do have enabled escalation ranked to run on primary server and I tried to run from web load balancer and from CMDB primary server which certainly not have Escalation running.

             

            But I believe it will run in the server group who has rank 1. because I compare both new and old system and when I ran DMT Job on old sytem on CMDB primary, it ran successfully and processed the records into staging form and dataset.

            But Job and Step status is still "In Progress" instead of being completed.

             

            No such error are there in arerror.log but found some error in arcarte.log. Please see below:-

            "INFO  09-09 03:14:40,884 - Bulk_Upload - Job execution finished

            INFO  09-09 03:34:53,082 - RepositoriesMeta - Reading repositories information from UDM:Config

            INFO  09-09 03:34:53,675 - Bulk_Upload - Start of job execution

            ERROR 09-09 03:34:55,188 - Excel Input - No file(s) specified! Stop processing.

            ERROR 09-09 03:34:55,188 - Excel Input - Error initializing step [Excel Input]

            ERROR 09-09 03:34:55,188 - Bulk Upload Transformation - Step [Excel Input.0] failed to initialize!

            ERROR 09-09 03:34:55,266 - Bulk Upload Transformation - Unable to open transformation: null

            ERROR 09-09 03:34:55,266 - Bulk Upload Transformation - java.lang.NullPointerException

            org.pentaho.di.trans.Trans.writeStepLogInformation(Unknown Source)

            org.pentaho.di.trans.Trans$8.transFinished(Unknown Source)

            org.pentaho.di.trans.Trans.fireTransFinishedListeners(Unknown Source)

            org.pentaho.di.trans.Trans.prepareExecution(Unknown Source)

            org.pentaho.di.trans.Trans.execute(Unknown Source)

            org.pentaho.di.job.entries.trans.JobEntryTrans.execute(Unknown Source)

            org.pentaho.di.job.Job.execute(Unknown Source)

            org.pentaho.di.job.Job.execute(Unknown Source)

            org.pentaho.di.job.Job.execute(Unknown Source)

            org.pentaho.di.job.Job.run(Unknown Source)

            INFO  09-09 03:34:55,266 - Bulk_Upload - Job execution finished"

             

            But I found something interesting.. Where we have below System Setting in old system and same is in new system:-

            I thought to replace the old server name with new one so I changed but while testing using "Test" button, it's throwing below error:-

            "Creating a directory under the path \\APPDADC397\DataLoad_Asset\MYLOGINID\ returned an errorcode of  (38).  Please check the server path is correct and that the current AR Server is permitted to Create/Delete folders/subfolder and files here. (ARWARN 48646)

            The preceding message occurred during the execution of active link SYS:SET:TestPathAccess -- action 2. (ARNOTE 1101)"

             

            Anyone please suggest how I can proceed from here? Should I create a shared folder in \\<Servr_Name>\<DataLoad_Asset> or it should create automatically..

             

            Thanks again,

            Pabitra

            • 3. Re: Data management Job stuck in In Progress and it's Step remains in Queued status

              For the message" Please check the server path is correct and that the current AR Server is permitted to Create/Delete folders/subfolder and files here. (ARWARN 48646)"

               

              I would recommend to create a new directory on the server and share it with Everyone.

               

              Give read and write permission for everyone to that directory and then test the connection.

               

              Once the test is successful you should get a message

               

              'The test has been executed and appears successful, please check and ensure the directory     was created (ARWARN 48646) '

               

              Hope this helps!!

              1 of 1 people found this helpful
              • 4. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                Pabitra Subudhi

                Thanks for the info, I have already done the same. But still Job Steps is in queued status, and have checked "DMT:StepParam" form entries with Job Instance ID, and it looks good in pointing to new server and path.

                 

                Let me know how we can get rid of this queued status for now because once this is fixed, then data would at least be loaded into staging form, which currently not happening.

                 

                Thanks again,
                Pabitra

                • 5. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                  Pabitra Subudhi

                  Just one more thing wanted to let you know that when tried to run the AI transformation from Spoon, getting error like No such file exist so run failed which I guess is expected, isn't it.

                   

                  Thanks,

                  Pabitra

                  • 6. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                    Pabitra Subudhi

                    Can someone please help me in fixing Queued Step Status in DMT job?

                     

                    Thanks in advance!

                    Pabira

                    • 8. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                      Pabitra Subudhi

                      Thanks for your help Chris!

                       

                      We have fixed the issue by following below steps:-

                      1. Installed MS Excel Viewer on CMDB Admin server.
                      2. Specified Remedy Application service password UDM:RAAppPassword on all server by referring to ar.cfg file.
                      3. Created a shared Directory called DataLoad_Asset on CMDB Admin and updated in "System Settings" under Application Administration->Custom Configuration->Foundation->Advanced-Options->System Settings with new CMDB server.
                      4. Added escalation "DMT:DJS:SetStatus" to pool 6.
                      5. Added CMDB Admin Server IP, Server name, and FQDN on hosts file on CMDB Admin server, similar to current old CMDB Admin.
                      6. Changed entry in "UDM:Repository:Slave" form with new CMDB server.
                      7. Updated Manage Data Store Connections with new CMDB server from AI Console's top tool bar icon.
                      8. Lastly, updated Directory name from DMT COnsole under Other Function->Atrium Integrator Jobs -> Bulk_Upload AI Job.

                       

                      But now the outstanding issue is simple but I am not ware of it and hope someone can point me in right direction to fix it.

                      Issue is we are able to run and complete the DMT UDM Job (contains Steps to call/run AI Job) from localhost or CMDB admin server, but we are facing issues while running it from Load Balancer or end user URL like itservicedesk.xyz.com.

                       

                      When checked, it's hitting one of the user facing server and in that server's arjavaplugin.log we found below error which looks like come connection issue. Need your quick help!

                       

                      <PLUGINSVR> <TNAME: pool-6-thread-5          > <ERROR> <ApiProxyFactory                                   > <                              ApiProxyFactory.java:169       > /* Wed Sep 14 2016 04:10:09.909 */  Connects to AppLoadBalancer.ab.xyz.net:9001 through Java Rpc failed with: ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect AppLoadBalancer.ab.xyz.net:9001

                      <PLUGINSVR> <TNAME: pool-6-thread-5          > <ERROR> <CAIFilterPlugin                                   > <                              CAIFilterPlugin.java:96        > /* Wed Sep 14 2016 04:10:09.909 */  Exception occured in filterAPICall with input: [PPGAA5V0FW12HAO378NRO2LXQPKK1S, SRMS, RemedyAppLB.na.averydennison.net, EVT000015833522, null, DLD_OUT_CREATE, 2, DMT:SYS:SequencingEngine, null, 0, 0, 0111939, 0]

                      ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect AppLoadBalancer.ab.xyz.net:9001

                       

                       

                       

                       

                      <PLUGINSVR> <TNAME: pool-6-thread-5          > <ERROR> <ARFilterAPIPluginRepository                       > <                  ARFilterAPIPluginRepository.java:127       > /* Wed Sep 14 2016 04:10:09.909 */  ARFilterApiCall FAILs in plugin: REMEDY.ARF.CAI

                      ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect AppLoadBalancer.ab.xyz.net:9001

                       

                      Thanks,

                      Pabitra

                      • 9. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                        Pabitra Subudhi

                        Hi There,

                         

                        Can someone please guide me on how to make DMT UDM jobs to run on load balancer URL as it's currently work from local server only. We have added host name entries in hosts file of all the user facing servers to avoid any connectivity issue and also update UDM:RAppPassword form password for eachserver by referring the remedy application service password from ar.cfg files from respective servers.

                         

                        Please suggest!

                         

                        Thanks,

                        Pabitra

                        • 10. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                          Pabitra Subudhi

                          Hello Team,

                           

                          Can someone please guide me how to configure DMT UDM jobs to run from Load balancer URL, currently it's only working from CMDB Admin server localhost URL.

                           

                          Already re-opened a case with BMC to get their suggestion but still no reply.. waiting...

                           

                          Thanks much in advance!

                          Pabitra

                          • 11. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                            Pabitra Subudhi

                            Can someone please help me on this. Kindly let me know if you need any more info.

                             

                            Thanks,

                            Pabitra

                            • 12. Re: Data management Job stuck in In Progress and it's Step remains in Queued status
                              Pabitra Subudhi

                              Hi Team,

                               

                              Phew!!! We have fixed the Issue now. We have performed below steps by analyzing logs and entries in some UDM related backend forms to fix the issue. Now DMT Jobs are running fine and getting completed with Data getting successfully imported, from End User Load Balancer URL.

                               

                              1. Installed MS Excel Viewer on CMDB Admin server.
                              2. Specified Remedy Application service password UDM:RAAppPassword on all server by referring to ar.cfg file.
                              3. Created a shared Directory called DataLoad_Asset on CMDB Admin and updated in "System Settings" under Application Administration->Custom Configuration->Foundation->Advanced-Options->System Settings with new CMDB server.
                              4. Added escalation "DMT:DJS:SetStatus" to pool 6.
                              5. Added CMDB Admin Server IP, Server name, and FQDN on hosts file on CMDB Admin server, similar to current old CMDB Admin.
                              6. Changed entry in "UDM:Repository:Slave" form with new CMDB server.
                              7. Updated Manage Data Store Connections with new CMDB server from AI Console's top tool bar icon.
                              8. Also updated Directory name from DMT Console under Other Function->Atrium Integrator Jobs -> Bulk_Upload AI Job.
                              9. Lastly. updated all the entries in CAI:AppRegistry form with RemedyApLB name and add RemedyAppLB.xy.abc.net in hosts file of all servers.

                               

                              Thanks all for your inputs!

                               

                              Regards,

                              Pabitra

                              1 of 1 people found this helpful