Control-M for z/OS jobs stuck in a WAIT SUBMISSION status in a CTMPLEX environment.

Version 1
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Control-M for z/OS


    COMPONENT:

    Control-M for z/OS


    APPLIES TO:

    Control-m for z/OS environments running in CTMPLEX mode.



    PROBLEM:

    Jobs stuck in a WAIT SUBMISSION status in a CTMPLEX environment. This  affected submission of jobs in the LOCAL monitor.


    CAUSE:

    A number of jobs, possibly hundreds, were submitted by the Controll-M monitor and then were manually held in JES so the jobs were in a held waiting execution status in JES.


    SOLUTION:

    Investigation found the following.

    A number of jobs, possibly hundreds, were submitted by the Controll-M monitor and then were manually held in JES so the jobs were in a held waiting execution status in JES.


    The cause of the problem relates to CTMPLEX environments (some internals of Global - Local communication) and may happen only in such the special situation when the Local reaches its maximum Capacity (LISTPLEX shows LOCAL - CTMMSTR .CTMMSTR  ON SYSTEM BMCX. ACTIVE JOBS 0080, CAPACITY 0080) but the corresponding jobs (active jobs submitted / tracked by Local which reached its maximum Capacity) are actually waiting in the JES2 spool  due to TYPRUN=HOLD on the eJCL card or being manually held in JES.

    This problem may happen only when BALANCEM=N is defined in the CTMPLEX parameters member. When BALANCEM=Y, the problem will not occur, even in the special situation described above.

    There are two options to resolve this problem:

    1) Set BALANCEM=Y in the CTMPLEX parameters member. This is highly recommended.

    2) Increase the MAXCAP= (for example for all entries) in the CTMPLEX member so that it would be enough for the concurrent number of TYPRUN=HOLD jobs in SPOOL so that Local(s) won't reach its Maximum Capacity. In the example LISTPLEX results above, this parameter is set to MAXCAP=80 for all entries. We don't see any problem with increasing this value to lets say, 800.

    Either of the two options described above is enough to bypass the problem. A combination of both actions could be used also.

    Some extra information regarding the changing parameters in CTMPLEX:

    Changes can be made without stopping Control-M processing. After changing the SYSTEM ENTRIES information in CTMPLEX (for example MAXCAP=) and saving the updated CTMPLEX member, a Modify operator command NEWPLEX can be issued to the Control-M Global Monitor to make the changes active.  

    After changing the general CTMPLEX parameters in CTMPLEX (for example BALANCEM=) and saving e CTMPLEX member updated, the Modify operator commands STOPPLEX and STARTPLEX may be issued to the Control-M GLOBAL monitor to make the changes active. STOPPLEX will cause the shut down of Local monitor (but Global would continue processing in regular - non CTMPLEX - mode). So after STARTPLEX it's necessary to restart Local monitor again.

    As a bypass, shut down the local monitor and run with the GLOBAL monitor only.

    If the problem is not resolved with the above actions then you will need to do the following when the problem is seen:

    1) Issue the LISTPLEX operator command in the CTMPLEX environment.

    2) Enable traces 31 and 32 fro ALL control-M monitors (this is for the submission performance problem).

    3) If the monitor appears to be hanging the take a DUMP of problematic Control-M Monitor. To produce the DUMP, issue the operator command DUMP :

    DUMP COMM=(CONTROL-M DUMP)

    You will see the following WTOR response :

    *xx IEE094D SPECIFY OPERAND(S) FOR DUMP COMMAND

    and you should reply

    R xx,JOBNAME=<Control-m started task name>, SDATA=(SUM,PSA,CSA,GRSQ,LPA,RGN,SQA,TRT),END

    4) Collecting the PFM (diagnostic performance data). These are the data produced in DATRACE  by default once per day. But it's possible to produce them more frequently by issuing the PERFDATA= operator Modify command (again to the Global monitor in CTMPLEX). For example 'F CONTROLM,PERFDATA=5' would indicate to produce PFM data every 5 minutes. It's very useful for investigating short time performance problems, when the daily average data (by default) is not precise enough. After collecting the data it may be returned to the default  'F CONTROLM,PERFDATA=1440'.

    Then open a case with BMC Support providing the sysouts of all the monitors as well as the dump generated.


    Article Number:

    000124302


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles