9 Replies Latest reply on Oct 28, 2019 11:07 AM by Emanuel Oliveira

    How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)

    Emanuel Oliveira
    Emanuel Oliveira Oct 21, 2019 11:11 AM
      Share This:

      Hi all,

       

      Following the announcement in Control-M v9 new feature "named pool variables" post:

      Trending in Support: Named Pool Variables in Control-M 9 Webinar Q&A

       

      I then asked the following:

      Header 1

      If a chain executes 4 times a day, where 1st job creates/sets named pool var=%%ORDERID.%%RUNCOUNT so 2nd and remaining jobsteps can reuse the same var value.

       

      But imagine:

      1st execution of chain

      -1st job sets named pool myvar to 123.

      - 2nd job fails (2nd job command line uses %%\\mypool\myvar).

       

      2nd execution chain kicks, 1st job again sets new value for the named pool myvar.

      2nd step and remaining jobs on this chain completes ok.

       

      Now... what value will 2nd job on 1st execution sees after we rerun it? Does somehow c/m "caches" initial value so chain still sees old value, or does job pick new value set by the 2nd chain execution?

       

      If c/m always resolves access to named pool var as it latest value, than how can we set/share 1 variable in all jobs of a chain but independent from run-execution of the chain?

       

      Thanks,

      Emanuel

       

      Which Kindly MunKeong Lee suggested the following:

      Header 1

      Hi Emanuel

      Accessing a named pool variable will always return the current value.

      To solve your issue, the name of the named pool has to be a variable. For example, if I need to differentiate the variables using order date, I can do the following:

      1. Place your jobs in a SMART folder.

      2. In the SMART folder, set %%POOLNAME=MYPOOL_%%ODATE. This will allow all jobs in the SMART folder to access %%POOLNAME

      3. Set the value of the pool variable accordingly in 1st job: e.g. %%\\%%POOLNAME\myvar = 123

      2. In 2nd job, the command will use %%\\%%POOLNAME\myvar

       

      Regards,

      MK

       

      I then reasked more precision (how to handle the fact of multiple executions in same ODATE):

      Header 1

      Hi Lee,

       

      Thanks so much for your feedback.

      So you suggesting using both "named pool" (dynamic name) + SMART folder.

       

      1. does we need to use SMART Folder ? On small test i simply declared and set a named pool var right on 1st job (i didnt declared the variable on the SMART Folder, and it works well, as out control-m engineer team "doenst like" SMART Folders in terms of best practice since they allow so much (sub folders etc..).
      2. In order to have independent variables for each of the 4 executions of the chain (4 times a day), imagine simple chain of 3 steps where we setting a comon variable so that can be accessed by 2nd and 3rd steps. We need to improve your suggestion of %%POOLNAME=MYPOOL_%%ODATE and add something like original scheduled time but i couldnt find system var like %%OTIME ? Im trying to imagine how can we avoid conflicts on the 4 intraday executions (since for example %%ORDERID remains the same for each of the 3 executions of the 1st job in the chain. Maybe if we added %%RUNCOUNT like %%POOLNAME=MYPOOL_%%ODATE_%%RUNCOUNT then we could have 4 pools, one for each of the 4 intraday executions ? would this work
      3. If it worked, then will those named pools remain on the system for ever or will some automatic C/M cleanup process deletes these pools after some time ? I think theres the commandline cmtvar utility that allows a lot of things, but would like a solution without needing extra steps executing cmtvar.

       

      thanks,

      Emanuel

       

      But meanwhile BMC, emailed me suggesitng this was interesting enough and should be given more visiblity, thats why Im copy/pasting here the 3 msgs on the community website, hoping someone can help/suggest ?

       

      Again objective is very simple really, on a simple chain of 3 job-steps,scheduled to be executed 4 times a day (2am, 8am, 2pm and 8pm) how can each have/share their own common variable ?

      Seems theres 2 ways:

      • SMART Folders= would each execution have its own local variables execution ? our Control-M team doenst "like devs to use SMART Folders".
      • Named Pool variables(without using SMART Folder as our Control-M team doenst "like them")= using dynamic named pool ?
      job-step
      		2am8am2pm8pm

      1 execute some ETL

      set common chain variable for 1st execution only  %%COMMON_VAR= %%ORDERID

      		its own %%COMMON_VARits own %%COMMON_VARits own %%COMMON_VARits own %%COMMON_VAR
      copy_src2dest.sh %%COMMON_VAR
      another.sh  %%COMMON_VAR

       

       

       

      Thanks,

      Emanuel Oliveira

        • 1. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
          George Lau
          George Lau Oct 21, 2019 1:19 PM (in response to Emanuel Oliveira)

          I am confused with the requirement of having variable for cyclic job flow that is independent at each of its multiple runs.

           

          Job1 sets a pool variable and Job2 consumes the variable. Job2 can only see the value assigned by the most recent completed run of Job1. Variable being referenced or consumed in a job has no tie to job's run count.

           

          If preserving value in variable of previous run is required, new value should be appended to variable instead of assigned. That is

          %%\\pool\variable=%%\\pool\variable NewValue

          Needless to say, variable must be initialized for the above to work.

          • 2. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
            Emanuel Oliveira
            Emanuel Oliveira Oct 21, 2019 2:47 PM (in response to George Lau)

            Thanks @George Lau for the syntax.

             

            Let me clarify our use case, we have 1 chain with 3 job-steps:

            1. execute ETL process that generates in one LANDING_FOLDER (yyymmdd) 1 file having some ID in the filename (aka, my COMMON_VAR).
            2. this chain executes 4 times in same ODATE (2am, 8am, 2pm and 8pm). That means that folder will have 1 file, 2 files, 3 files and 4 files in end of day.
            3. BUT.. 2nd step we only want to copy 1 file from LANDING_FOLDER to LOAD_FOLDER. Note: and yes we talking on S3 which is hard to work with filesystem (so one can't easy do all the magic available to shell scripts on UNIX/LINUX).So 2nd job simply deletes LOAD_FOLDER and COPY LANDING_FOLDER/*COMMON_VAR* LOAD_FOLDER (1 file).
            4. Finally 3rd job-step executes .sql that loads file into database table.

             

            Heres execution example with chain failing at different job-steps in different daily executions:

            Header 1
            job-step1 - execute etl( COMMON_VAR )
            job-step2 - execute clear_copy.sh ( COMMON_VAR ) - clear LOAD_FOLDER + cp LANDING_FOLDER/*12345* LOAD_FOLDER/
            job-step3 - executes .sql COMMON_VAR - loads single file on LOAD_FOLDER into DB

             

             

             

            LANDING_FOLDER2am chain 8am chain2pm chain8pm chain
            file_111.jsonjob-step1 - ETL(111) --> OK creates file_111.json
            job-step2 - DEL_COPY(111) --> OK
            job-step3 - LOAD_FILE_DB(111) --> OK

             

             

            LANDING_FOLDER2am chain8am chain2pm chain8pm chain
            file_111.jsonjob-step1 - OKjob-step1 - ETL(222) --> OK creates file_222.json
            file_222.jsonjob-step2 - OKjob-step2 - DEL_COPY(222) --> ERROR for some infrastructure reason
            job-step3 - OK

             

             

             

            LANDING_FOLDER

            2am chain

            common_var=111

            8am chain

            common_var=222

            2pm chain

            common_var=333

            8pm chain

            common_var=4444

            file1_12345.jsonjob-step1 - OKjob-step1 - ETL(222) --> OK creates file_222.jsonjob-step1 - ETL(333) --> OK creates file_333.json
            file2_6789.jsonjob-step2 - OKjob-step2 - DEL_COPY(222) --> ERROR for some infrastructure reasonjob-step2 - DEL_COPY(333) --> OK
            file3_3333.jsonjob-step3 - OKjob-step3 - LOAD_FILE_DB(333) --> OK

             

            Now heres the point:

            When one RERUNS job-step2 8am - you expect/want - it to still use 222, and not 333 which was latest value set on 3rd daily execution!

            So thats why, on this use case, each daily execution must keep their own private variable, and accessible by the 3 job-steps.

             

            Hope this clarifies better the use case, Im trying to figure out how to accomplish this with Control-M, but without using SMART Folders (as our C/M team still doesnt "like" them).

             

            But nevertheless my wonderings are:

            1. Could SMART Folders be a solution to have common var private to each chain execution ? And what would be the syntax needed to setup the variable/value ?
            2. Is there a solution with NAMED POOL VARIABLES ? I could imagine some syntax allowing dyamic pool name having %%ODATE for the day, but didnt found any other Control-M system variable that could be used to represent a specific execution like Original Time for example, then one could imagine a poolname  named as MYCHAINNAMEPOOLNAME_%%ODATE_%%OTIME and set 1 variable there. Or have 1 poolname named %%MYCHAINNAMEPOOLNAME_%%ODATE and having 4 variables named VAR_%%OTIME (so each named pool would endup holding 4 variables dynamic named. But I wonder if this makes sense ? What would be the control-m syntax ? Is there any %%OTIME or something that allows to represent the 1st scheduled execution, 2nd etc.. no matter how many reruns each job internally got executed eventually ? And finally, even if this dynamic pool names and/or dynamic variable name + way to destiguinsh of each execution, is possible, then what would be maintenance needed to delete all those named pools ? Or do they get purged after x days or something ?

             

            Thanks and look forward for suggestions

            • 3. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
              George Lau
              George Lau Oct 21, 2019 3:48 PM (in response to Emanuel Oliveira)

              If below is a correct understanding of the sequences in the table, there is no need to keep variable independent for each run in the loop.

               

              1st run at 2am:

              Job1 sets variable VAR to 111

              Job2 uses variable VAR with value 111

              Job3 uses variable VAR with value 111

               

              2nd run at 8am:

              Job1 sets variable VAR to 222

              Job2 uses variable VAR with value 222

              but Job2 fails and the flow waits to start from the top for next run

               

              3rd run at 2pm:

              Job1 sets variable VAR to 333

              Job2 uses variable VAR with value 333

              Job3 uses variable VAR with value 333

               

              After Job2 has failed and before 2pm, if Job2 is manually kicked off (without touching Job1) and with OK result to kick off Job3, variable VAR still has value 222 for both jobs. 3rd will continue as above at 2pm.

               

              Nothing in job execution ties to its run counter, unless with the explicit use of RUNCOUNT variable somewhere. If variable were to tie to run counter, you would have encountered a different issue. For the 3pm run without a rerun of Job2 after its failure, run count of Job3 (and any jobs below it) is at 2 while all jobs above it are at 3. Job3 would have accessed a counter-dependent variable different from that of Job1 and Job2. For the 3pm run with a rerun of Job2, the run count of Job2 is at 4 while all others are at 3.

              • 4. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
                MunKeong Lee
                MunKeong Lee Oct 21, 2019 10:21 PM (in response to Emanuel Oliveira)

                Hi Emanuel

                I think I understand your requirements better now. Making these jobs cyclic will not work. When you rerun Job2, you can't specify that you are rerunning for the 8am chain which failed.

                In order for this to work, you will need to create 4 chains of 3 non-cyclic jobs with different start time for Job1: 2am, 8am, 2pm & 8pm. In Job1 of the 1st chain, you will make use of named pool called MY_2AM_POOL by setting the name pool variable, %%\\MY_2AM_POOL\COMMON_VAR, to the value to be used by job2 & job3. In job1 of the 2nd chain, you set the value of %%\\MY_8AM_POOL\COMMON_VAR. Same goes for your 3rd & 4th chain. Now you can rerun jobs independently without impacting jobs in other chains.

                There's no need to delete these pool variables as they will be reused again and will not grow over time.

                Hope this helps.

                Regards,

                MK

                • 5. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
                  Emanuel Oliveira
                  Emanuel Oliveira Oct 25, 2019 3:28 AM (in response to MunKeong Lee)

                  Hi, its interesting having 4 independent chains (2am, 8am, 2pm and 8pm), but your suggestion if only having 4 namedpools means 2nd day would overwrite values (always imagine a chain may not be rerun in same day).

                   

                  I wonder we could still only 4 named pools as you suggest but we dynamically name variable itself with %%ODATE ? what would be the syntax?

                  And any need to do maintenance purging old variables ? or not needed because of some automatic Control-M purge process?

                   

                  Thanks,

                  Emanuel O.

                  • 6. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
                    MunKeong Lee
                    MunKeong Lee Oct 25, 2019 4:00 AM (in response to Emanuel Oliveira)

                    Hi Emanuel:

                    I know you don't wish to use SMART folders but the easiest way to meet your requirements is to use 4 SMART folders. Each chain of 3 jobs will be in their own SMART folder. SMART folder variables are created per instance of SMART folder although the variable name is the same. You don't even need to make use of named pool variable. First job will just update the folder variable called COMMON_VAR. No need to differentiate 2am, 8am, 2pm, 8pm. You just need to make use of variable %%COMMON_VAR in 2nd and 3rd job.

                    Regards,

                    MK

                    2 of 2 people found this helpful
                    • 7. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
                      Emanuel Oliveira
                      Emanuel Oliveira Oct 25, 2019 1:04 PM (in response to MunKeong Lee)

                      I like that, and suspected would be the easy way to work as typically a folder is private(local vars), the entropic element here was the fact chain is cycle runs 4x times day.

                      Also i been suggesting team that its very hard look to 1 single chain to the understand whats failed and in which of the 4 executions.

                       

                      So my Spiderman gut feeling was your solution, 4 independent chains as would make easy to see what failed on each the 4 daily executions.

                       

                      Just one last doubt then, do multiple odate runs of same folder also have their own private var value? or will for example 2nd first exec(2a) overwrites value from previous odate?

                       

                      Thanks,

                      Emanuel O.

                      • 8. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
                        MunKeong Lee
                        MunKeong Lee Oct 28, 2019 7:30 AM (in response to Emanuel Oliveira)

                        Hi

                        Every Smart folder ordered will have it's own folder variable. Therefore, future orders will not interfere with folder variables from previous orders. There's also no need to housekeep folder variables. They will be remove together with folder removal.

                        Regards,

                        MK

                        1 of 1 people found this helpful
                        • 9. Re: How to share 1 common variable across all job-steps in a chain (scheduled 4 times a day)
                          Emanuel Oliveira
                          Emanuel Oliveira Oct 28, 2019 11:07 AM (in response to MunKeong Lee)

                          Thanks Lee for your support on this thread, which i'm sure will not only benefit me but anyone understanding the basics in control-m.