3 Replies Latest reply on Jul 6, 2020 9:22 AM by Bentze Perlmutter

    Control-M Archiving 9.0.19.200

    Marco Bellusci
      Share This:

      Hi all

       

      We run Archiving 9.0.18.200 since about one year and upgraded EM, EM-Disrb. and CTMSRV to 9.0.19.200 end of May.

      All worked fine till 20.06.20.

      Since then ARC is not archiving anymore and has such Warnings in the Log:

       

      ST 2020]. Collect time : [197ms] CollectJobHandler::call

      0701_13:34:26.984 [WARNING] T@67-Clct-5pgx2-1-ctmdatacentername_job 5pgx2-1-ctmdatacentername was already archived (exist in DB), duplicate data will not be saved

      . PersistenceManager::saveJob

       

       

      I found out, the mentioned OrderID 5pgx2-1 (cylic Job Runno 1) is already archived in the DB. So it looks like ARC only checks for OrderID and denies to archive the sysout&log,

      because that orderID is already saved in the OracleDB. Even if it is a completely other Job from a different Application, Folder and Date.

       

      Thanks a lot for your support/hints/ideas/experiences.

       

      Regards

      Marco

        • 1. Re: Control-M Archiving 9.0.19.200
          Bentze Perlmutter

          Hi Marco,

           

          The "key" for archive is actually all three of these: 5pgx2-1-ctmdatacentername

          This is the <Job OrderID>-<Job Run Count>-<CTM Name>.

          Because OrderIDs are unique per CTM/Server this combination should always be unique.

           

          It looks like as part of the CTM/Server upgrade the OrderID counter was reset or brought back and after a month or so it caught up to where it was and now the OrderIDs are effectively being re-used which caused this issue in Workload Archiving.

          Example:

          It was at 5000, but numbers 3000-3999 weren't used, so it was reset to 3000.

          Runs between 3000 and 3999 got archived ok, but once it hit 4000 it stopped working because 4000-5000 already were used. (once it gets to 5001 the issue will be fixed on its own)

          As far as I know the upgrade of CTM/Server should not reset OrderID counter, but maybe due to a bug it did?!

           

          Did you have any issues since the upgrade where you used SQL to update the OrderID counter in Control-M/Server database? (that may have caused this and not the upgrade)

           

          One easy solution is to rename your Control-M/Server (via CCM you can right-click and rename it). This will fix the issue but may have other implications for your site and most likely the current name means something to users to renaming is probably not a good idea.

           

          The better solution is to contact BMC Support and ask them to advise the SQL to:

          1. find the highest OrderID in WA database

          2. update the Next OrderID counter in Control-M to the value found in WA+1

          After this is done, from the next job order (manual or new day or user daily) the jobs that get ordered should get archived correctly)

           

          Regards,

          Bentze

          1 of 1 people found this helpful
          • 2. Re: Control-M Archiving 9.0.19.200
            Marco Bellusci

            Hi Bentze

             

            Thanks a lot for your help!

             

            We didn't changed the counter, I dont't even know how to do it.

            Your points sounds all logic to me, but one thing I still do not understnad. With the 5 digits (alphanumeric) OrderIds, one day odrerIDs are not unique anymore. Depending on how many daily Jobs you run in a Datacenter.

            I think this happened to us. I guess ARC should also consider Jobname or Rundate/Times or Odate (with year YYMMDD, not MMDD).

             

            Renaming the datacenter is not an option, as it is used evreywhere (jobs, xmls, scripts, etc.).

             

            We are in contact with Support and waiting for their answer/solution.

            Perhapes it is easier to rename all allready archived Jobs in our Oracle DB, so that ARC will not find the OrderID and stop archiving.

            • 3. Re: Control-M Archiving 9.0.19.200
              Bentze Perlmutter

              Hi Marco,

               

              The OrderID in EM GUI is displayed in base36 while in CTM/Server database it is stored in base10 (this is so the user seas shorter/smaller numbers so more "user friendly").

              So in theory the highest base36 number in 5 digits is zzzzz which in base10 is 60,466,175.

              But, I found a thread on the communities (Order ID limit or rollover threshold - system setting? ) where Andrew Wong advises that BMC Support explained that once it reaches 9,999,999 (=5yc1r in base36) it resets to 00001.

              So maybe you have reached this limit?!

              From your error message example, OrderID 5pgx2 is 9,586,406 so not far from the apparent limit.

               

              But I find this strange because it means your OrderIDs would have reset to 00001 and then grown to 5pgx2 where the job ORDERID in WA DB existed and then you got the error. Maybe the "reset" happened before the upgrade and is not related to it. I.e. just natural growth and evolution of the ORDERIDs in Control-M/Server and something BMC didn't consider when they designed WA.

               

              I.e.

              If you order 10,000 jobs a day in CTM/Server, it should take approx. 999 days to get to the max OrderID of 9,999,999 so between 2.5-3 years. If you archive data in WA for more than 2.5 years so you can run into the issue you have.

              But in your case, since you only have WA for 1 year, that would be the max time you have data.

              Question:

              1. Approx. how many jobs get ordered per day in this Control-M/Server?

              2. How long are your archive policies setup to keep data for? (maybe the cleanup isn't happening as expected)

               

              BTW, in CTM/Server database the OrderID counter is in CMR_LASTNO table:

              SQL> select LASTISN from CMR_LASTNO where TABLENAME='CMR_AJF'; --> will show the highest orderid, in base10, where CTM/Server is up to.

               

              In WA database, if you're using PGSQL, you can run the following to see all the tables that hold archived job information:

              arc psql

              archivedb> \dt job_info*

               

              One of the fields will be the orderid ('\d job_info_1' for example will show the exact fields in the table)

              So you can use min/max on this field to get the range of OrderIDs still in the WA database.

               

              Anyway, please share the solution BMC advise as I'm thinking this may start happening to more sites, especially ones that have large CTM/Servers (many jobs ordered per day) and that keep WA archives for a long time, so will be good to know what the solution is.

               

              Thanks and Regards,

              Bentze