1 of 1 people found this helpful
The "key" for archive is actually all three of these: 5pgx2-1-ctmdatacentername
This is the <Job OrderID>-<Job Run Count>-<CTM Name>.
Because OrderIDs are unique per CTM/Server this combination should always be unique.
It looks like as part of the CTM/Server upgrade the OrderID counter was reset or brought back and after a month or so it caught up to where it was and now the OrderIDs are effectively being re-used which caused this issue in Workload Archiving.
It was at 5000, but numbers 3000-3999 weren't used, so it was reset to 3000.
Runs between 3000 and 3999 got archived ok, but once it hit 4000 it stopped working because 4000-5000 already were used. (once it gets to 5001 the issue will be fixed on its own)
As far as I know the upgrade of CTM/Server should not reset OrderID counter, but maybe due to a bug it did?!
Did you have any issues since the upgrade where you used SQL to update the OrderID counter in Control-M/Server database? (that may have caused this and not the upgrade)
One easy solution is to rename your Control-M/Server (via CCM you can right-click and rename it). This will fix the issue but may have other implications for your site and most likely the current name means something to users to renaming is probably not a good idea.
The better solution is to contact BMC Support and ask them to advise the SQL to:
1. find the highest OrderID in WA database
2. update the Next OrderID counter in Control-M to the value found in WA+1
After this is done, from the next job order (manual or new day or user daily) the jobs that get ordered should get archived correctly)
Thanks a lot for your help!
We didn't changed the counter, I dont't even know how to do it.
Your points sounds all logic to me, but one thing I still do not understnad. With the 5 digits (alphanumeric) OrderIds, one day odrerIDs are not unique anymore. Depending on how many daily Jobs you run in a Datacenter.
I think this happened to us. I guess ARC should also consider Jobname or Rundate/Times or Odate (with year YYMMDD, not MMDD).
Renaming the datacenter is not an option, as it is used evreywhere (jobs, xmls, scripts, etc.).
We are in contact with Support and waiting for their answer/solution.
Perhapes it is easier to rename all allready archived Jobs in our Oracle DB, so that ARC will not find the OrderID and stop archiving.
The OrderID in EM GUI is displayed in base36 while in CTM/Server database it is stored in base10 (this is so the user seas shorter/smaller numbers so more "user friendly").
So in theory the highest base36 number in 5 digits is zzzzz which in base10 is 60,466,175.
But, I found a thread on the communities (Order ID limit or rollover threshold - system setting? ) where Andrew Wong advises that BMC Support explained that once it reaches 9,999,999 (=5yc1r in base36) it resets to 00001.
So maybe you have reached this limit?!
From your error message example, OrderID 5pgx2 is 9,586,406 so not far from the apparent limit.
But I find this strange because it means your OrderIDs would have reset to 00001 and then grown to 5pgx2 where the job ORDERID in WA DB existed and then you got the error. Maybe the "reset" happened before the upgrade and is not related to it. I.e. just natural growth and evolution of the ORDERIDs in Control-M/Server and something BMC didn't consider when they designed WA.
If you order 10,000 jobs a day in CTM/Server, it should take approx. 999 days to get to the max OrderID of 9,999,999 so between 2.5-3 years. If you archive data in WA for more than 2.5 years so you can run into the issue you have.
But in your case, since you only have WA for 1 year, that would be the max time you have data.
1. Approx. how many jobs get ordered per day in this Control-M/Server?
2. How long are your archive policies setup to keep data for? (maybe the cleanup isn't happening as expected)
BTW, in CTM/Server database the OrderID counter is in CMR_LASTNO table:
SQL> select LASTISN from CMR_LASTNO where TABLENAME='CMR_AJF'; --> will show the highest orderid, in base10, where CTM/Server is up to.
In WA database, if you're using PGSQL, you can run the following to see all the tables that hold archived job information:
archivedb> \dt job_info*
One of the fields will be the orderid ('\d job_info_1' for example will show the exact fields in the table)
So you can use min/max on this field to get the range of OrderIDs still in the WA database.
Anyway, please share the solution BMC advise as I'm thinking this may start happening to more sites, especially ones that have large CTM/Servers (many jobs ordered per day) and that keep WA archives for a long time, so will be good to know what the solution is.
Thanks and Regards,