1 2 Previous Next 17 Replies Latest reply on Sep 18, 2019 2:58 PM by Robert Stinnett

    Jobs as code

    Mike Landis
      Share This:

      Is anyone building jobs using jobs as code?  We have been taking a look at it, but haven't really seen a benefit. Does your developers write out their own jobs? How do you control standards you have in place to ensure that jobs are built correctly?

        • 1. Re: Jobs as code
          Robert Stinnett

          Mike,

           

          We have been using it for about as long as the product has been out.  It's definitely a game changer, and introduces a whole new world of possibilities not only for your developers, but also for your Control-M team.

           

          Standards can be achieved in several ways.  Keep in mind Jobs as Code doesn't mean that you can just throw anything over the fence and hope for the best!  There are things like transformation (deploy descriptor) files that let you setup promotion rules across environments, plus you can integrate Control-M into build pipelines and the such now through the Automation API to help enforce standards.  In many ways, you have more power to control the environment now than you ever had in the past.

           

          One thing I like to point out when talking about Jobs as Code is that its not just purely "for the developers" -- it can be very useful for operations teams and such.  It allows you to move from a static environment to a more dynamic one, one where jobs can truly be created on the fly and destroyed just as easily.  Where applications can truly "talk" to the automation engine (Control-M) and further bring them together for that single pane of glass view.

           

          A case example in point:  Using the Automation API and Jobs as Code we were able to save tens of thousands of dollars by designing our own in-house patching system.  Everything we looked at we had to customize, so we use Jobs as Code to create a single master JSON file that drives the patching of over 4,000 servers a month!  Efficiency, scalability and purpose-built!

           

          Hope this helps give you a few ideas of how it might help.  If I can be of any help or share any information about what we did or how we did it, don't hesitate to ask.

           

          Robert

          3 of 3 people found this helpful
          • 2. Re: Jobs as code
            Mike Landis

            Thanks for the reply Robert!  Have you noticed any impact on performance when they run in this manner?

            • 3. Re: Jobs as code
              Robert Stinnett

              Mike,

               

              There really isn't a performance perspective to the actual jobs running involved here.  Jobs as Code is merely another mechanism for loading stuff into the Control-M database, performing actions, etc.  When you submit your job/folder it looks just like any other job you create today - in fact, you can't tell the difference.

               

              In most organizations, I'd say 99% of them, you will never "get rid of" traditional scheduling via the GUI or other means - this is merely another tool to help integrate and automate via Control-M.  In the end, it's all going into the same engine, that being Control-M.

               

              Cheers,

              Robert

              2 of 2 people found this helpful
              • 4. Re: Jobs as code
                Paul Robins

                We are just starting our 'Jobs As Code' journey, and one of the common misconceptions we found was that 'Jobs As Code' and Automation API equals developers coding schedules which caused great concern among operations teams! Whilst that's a possibility, in reality Jobs As Code is so much more. As Robert touched on we see the major benefits in our organisation in operational and DevOps areas. Automating schedule migrations, new schedule environment builds, quick automated deployment and shutdown of compute power for batch, and dynamic building of schedules for things like server updates, SQL server housekeeping etc.

                4 of 4 people found this helpful
                • 5. Re: Jobs as code
                  Ilyas Shaikh

                  Hello Mike,

                   

                  We have started using Job as code, it is really helpful when there is bulk requirement to configure similar jobs/flows; when there is a bulk requirement we just create 1 flow, test it and if it works, we just copy and paste the code and believe me it saves a lot of time.

                   

                  Also we have given it to some end users in development env where the TAT for making the jobs live is very short, end users do the changes in the code and run in development env and share the json with us to move the job in production with host name changes.

                  2 of 2 people found this helpful
                  • 6. Re: Jobs as code
                    Mike Landis

                    Rob and everyone else.

                    Have you also used Change Manager to allow developers to build jobs?  Or is Automation API the only tool you decided to use?

                    • 7. Re: Jobs as code
                      Robert Stinnett

                      Mike

                      We definitely do both approaches - our more technical developers use Jobs as Code, while our business and less technical users find Change Manager to be more suited to their needs. 

                       

                      In the end, we look at Control-M as the engine, and the job definitions are the "fuel" -- and that fuel can come from many sources.

                       

                      Hope that helps,

                      Rob

                      2 of 2 people found this helpful
                      • 8. Re: Jobs as code
                        Paul Robins

                        Unfortunately we haven't been able to justify the cost of Change Manager. Which is a shame because I don't think our production operations team will ever trust a schedule written by a developer until we have more safeguards.

                        • 9. Re: Jobs as code
                          Robert Stinnett

                          Paul,

                           

                          Don't we know each other from the Bladelogic community?

                           

                          What's their biggest worry?

                           

                          Robert

                          • 10. Re: Jobs as code
                            Paul Robins

                            Hi Robert, I haven't had anything to do with bladelogic. Must be an impostor!

                            The biggest worry is that developers won't produce definitions that are production worthy. They may have hard-coding, insufficient alerting, insufficient error checking, and not be integrated properly into other applications. Items like retention days etc. may not be considered, running over newday, duplicating of conditions.....

                            I could go on and on.

                            So we will be taking baby steps to begin with. We've just introduced some automation for defining MSSQL housekeeping job sets, and some scripting to update host names for scheduled server swap-outs so people are already starting to see the value, and I only really started playing with it 2 months ago.

                            • 11. Re: Jobs as code
                              Mike Landis

                              Paul,

                              I'm have only scratched the surface of automation api, but I believe you can put standards in place to help with some of the concerns you have.

                              You mentioned the cost of change manager, we own it, but have not implemented it so we are seeing what is the best route.

                              Has allowed developers access to the GUI Client?  If so, in what manner; browse, update, create jobs, etc.?

                              • 12. Re: Jobs as code
                                Paul Robins

                                You can enforce standards if you have WCM to define the standards. To a certain extent you can implement standards using deploy descriptors.
                                We have one environment for production and one environment for dev\test. Our developers and testers have access to run their jobs but only limited access to define jobs. Jobs are defined by Control-M SME's (except for unit testing). We also have some limited migration setup using XML extracts which was developed prior to Automation API.
                                I think the route forward for us will be CTM SMEs developing jobs in Control-M, exporting to JSON then migrating through the environments.
                                There are so many implications for us also in terms of security setup, folder naming standards etc. Reviews of these things have been needed for a while so it's not a bad thing, just time consuming.

                                1 of 1 people found this helpful
                                • 13. Re: Jobs as code
                                  Jess Knutson

                                  Paul,

                                  We are having the exact same issues as you with job and folder naming standards.  We also have the same concerns that programmers worries that they will not create "Production worthy" job definitions.  We do not currently have WCM and I have been pushing for us to get it.  Where I see WCM playing the biggest role is for enforcing "Site Standards".  Yes we can use the Deploy Descriptors but I have not seen anything requires their use or to verify if the correct Deploy Descriptor was used.  Until we get WCM I am not sure if we will ever allow programmers to fully use the AAPI to promote into Production. 

                                  1 of 1 people found this helpful
                                  • 14. Re: Jobs as code
                                    Robert Stinnett

                                    Paul,

                                     

                                    It sounds like you are doing it exactly the same way we introduced it.  I believe the best path forward for success is to introduce the Automation API first as a way to programatically interface with the existing Control-M system.  Let folks use it as a CLI tool, use it to query via the REST calls, etc.  This will be a major step forward in many ways, and will open up folks to the possibilities. 

                                     

                                    Jobs as Code, while awesome in itself, you are 100% correct, going willy nilly without defining a few borders can lead to all sorts of fun and headaches. When I meet someone who wants to go gung-ho with it, I refer them to the Workbench to allow them to have their own sandbox to play in first.  Once they get their feet wet, then work with them to see how they can begin moving it into a more production like environment.

                                     

                                    Sounds like you are on a good path - and keep in mind that the API and JaC are still developing.  It's not a race to the finish line, just a sprint forward!

                                     

                                    Robert

                                    1 2 Previous Next