7 Replies Latest reply on Jan 23, 2020 2:19 PM by Adam Lawson

    Remedy AR System Server service slow to start

    Kyle Smithey
      Share This:

      When we restart the AR service on our server we get the below message:

       

      If you click OK the status will show as "Starting" and after several minutes will change to "Running" and it works fine. I would say on average it takes 5-6 minutes for the AR service to start, but we've also had it take approximately 15 minutes to start. Is this normal and if not are there any steps that we can take to reduce the start time?

       

      We only have 1 AR Server in our environment and it has 16GB of RAM so I wouldn't think that would be an issue.

       

      Any feedback is appreciated. Thanks!

        • 1. Re: Remedy AR System Server service slow to start
          rodrigo barcat

          Hi Kyle Smithey

           

          This scenario has been happening since I was very young. I found an explanation for you. Error1053 the service did not respond to the startor control in a timely fashion. And paste it below.

           

           

          Legacy ID:KA394401

           

          This is expected behaviour. The stop/start of that service also performs stop/start of other processes such as jserver, dbsrv, apache, cell etc. Windows has an internal timeout of 30 seconds for service stop/start. As stopping/starting that service is > 30 seconds then you will see the error 1053.

           

          -----------

           

          Please note that this is not Remedy issue, this is Microsoft Windows defect and for that Microsoft has provided the hotfix.

           

          Root cause : Windows defect

           

          Solution

           

          There is a Microsoft article (http://support.microsoft.com/?id=886695) that references this error and offers a hot fix. However, in some case we have observed that the customer has already applied either the hotfix or Microsoft Windows Service Pack 2, which includes and supersedes the hot fix. The problem still occurs, even with the fix applied. We recommend that the customer contact Microsoft regarding this issue.

          4 of 4 people found this helpful
          • 2. Re: Remedy AR System Server service slow to start
            rodrigo barcat

            About the start time, for BMC components when you have AR Server + ITSM + Email Engine + plugins...etc, the time really is too long.

            2 of 2 people found this helpful
            • 3. Re: Remedy AR System Server service slow to start
              Adam Lawson

              Good Afternoon Kyle,

               

              Thanks for the comment.  The hotfix above is (at this point) 10 years old, and the defect is only if you see the message and then the process actually does fail to start.  Most customers don't run into this issue.

               

              But in short, AR System Server does take a long time to start.  The reason is fairly simple: It has to load all of the system objects (forms, filters, escalations, guides, menus, etc.) into its memory.  So this depends on the database being available and performing at its best.  You might be able to tweak your DB to include a index or something on some of the actlink or field tables (or any of the other tables) that will help a little bit.  But, I don't think there is any way to make such a monumental impact to the system where you'll be able to boot before the timeout of that message.

               

              The message is not a error, it is just telling you exactly what it says: the service could not start in a timely manner.  Not that it couldn't start, not that it wont start, or any other message.  It just didn't start before Microsoft felt like a normal program should.  AR System Server is not a normal application though

               

              The real trade-off here is that AR System does a lot of extra work up front to load objects into cache, so later it can serve that information much more quickly, AND so it doesn't make DB calls back in the middle of a user operation to load the objects on demand.

               

              TLDR:  It's like a space rocket, it starts up slow so it can go super fast later!  (And I'm not sure there are very many things you can do to make more than a 10% difference)

              5 of 5 people found this helpful
              • 4. Re: Remedy AR System Server service slow to start
                Kyle Smithey

                Thanks for the response!

                • 5. Re: Remedy AR System Server service slow to start
                  Kyle Smithey

                  Thank you! Good info.

                  • 6. Re: Remedy AR System Server service slow to start
                    Jason Miller

                    To the point of it takes longer for AR Server to start than Windows wants to wait around for, I have developed the habit of closing the service stating window as soon as it pops up. I already know it is going to error so why not just close it.

                     

                    I tail the armonitor.log and arerror.log during startup. That is what truly shows progress and errors as the start up process proceeded.

                    • 7. Re: Remedy AR System Server service slow to start
                      Adam Lawson

                      This is true.  I've actually built a powershell script to do it all for me:

                       

                      How to Restart All of Remedy, With One Click!

                       

                      I can, with that script, restart my whole environment (across the network) with one click...  Including MT's, SmartIT, or anything else you can imagine...  (not to include there are no messages with it.

                       

                      AR System is a Java process, meaning it is spun up inside of a JVM, of which Windows does NOT have access to.  Windows only gets a "up/down" flag from the process, as it run inside of a walled garden that Windows doesn't naturally understand.  So - I say all that to say this - The AR Monitor log is the only place you can go to see anything important about AR System Server.

                      2 of 2 people found this helpful