Remedy - D2P - ARS d2p package rolled back with error "ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start"

Version 7
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    Remedy AR System Server


    COMPONENT:

    AR System


    APPLIES TO:

    AR System Server 9.1.04 and higher.



    PROBLEM:

    After having applied 9.1.04 Patch 002 the SHARE:ApplicationProperties form shows the expected version:


              User-added image


    The Deployment Management Console shows also the package having been deployed:

              User-added image


    However in the arerror.log and arexception.log the AR System and other clients version string remains with 9.1.04.

    arerror.log showing the Assignment Engine

              User-added image

    arexception.log where Patch 002 is originally displayed but then is back to the unpatched version

              User-added image


    Where can be checked what happened?

     


    CAUSE:

    The Email Engine service was disabled and since the deployer could not start it, it is treated as a failure which triggers a roll back because the Email Engine requires a restart to be upgraded to Patch 002, it is working as designed.


    SOLUTION:

    The log to check the activity performed during the Patch deployment is arfiledeployer.log.

    Note, the reasons for the patch to roll back can be diverse and have different root causes so the arfiledeployer.log needs to be checked to find out what happened.
    Below is an example of a scenario where the E-mail Engine Windows Service was disabled therefor it would not be able to be started by the deployer and the procedure fails.

    If you encounter a similar scenario perform the following:

    1- The Email Engine service needs to be enabled and started prior the Patch deployment - BUT - check below.
    2- Note, if you have a lot of messages in the main AR System Email Messages form and the Email Engine was disabled for any specific reason to not send any messages or notifications, enabling the Email Engine service these will try to be sent.
    3- To avoid this and if you don't want the Email Engine to process any messages, before enabling and starting its service go the main incoming/outgoing mailbox configuration and set the status to "Disabled".
    4- Then enable and start the Email Engine service/process.
    5- Roll back the CMDB and ARS patches via the AR System Deployment Manager Console.
    6- Re-import the patches and deploy them.


    Now here below is what the arfiledeployer.log displays when rolling back the Patch where the Email Engine service is disabled.

    It can be observed that after having backup the original files the deployment process will restart all the AR processes/services:
    -------------
    2018-03-16 09:45:15,813 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Connected to ARServer.
    2018-03-16 09:45:15,813 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Server process started successfully
    2018-03-16 09:45:15,813 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Starting process BMC:CarteServer
    2018-03-16 09:45:16,847 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Starting process BMC:EmailEngine
    2018-03-16 09:45:17,004 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Starting process BMC:JavaPluginServer
    2018-03-16 09:45:20,083 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verifying if child process is running
    2018-03-16 09:45:30,090 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:JavaPluginServer is running
    2018-03-16 09:45:30,090 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:DefaultJavaPluginServer is up and running
    2018-03-16 09:45:30,090 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:NormalizationEngine is up and running
    2018-03-16 09:45:30,090 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:AtriumSharedPluginServer is up and running
    2018-03-16 09:45:30,090 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:CarteServer is running
    2018-03-16 09:45:30,090 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:CarteServer is up and running
    2018-03-16 09:45:40,105 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:JavaPluginServer is running
    2018-03-16 09:45:40,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:DefaultJavaPluginServer is up and running
    2018-03-16 09:45:40,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:NormalizationEngine is up and running
    2018-03-16 09:45:40,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:AtriumSharedPluginServer is up and running
    2018-03-16 09:45:40,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:CarteServer is running
    2018-03-16 09:45:40,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:CarteServer is up and running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:JavaPluginServer is running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:DefaultJavaPluginServer is up and running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:NormalizationEngine is up and running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:AtriumSharedPluginServer is up and running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:CarteServer is running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:CarteServer is up and running
    2018-03-16 09:45:50,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verifying if external process is running
    2018-03-16 09:46:00,121 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running

    2018-03-16 09:46:00,153 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start.
    2018-03-16 09:46:10,166 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running
    2018-03-16 09:46:10,198 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start.
    2018-03-16 09:46:20,198 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running
    2018-03-16 09:46:20,229 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start.
    2018-03-16 09:46:30,229 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running
    -------------


    The Email Engine could not start and this was detected as a failure:
    -------------
    2018-03-16 09:48:50,763 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Error while deploying payload with DEPLOYMENT ID IDGAA5V0GEOPXAPE1HFZPD4R5AQKZ5 : Errors while starting processes and verifying deployment. [Errors in starting processes and verifying deployment. [Process BMC:EmailEngine of type BMC:EmailEngine failed to start in 180 seconds]]

    2018-03-16 09:48:50,763 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Update the status of the payload as DEPLOYMENT FAILED

    2018-03-16 10:06:42,419 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start.

    2018-03-16 10:06:42,419 [main] WARN  com.bmc.arsys.filedeployer.PayloadProcessor  - Failed to start processes during rollback
    java.lang.Exception: Errors while starting processes and verifying deployment. [Errors in starting processes and verifying deployment. [Process BMC:EmailEngine of type BMC:EmailEngine failed to start in 180 seconds]]
    -------------


    Since the E-mail engine could not be started the deployment process will then start to roll back the Patch files:
    -------------
    2018-03-16 09:48:50,857 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Update the status of the payload as PENDING ROLLBACK
    ...etc...
    2018-03-16 09:48:55,717 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rolling back deployed files
    2018-03-16 09:48:55,717 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rolling back zip file D:\Program Files\BMC Software\ARSystem\server_common.zip
    2018-03-16 09:48:55,717 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rollback AREmail/emaildaemon.jar
    2018-03-16 09:48:55,810 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rollback AREmail/arapi91_build003.jar
    2018-03-16 09:48:55,810 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rollback approval/bin/arapi91_build003.jar
    2018-03-16 09:48:55,810 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rollback approval/bin/arasj91_build003.jar
    2018-03-16 09:48:55,951 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rollback diserver/data-integration/lib/arapi91_build003.jar
    2018-03-16 09:48:55,951 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Rollback lib/com.bmc.arsys.boot-9.1.04-SNAPSHOT.jar
    ...etc...

    2018-03-16 09:48:57,170 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Successfully rolled back the files.
    -------------


    And then the processes will be restarted again but since the Email Engine service is disabled it will not be possible to start:
    -------------
    2018-03-16 10:02:59,013 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Server process started successfully
    2018-03-16 10:02:59,013 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Starting process BMC:CarteServer
    2018-03-16 10:03:00,122 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Starting process BMC:EmailEngine
    2018-03-16 10:03:01,435 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Starting process BMC:JavaPluginServer
    2018-03-16 10:03:04,501 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verifying if child process is running
    2018-03-16 10:03:14,524 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:JavaPluginServer is running
    2018-03-16 10:03:14,524 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:DefaultJavaPluginServer is up and running
    2018-03-16 10:03:14,524 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:NormalizationEngine is up and running
    2018-03-16 10:03:14,524 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:AtriumSharedPluginServer is up and running
    2018-03-16 10:03:14,524 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:CarteServer is running
    ...etc...

    2018-03-16 10:03:44,821 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running
    2018-03-16 10:03:45,040 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start.
    2018-03-16 10:03:55,040 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running
    2018-03-16 10:03:55,165 [main] ERROR com.bmc.arsys.filedeployer.PayloadProcessor  - Process BMC:EmailEngine failed to start.
    2018-03-16 10:04:05,165 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Verify if process BMC:EmailEngine is running
    ...etc...2018-03-16 10:06:42,419 [main] WARN  com.bmc.arsys.filedeployer.PayloadProcessor  - Failed to start processes during rollback
    java.lang.Exception: Errors while starting processes and verifying deployment. [Errors in starting processes and verifying deployment. [Process BMC:EmailEngine of type BMC:EmailEngine failed to start in 180 seconds]]
        at com.bmc.arsys.filedeployer.PayloadProcessor.startAffectedProcessesAndVerifyDeployment(PayloadProcessor.java:821)
        at com.bmc.arsys.filedeployer.PayloadProcessor.processPendingRollback(PayloadProcessor.java:507)
        at com.bmc.arsys.filedeployer.PayloadProcessor.processDeploymentFailed(PayloadProcessor.java:478)
        at com.bmc.arsys.filedeployer.PayloadProcessor.processDownloaded(PayloadProcessor.java:694)
        at com.bmc.arsys.filedeployer.PayloadProcessor.process(PayloadProcessor.java:435)
        at com.bmc.arsys.filedeployer.PayloadProcessor.processNewPayloads(PayloadProcessor.java:360)
        at com.bmc.arsys.filedeployer.ARFileDeployerDaemon.startMainProcessing(ARFileDeployerDaemon.java:342)
        at com.bmc.arsys.filedeployer.ARFileDeployerDaemon.main(ARFileDeployerDaemon.java:631)
    -------------


    The deployer will then show the roll back completed:
    -------------
    2018-03-16 10:06:42,419 [main] INFO  com.bmc.arsys.filedeployer.PayloadProcessor  - Update the status of the payload as ROLLED BACK SUCCESS
    -------------

     


    Article Number:

    000150718


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles