Thanks for Your suggestions Andreas Mitterdorfer Ravindrakumar Rodge .
Apologies for the confusion. Based on the interactions with BMC support, we tried few things to see if the server comes up
Earlier the start ups were failing with unique constraint violation errors. Server tries to import the system forms, instead of overwriting it tries to INSERT new rows in field table for existing columns and gets the below error and (after 7-8 minutes) fails with an initialization error. Neither We nor BMC have an explanation for this behavior.
An example here. Please note that this just for one of the forms and this error repeats for all almost all the system/install form imports (50+ forms) as part of the start up.
<SQL > <TID: 0000000077> <RPC ID: 0000000000> <Queue: Init > <Client-RPC: 10 > <USER: ARSERVER > <Overlay-Group: 0 > /* Fri Jan 05 2018 13:03:31.9780 */ INSERT INTO field (fieldId, schemaId, overlayGroup, changeDiary, createMode, defaultValue, fbOption, fieldType, fOption, helpText, lastChanged, fieldName, objProp, overlayProp, owner, recordId, resolvedfieldId, resolvedName, smObjProp, timestamp, datatype) VALUES (1, 3, '0', NULL, 2, NULL, 0, 1, 3, NULL, 'jbliss', 'Request ID', NULL, 0, 'ARSERVER', 0, 1, 'Request ID', NULL, 1157149850, 4)
<SQL > <TID: 0000000077> <RPC ID: 0000000000> <Queue: Init > <Client-RPC: 10 > <USER: ARSERVER > <Overlay-Group: 0 > /* Fri Jan 05 2018 13:03:32.0530 */ SELECT 1 FROM DUAL
<SQL > <TID: 0000000077> <RPC ID: 0000000000> <Queue: Init > <Client-RPC: 10 > <USER: ARSERVER > <Overlay-Group: 0 > /* Fri Jan 05 2018 13:03:32.0860 */ OK
<SQL > <TID: 0000000077> <RPC ID: 0000000000> <Queue: Init > <Client-RPC: 10 > <USER: ARSERVER > <Overlay-Group: 0 > /* Fri Jan 05 2018 13:03:32.0920 */ org.springframework.orm.jpa.JpaSystemException: Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.4.2.v20130514-5956486): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (ARADMIN.SYS_C00226017) violated
ARADMIN.SYS_C00226017 is composed of the following columns
SQL> SELECT column_name, position FROM all_cons_columns WHERE constraint_name = 'SYS_C00226017';
SCHEMAID 1
FIELDID 2
OVERLAYGROUP 3
To isolate the root cause and in an attempt to get the server up
(1) BMC suggested to remove/rename the system & install forms so that installer can't find them to import. Hence arerror.log reports since it can't find the def files, while the SQL log wont have any trace since the actual import did NOT happen
(2) Since we have so many group records, we suspected that could be another reason for the server to choke, so we cleaned up all custom group and just retained the bare minimum. This caused the server to fail fast on start up ( instead of waiting for 7-8 mins)
(3) We ran the DB check and gave the logs ( Running the database consistency checker - Documentation for BMC Remedy Action Request System 9.1 - BMC Documentation ) and BMC did not find any red flags.
(4) Currently, start up logging is enabled as well (arserver.jar -t -s) , to get any clue for the start up failures.
And thanks for the hint on bundle cache, this is the error in see in the latest log under bundle cache
!ENTRY com.bmc.arsys.restapi 4 0 2018-02-02 10:40:21.886
!MESSAGE FrameworkEvent ERROR
!STACK 0
java.lang.NullPointerException
at com.bmc.arsys.restapi.resourcemanager.RestfulConnector.removeResource(RestfulConnector.java:85)
at com.bmc.arsys.restapi.resourcemanager.ResourceTracker.removedService(ResourceTracker.java:57)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:956)
at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1)
at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:902)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:107)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEvent(BundleContextImpl.java:861)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:225)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.unregisterServices(ServiceRegistry.java:635)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.close(BundleContextImpl.java:88)
at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:514)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:566)
at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1206)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.decFWSL(StartLevelManager.java:592)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:257)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:284)
at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:692)
at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:600)
at org.eclipse.osgi.framework.internal.core.InternalSystemBundle$1.run(InternalSystemBundle.java:261)
at java.lang.Thread.run(Thread.java:748)
I will try to clear the bundle cache and rerun the db checker and see if i can find any more info.
On a different note, we have a weird confusion, our DB character set is UTF8 and we store and some unicode chars as well, but BMC thinks that we are non-unicode ( to decide which dbchecker has to be run, since unicode and nonunicode has different versions of checker). Any inputs?
ar.conf ---> Db-Character-Set: UTF-8
DB --> SQL> SELECT PARAMETER, VALUE FROM v$nls_parameters WHERE parameter LIKE '%CHARACTERSET';
NLS_CHARACTERSET
UTF8
NLS_NCHAR_CHARACTERSET
AL16UTF16
Just want to make sure that we are running the correct version of DB checker.