3 Replies Latest reply on Jul 9, 2020 4:46 AM by Brice-Emmanuel Loiseaux

    CMDB connection error - CORBA.TIMEOUT

    Bernard Stern
      Share This:

      Since a couple of weeks we have this kind of errors on the consolidator

       

      140556830754560: 2020-07-03 14:53:37,564: cmdb_sync.target: INFO: CMDB Main: Failed to contact server. 13 attempts so far.

      140556830754560: 2020-07-03 14:55:02,581: cmdb_sync.target: ERROR: CMDB Main: Error retrieving server details - CORBA.TIMEOUT(omniORB.TIMEOUT_CallTimedOutOnClient, CORBA.COMPLETED_MAYBE)

       

      Running 13.3.0.5, one consolidtion and 2 scanning appliances.

       

      CMDB Dataset was emptied, and resync started (today).

       

      All CIs got created in the CMDB. During resync, at the beginning it was extremely slow, then sometimes connection errors,

      then suddenly it started to run correctly.

       

      Some logs of the tw_svc_cmdbsync_transformer.log

       

      140556813969152: 2020-07-03 11:58:26,293: cmdb_sync.resync: INFO: CMDB Main: Updated 1900 of 147085 CIs

      140556813969152: 2020-07-03 11:58:26,524: cmdb_sync.resync: INFO: CMDB Main: Updated 2000 of 147085 CIs

      140556813969152: 2020-07-03 12:01:08,015: cmdb_sync.resync: INFO: CMDB Main: Updated 2100 of 147085 CIs

      140556813969152: 2020-07-03 12:01:08,567: cmdb_sync.resync: INFO: CMDB Main: Updated 2200 of 147085 CIs

      140556813969152: 2020-07-03 12:01:13,729: cmdb_sync.resync: INFO: CMDB Main: Updated 2300 of 147085 CIs

      140556813969152: 2020-07-03 12:01:13,787: cmdb_sync.resync: INFO: CMDB Main: Updated 2400 of 147085 CIs

      140556813969152: 2020-07-03 12:01:13,787: cmdb_sync.resync: INFO: CMDB Main: Updated 2500 of 147085 CIs

      140556813969152: 2020-07-03 12:03:58,292: cmdb_sync.resync: INFO: CMDB Main: Updated 2600 of 147085 CIs

      140556813969152: 2020-07-03 12:04:00,115: cmdb_sync.resync: INFO: CMDB Main: Updated 2700 of 147085 CIs

      140556813969152: 2020-07-03 12:04:38,369: cmdb_sync.resync: INFO: CMDB Main: Updated 2800 of 147085 CIs

      140556813969152: 2020-07-03 12:04:38,371: cmdb_sync.resync: INFO: CMDB Main: Updated 2900 of 147085 CIs

      140556813969152: 2020-07-03 12:04:38,371: cmdb_sync.resync: INFO: CMDB Main: Updated 3000 of 147085 CIs

      140556813969152: 2020-07-03 12:07:24,034: cmdb_sync.resync: INFO: CMDB Main: Updated 3100 of 147085 CIs

      140556813969152: 2020-07-03 12:07:24,922: cmdb_sync.resync: INFO: CMDB Main: Updated 3200 of 147085 CIs

      140556813969152: 2020-07-03 12:08:03,153: cmdb_sync.resync: INFO: CMDB Main: Updated 3300 of 147085 CIs

      140556813969152: 2020-07-03 12:08:03,630: cmdb_sync.resync: INFO: CMDB Main: Updated 3400 of 147085 CIs

      140556813969152: 2020-07-03 12:08:03,976: cmdb_sync.resync: INFO: CMDB Main: Updated 3500 of 147085 CIs

      140556561200896: 2020-07-03 12:08:41,392: cmdb_sync.target: ERROR: CMDB Main: Error retrieving server details - CORBA.TIMEOUT(omniORB.TIMEOUT_CallTimedOutOnClient, CORBA.COMPLETED_MAYBE)

      140556813969152: 2020-07-03 12:10:53,655: cmdb_sync.resync: INFO: CMDB Main: Updated 3600 of 147085 CIs

       

      The CIs were completely resynchronized, but not the relations. This is what I see in the log file:

       

      140556813969152: 2020-07-03 13:43:56,269: cmdb_sync.resync: INFO: CMDB Main: Updated 147000 of 147085 CIs

      140556813969152: 2020-07-03 13:43:56,698: cmdb_sync.resync: INFO: CMDB Main: Updated 147086 of 147085 CIs

      140556813969152: 2020-07-03 13:44:00,706: cmdb_sync.resync: INFO: CMDB Main: Sync creation of 200714 relationships

       

      140556619949824: 2020-07-03 14:14:01,092: cmdb_sync.target: ERROR: CMDB Main: Failed to sync 100 CREATE relationship actions: (attempt 1): ERROR (91):

      RPC call failed; ONC/RPC call timed out

       

      140556619949824: 2020-07-03 14:14:01,092: cmdb_sync.target: INFO: CMDB Main: Check existence of 100 relationships after failed create (check attempt 1

      )...

      140556603164416: 2020-07-03 14:14:01,352: cmdb_sync.target: ERROR: CMDB Main: Failed to sync 100 CREATE relationship actions: (attempt 1): ERROR (91):

      RPC call failed; ONC/RPC call timed out

       

      and on it goes like that.

       

      We have another dataset on that CMDB which is fed from another consolidation appliance.

      That dataset does not show any problems at all. Any idea where I should be looking at? I have

      no access to the CMDB side of the show, so I'd have to organise a colleague for that.

        • 1. Re: CMDB connection error - CORBA.TIMEOUT
          Brice-Emmanuel Loiseaux

          "140556561200896: 2020-07-03 12:08:41,392: cmdb_sync.target: ERROR: CMDB Main: Error retrieving server details - CORBA.TIMEOUT(omniORB.TIMEOUT_CallTimedOutOnClient, CORBA.COMPLETED_MAYBE)"

          This is a message in sync transformer logs with means that the set of calls in order to regularly retrieve the CMDB server details (is CMDB reachable, is dataset there, is it empty or not, what are the defined companies) took longer to return from CMDB. This denotes the AR/CMDB/DB server is much busier than usual.

           

          "ERROR (91):  RPC call failed; ONC/RPC call timed out"

          This is also a message in sync transformer logs that says the CMDB API call (here a relationship creation bulk) fails with a time-out error meaning the API does not know if the call was processed by AR. It denotes a serious performance issue on AR side. You need to involve your CMDB expert.

          2 of 2 people found this helpful
          • 2. Re: CMDB connection error - CORBA.TIMEOUT
            Bernard Stern

            Thanks for the hints Brice-Emmanuel.

             

            For the record.

             

            The amount of RAM on the AR server was extended from 12 GB (BMC recommendation says the AR server admin) to 18 GB. The resync ran without any trouble after that.

             

            Nonetheless I find it strange that we experience troubles in cmdbsync and resync from one day to the next, without any configuration changes on the Discovery and the CMDB side of things.

            1 of 1 people found this helpful
            • 3. Re: CMDB connection error - CORBA.TIMEOUT
              Brice-Emmanuel Loiseaux

              Compared to continuous sync activities, resync asks much more to the AR server. Also, over time the CI population handled by AR is enriched and grows, as well as AR may have more work to do. You may have crossed a memory threshold lately.