Sentry for Hitachi failing with, "BCO_SCH_ERR011: ConnectionException ... WBEMException: CIM_ERR_FAILED ()" due to the SMI-S provider disconnecting during object discovery if an unsupported GXXX array is defined in the Hitachi Device Manager (HDvM)

Version 1
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    TrueSight Capacity Optimization Storage Extensions by Sentry


    COMPONENT:

    Sentry Integrator for TrueSight Capacity Optimization


    APPLIES TO:

    BMC TrueSight Capacity Optimization Storage Extensions by Sentry; BMC TrueSight Capacity Optimization 10.3, 10.0



    PROBLEM:

    The Sentry Hitachi Device Manager ETL is failing with the following error:

    2015-10-23 15:45:00,552 FAILED [service-136-1]- A connection error occured during processing.protocol:WBEM/CIM-XML, host:https://myhost:5989, login:bmc
    2015-10-23 15:45:00,553 ERROR [service-136-1]- BCO_SCH_ERR011: ConnectionException [Error=protocol.connection.full, Context=[WBEM/CIM-XML, https://myhost:5989, bmc, ]]
            at com.sentrysoftware.portal.client.wbem.impl.WBEMClientImpl.createError(WBEMClientImpl.java:368)
            at com.sentrysoftware.portal.client.wbem.impl.WBEMClientImpl.doRequest(WBEMClientImpl.java:295)
            at com.sentrysoftware.portal.client.wbem.impl.WBEMClientImpl.request(WBEMClientImpl.java:101)
            at com.sentrysoftware.portal.client.wbem.impl.WBEMClientImpl.request(WBEMClientImpl.java:60)
            at com.sentrysoftware.portal.common.client.concurrent.RequestTask.run(RequestTask.java:80)
            at java.lang.Thread.run(Thread.java:745)
    Caused by: WBEMException: CIM_ERR_FAILED ()
            at org.sblim.cimclient.internal.wbem.CloseableIteratorSAX.hasNext(CloseableIteratorSAX.java:111)
            at org.sblim.cimclient.internal.wbem.WBEMClientCIMXML.getIterator(WBEMClientCIMXML.java:1579)
            at org.sblim.cimclient.internal.wbem.WBEMClientCIMXML.enumerateInstances(WBEMClientCIMXML.java:707)
            at com.sentrysoftware.portal.client.wbem.impl.EnumeratesInstancesOperation.execute(WBEMClientImpl.java:476)
            at com.sentrysoftware.portal.client.wbem.impl.WBEMClientImpl.doRequest(WBEMClientImpl.java:205)
            ... 4 more


    The error is reproducable via the Hitachi Connection Tool, but the desired behavior is for the ETL catch this error and continue on to collect all of the other data that is available from the Hitachi API.

    In the Hitachi Connection Tool output you can see the error, but there are a lot of other messages around it:
    <-- cut -->
    HITACHI_StoragePoolPrimordial.InstanceID="VSP.94583.StoragePoolPrimordial.0";HITACHI_StoragePool.InstanceID="VSP.94583.9.96";1152787050496;

    WBEMException: CIM_ERR_FAILED ()
    org.sblim.cimclient.internal.wbem.CloseableIteratorSAX.hasNext(CloseableIteratorSAX.java:122)
    org.sblim.cimclient.internal.wbem.WBEMClientCIMXML.getIterator(WBEMClientCIMXML.java:2078)
    org.sblim.cimclient.internal.wbem.WBEMClientCIMXML.enumerateInstances(WBEMClientCIMXML.java:1050)
    net.sentrysoftware.wbem.AbstractWBEMEnvironment.executeQuery(WBEMEnvironment.java:145)
    net.sentrysoftware.wbem.SBLIMEnvironment.executeQuery(SBLIMEnvironment.java:1)
    net.sentrysoftware.hitachi.Hitachi_Client.HitachiClient(Hitachi_Client.java:88)
    net.sentrysoftware.wbem.browser.Hitachi_Browser$checkTimeout.run(Hitachi_Browser.java:505)
    java.lang.Thread.run(Unknown Source)


    WBEMException: CIM_ERR_FAILED ()
    org.sblim.cimclient.internal.wbem.CloseableIteratorSAX.hasNext(CloseableIteratorSAX.java:122)
    org.sblim.cimclient.internal.wbem.WBEMClientCIMXML.getIterator(WBEMClientCIMXML.java:2078)
    org.sblim.cimclient.internal.wbem.WBEMClientCIMXML.enumerateInstances(WBEMClientCIMXML.java:1050)
    net.sentrysoftware.wbem.AbstractWBEMEnvironment.executeQuery(WBEMEnvironment.java:145)
    net.sentrysoftware.wbem.SBLIMEnvironment.executeQuery(SBLIMEnvironment.java:1)
    net.sentrysoftware.hitachi.Hitachi_Client.HitachiClient(Hitachi_Client.java:88)
    net.sentrysoftware.wbem.browser.Hitachi_Browser$checkTimeout.run(Hitachi_Browser.java:505)
    java.lang.Thread.run(Unknown Source)

    HITACHI_StorageSystemDeviceDiskExtent:
    ======================================
    Time to complete: 3.214 seconds
    <-- cut -->


    CAUSE:

    The SMI-S provider side is what is terminating the object discovery call when the SMI-S provider tries to query the G600 devices list of primordial pools


    SOLUTION:

    Sentry has investigated this behavior and indicates that the SMI-S provider side is what is terminating the object discovery call when the SMI-S provider tries to query the G600 devices list of primordial pools.  
    The problem is that the discovery call itself is being terminated on the SMI-S provider side (the Hitachi side) and the ETL isn't able to prevent this disconnect.
    The ETL can't continue from this point because the object discovery return is incomplete and there is no way to restart the discovery from the point of the error (thus no way to get a complete discovery of everything that isn't related to the G600 array).

    So, the key point here is that it isn't the Sentry ETL code that is terminating during the discovery phase -- it is the return of object discovery data from the Hitachi SMI-S provider that is being terminated on the provider side and the ETL can't continue beyond that point because the object list is incomplete (everything that should be discovered beyond the first G600 primordial pool encountered is missing from the discovery).

    Resolution

    The problem is believed to be related to a defect in the Hitachi Device Manager (HDvM) version you are running and there is a newer HDvM version available that includes a fix.  The expectation is that if you upgrade to the version of the HDvM that includes this fix the entity discovery call will not generate the   WBEMException: CIM_ERR_FAILED error on the Hitachi SMI-S provider side during discovery and thus the Sentry Hitachi ETL will be able to successfully complete the discovery and move onto the data extraction phase. 

    Per Hitachi, there is a defect in HDvM where an error may occur when the CIM/WBEM function tries to obtain performance data information when the following storage systems are registered in HDvM; VSP G1000, VSP G200, G400, G600, G800. 

    A bug fix was made in v8.2.0-03, where the data of the older generation storage systems (i.e VSP, USP-V, etc) are obtainable even when VSP1000, VSP G200-800 are registered in HDvM.  

    Hitachi recommended you upgrade your HDvM to v8.2.0-03 or higher.  

    Workaround Recommendation

    What Sentry has suggested (based upon Hitachi's recommendation) in the interim, before the storage administrators are able to apply the required Hitachi patch, is to move the G600 array to a separate SMI-S provider instance.  According to Hitachi, the SMI-S Provider of Hitachi Device Manager will NOT support Hitachi VSP Gxxxx systems. Such systems ship with an embedded SMI-S Provider and Hitachi strongly advises to connect to this Embedded SMI-S Provider, as this is what they will focus on in the future. 

    Sentry understands that you may not want to have a separate instance of the Hitachi Device Manager just for one G600 system. However, to keep things manageable as much as possible, you could keep this instance of Hitachi Device Manager with all the systems declared in it (non G600 and also the G600), and use this instance to manage the storage systems from there. Then, separately, set up another instance of Hitachi Device Manager will all non-G600 systems (actually non Gxxxx systems), and point the Sentry ETL to that separate instance of Hitachi Device Manager. 

    Here is information regarding the Hitachi HDvM defect that Hitachi believes is the likely root cause of the problem you are seeing in the Sentry Hitachi ETL:
                                                                                                                                             
    TitleWhen VSP G1000, VSP G200, G400, G600, or G800 storage system is registered with Device Manager, the performance data of all storage systems cannot be obtained by CIM/WBEM function.
    Products and Versions AffectedHitachi Device Manager Software 8.0.0-00 to earlier than 8.2.0-03
    Products and Versions FixedHitachi Device Manager Software 8.2.0-03
    Problem DescriptionIf any of the following storage systems are registered with Device Manager, the performance data of the storage systems cannot be obtained by using the CIM/WBEM function:
    - VSP G1000
    - VSP G200, G400, G600, G800
    SymptomsA CIM client cannot obtain a performance data of all storage systems because the following requests from the client fail.
    The requests
    - EnumerateInstanceNames
    - EnumerateInstances

    The classes
    - CIM_ElementStatisticalData
    - HITACHI_BlockStatisticalDataFCPort
    - HITACHI_BlockStatisticalDataStorageSystem
    - HITACHI_BlockStatisticalDataStorageVolume
    Conditions Of OccurrenceThis problem occurs when any of the storage system is registered with Device Manager.
    - VSP G1000
    - VSP 200, G400, G600, G800
    Cause For FailureThis problem is due to a defect of the CIM/WBEM function of Device Manager server.
    The CIM/WBEM function tries to obtain unsupported performance data information about the following storage systems. As a result, the error occurs.
    - VSP G1000
    - VSP G200, G400, G600, G800
      
      

     


    Article Number:

    000100348


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles