1 2 Previous Next 15 Replies Latest reply: Jul 31, 2012 10:52 AM by Jim Wilson RSS

BL 8.2 vCenter and WM

Ulf Karlsson

Hello,

 

We are working to setup BL8.2 talking to a vCenter host WM ware and then looking to use the GuestPackages jobs and features.

Got past the first hurdle to get the vcenter node connected, and we have been able to succesfully try some simple action such as reboot guest etc.

 

However, when we look to be more advanced creating a virtual guset package / job we get a popup in the concosle complaining about heap space on the vCenter node. "Java Heap Space:  on host <our node>.

 

This happens at step 5 in the creating a Job wizard.

 

On the vcenter client side, there are error logs in the BL agent such as:

 

e13092023de20b492e47 0000000327 05/02/12 16:10:31.515 ERROR    rscd -  <IPADDR> 5640 BladeLogicRSCD@ <VCENTERNODE>->administrator@ <VCENTERNODE>1:PrivilegeMapped (BLAdmins:<USER> ): CM: DAAL exception in DAALService::getAssetChildrenStream ; Asset: BMC_VMware_Networks: <VCENTERNODE>:/BMC_VMware_VirtualInfrastructureManager/BMC_VMware_Networks ; Module: DAAL::AssetStream::openGetDescendants ; Error code: 800 ; Error message: "Java heap space" ; Detail message: "Error returned from plug-in ; Plug-in: /BMC_VMware_VirtualInfrastructureManager_win64 ; Function: blAsset_OpenGetDescendants ; Stream root asset: BMC_VMware_Networks: <VCENTERNODE>:/BMC_VMware_VirtualInfrastructureManager/BMC_VMware_Networks ; Error code: 800 ; Error message: Java heap space ; Error details: bl_OpenGetDescendants failed: Exception occurred in plugin: com.bladelogic.om.infra.daal.jco.CustomObjectException: Java heap space"

 

Another observation is that agent is using quite a lot of memory.

 

Starting this thread in hope for some advice or at least some confirmations that someone out there got past this point and are able to use these features.

 

We tried the first support advice about java heapspace on our Windows  BlApp server, with no success and are are still waiting for

the next support advice.

 

// Ulf

  • 1. BL 8.2 vCenter and WM
    Amol Vaikar

    >>We tried the first support advice about java heapspace on our Windows  BlApp server, with no success

    Could you please elaborate on this?

    What did they suggest?

    Increasing the heap memory allocated for this?

    How much did you change it to?

     

    >>Agent is using quite a lot of memory

    How much?

    What other operations are being done on this agent?

    Could you take a look at the agent log and attach paste its tail here?

  • 2. BL 8.2 vCenter and WM
    Ulf Karlsson

    Thanks for the discussion activity,

     

    1st' support advice was to up the java heapspace on our Win 64 bit Blapp server - 4G, we done that, but the issue really is on the client side on the vCentere node. The tail of the agent log message is what is in the intital post.

     

    We have around 600 virtual servers in this VM cluster behind, Its possible to list and see info about them over the vcenter component. Mem Consumption for BL agent on the vCenter node was around 5-600 Mb

    if I dont remember wrong.

     

    No payload of usueful traffic to the BL agent, only the BL Console GUI actions mentioned above.

     

    I would just feal good to hear another user/customer had successfully got beyond this point or to hear someone else suffering from a similar issue.

     

    Heapspace message log can also be misleading, since it could be agent software just misbehaving.

    I find it quite difficult to troublehoot this whole chain of stuff, any further help is welcome.  

  • 3. Re: BL 8.2 vCenter and WM
    Umar Shaikh

    Could you let us know the configuration of your vCenter server? Is it a 32-bit server? What is the available RAM on this server?

     

    How many Virtual Networks are configured on this vCenter? Can you fire the following BLCLI? Does this fail too?

     

    blcli -v profileName -r roleName Virtualization listVirtualEntitiesByEntityManagerServerNameAndEntityType vCenterServerName VMwareNetwork

     

    Can you restart the RSCD agent on the vCenter server and try this again?

     

    We can increase the heap space on the agent side as well, but the default should be good enough to handle a pretty large environment typically.

     

    It would be essential to have this information before a next step could be decided on...

  • 4. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Thanks, here is what I can provide now about our vCenter node - data extracted via BL information elements :

     

    64 bit Win OS.     :    Microsoft Windows Server 2008 R2 Standard
    BL Agent version  :    8.2.0.158
    PhyMemFree    90272
    PhyMemTotal   10485304
    VirtMemFree    3717320
    VirtMemTotal    17275316

     

    I need to get back later about the # virtual network question and running the specific BL CLI cmd.

     

     

     

  • 5. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Mem consumption for agent is: 460 Mb

     

    88 Distributed virtual switches. (If that was the question, I am a bit unsure)

     

    We worked with a fresh restarted agent:

     

    blcli -v profileName -r roleName Virtualization listVirtualEntitiesByEntityManagerServerNameAndEntityType vCenterServerName  : Goes well and list ~600 line items.


    The problem is simplest reproducable by:

    Having a depot virtual guest BLP package. Going into the WM Network Connection Tab hitting the Plus and then selecting the vcenter server in first dialog.

    Generates the Hepaspace popup in the Console plus this entry in the Agent Logfile.


    05/09/12 09:15:47.496 ERROR    blcoserver -  VC0001 6220 BladeLogicRSCD (SYSTEM): (Not_available): Java heap space
    05/09/12 09:15:47.496 ERROR    blcoserver -  VC0001 6220 BladeLogicRSCD (SYSTEM): (Not_available): Exception occurred in plugin: com.bladelogic.om.infra.daal.jco.CustomObjectException: Java heap space com.bladelogic.virtualization.jpa.VirtCustomObject.getDescendants(VirtCustomObject.java:227) com.bladelogic.om.infra.daal.jco.GenericPlugin.notify(GenericPlugin.java:156) com.bladelogic.om.infra.daal.jco.JPAShim.getDescendants(JPAShim.java:447) com.bladelogic.om.infra.daal.jco.JPAShim.getDescendants(JPAShim.java:419) com.bladelogic.om.infra.daal.jco.JPAShim.getDescendants(JPAShim.java:398) Caused by: java.lang.OutOfMemoryError: Java heap space org.apache.axis.message.SAX2EventRecorder$objArrayVector.add(SAX2EventRecorder.java:254) org.apache.axis.message.SAX2EventRecorder.startElement(SAX2EventRecorder.java:85) org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1052) com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) javax.xml.parsers.SAXParser.parse(Unknown Source) org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227) org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696) org.apache.axis.Message.getSOAPEnvelope(Message.java:435) org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62) org.apache.axis.client.AxisClient.invoke(AxisClient.java:206) org.apache.axis.client.Call.invokeEngine(Call.java:2784) org.apache.axis.client.Call.invoke(Call.java:2767) org.apache.axis.client.Call.invoke(Call.java:2443) org.apache.axis.client.Call.invoke(Call.java:2366) org.apache.axis.client.Call.invoke(Call.java:1812) com.vmware.vim25.VimBindingStub.retrieveProperties(VimBindingStub.java:44531) com.bladelogic.om.virtualization.vmware.VMWareWebService.getManagedObjectReferences(VMWareWebService.java:3709) com.bladelogic.om.virtualization.vmware.livebrowse.DataCenterNetworksProcessor.process(DataCenterNetworksProcessor.java:135) com.bladelogic.om.virtualization.vmware.livebrowse.AbstractVMWareProcessor.process(AbstractVMWareProcessor.java:167) com.bladelogic.om.virtualization.vmware.VMWareWebService.getAssetData(VMWareWebService.java:3943) com.bladelogic.om.virtualization.vmware.VMWareWebService.getVMWareInfo(VMWareWebService.java:2356) com.bladelogic.om.virtualization.vmware.VMWareWebService.getVMWareInfo(VMWareWebService.java:2262) 
    05/09/12 09:15:47.543 ERROR    blcoserver -  VC0001 6220 BladeLogicRSCD (SYSTEM): (Not_available): Exception occurred in plugin: com.bladelogic.om.infra.daal.jco.CustomObjectException: Java heap space bl_OpenGetDescendants failed: Exception occurred in plugin: com.bladelogic.om.infra.daal.jco.CustomObjectException: Java heap space com.bladelogic.virtualization.jpa.VirtCustomObject.getDescendants(VirtCustomObject.java:227) com.bladelogic.om.infra.daal.jco.GenericPlugin.notify(GenericPlugin.java:156) com.bladelogic.om.infra.daal.jco.JPAShim.getDescendants(JPAShim.java:447) com.bladelogic.om.infra.daal.jco.JPAShim.getDescendants(JPAShim.java:419) com.bladelogic.om.infra.daal.jco.JPAShim.getDescendants(JPAShim.java:398) Caused by: java.lang.OutOfMemoryError: Java heap space org.apache.axis.message.SAX2EventRecorder$objArrayVector.add(SAX2EventRecorder.java:254) org.apache.axis.message.SAX2EventRecorder.startElement(SAX2EventRecorder.java:85) org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1052) com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source) com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) javax.xml.parsers.SAXParser.parse(Unknown Source) org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227) org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696) org.apache.axis.Message.getSOAPEnvelope(Message.java:435) org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62) org.apache.axis.client.AxisClient.invoke(AxisClient.java:206) org.apache.axis.client.Call.invokeEngine(Call.java:2784) org.apache.axis.client.Call.invoke(Call.java:2767) org.apache.axis.client.Call.invoke(Call.java:2443) org.apache.axis.client.Call.invoke(Call.java:2366) org.apache.axis.client.Call.invoke(Call.java:1812) com.vmware.vim25.VimBindingStub.retrieveProperties(VimBindingStub.java:44531) com.bladelogic.om.virtualization.vmware.VMWareWebService.getManagedObjectReferences(VMWareWebService.java:3709) com.bladelogic.om.virtualization.vmware.livebrowse.DataCenterNetworksProcessor.process(DataCenterNetworksProcessor.java:135) com.bladelogic.om.virtualization.vmware.livebrowse.AbstractVMWareProcessor.process(AbstractVMWareProcessor.java:167) com.bladelogic.om.virtualization.vmware.VMWareWebService.getAssetData(VMWareWebService.java:3943) com.bladelogic.om.virtualization.vmware.VMWareWebService.getVMWareInfo(VMWareWebService.java:2356) com.bladelogic.om.virtualization.vmware.VMWareWebService.getVMWareInfo(VMWareWebService.java:2262) 
    05/09/12 09:15:47.621 ERROR    blcoserver -  VC0001 6220 BladeLogicRSCD (SYSTEM): (Not_available): DAAL exception in DAALService::assetStreamGetNext ; Asset: [NULL] ; Module: DAAL::AssetStream::openGetDescendants ; Error code: 800 ; Error message: "Java heap space" ; Detail message: "Error returned from plug-in ; Plug-in: /BMC_VMware_VirtualInfrastructureManager_win64 ; Function: blAsset_OpenGetDescendants ; Stream root asset: BMC_VMware_Networks:vc0001.tclk.se.om.com:/BMC_VMware_VirtualInfrastructureManager/BMC_VMware_Networks ; Error code: 800 ; Error message: Java heap space ; Error details: bl_OpenGetDescendants failed: Exception occurred in plugin: com.bladelogic.om.infra.daal.jco.CustomObjectException: Java heap space"

  • 6. Re: BL 8.2 vCenter and WM
    Umar Shaikh

    You've run the BLCLI for VirtualNetworks right? Or for VirtualMachines? Wanted the number of Virtual Networks You mentioned earlier you had 600 VM's approx and the results you are getting here too are 600. Do you have 600 Virtual Networks? Can you confirm the BLCLI you ran? It should be:

     

    blcli -v profileName -r roleName Virtualization listVirtualEntitiesByEntityManagerServerNameAndEntityType vCenterServerName VMwareNetwork


    I'll look for options and get back to you in some time meanwhile..

  • 7. Re: BL 8.2 vCenter and WM
    Umar Shaikh

    You'll have to increase the heap space for CO Server to go further.

     

    To increase the heap space, browse to the JPAProgramConfig.xml file on the vCenter server within the following folder:

     

    <RSCD_DIR>/daal/Implementation/BMC_VMware_VirtualInfrastructureManager_win64/win64/JPAProgramConfig.xml

     

    You'll have to change the following:

     

    <jvmArg>-Xms64m -Xmx512m</jvmArg>

     

    to:

     

    <jvmArg>-Xms64m -Xmx1024m</jvmArg>

     

    Restart the RSCD agent on the vCenter post this and try your use case once more.

     

    Meanwhile, can you cross-check your BLCLI command again for VirtualNetworks? Post you apply the fix, try running the same BLCLI multiple times. Also, would like to know the number of ESX Hosts and Clusters you have in your environment.

     

    Do let us know how it worked out...

  • 8. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Thanks, we will try the proposed heapsace change on the agent and post our findings.

     

    Regarding the other open questions:

     

    Yes the cmd listVirtualEntitiesByEntityManagerServerNameAndEntityType <server> VMwareNetwork

    fails, and provokes the samer error.

     

    We have 4 clusters, and number of ESX host is 43.

  • 9. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Thanks for the ongoing discussion,

     

    Unfortunately we are still stuck, I repeat that we have 88 virtual networks we suppose to get listed.
    We tried upping the java heap for the agent to 1024 and then even to 2048 and still no positive results.
    Thing is the agent can't consume the whole host either.

     

    Time for me to digest what to do about the whole situation.

  • 10. Re: BL 8.2 vCenter and WM
    Bill Robinson

    Have you opened a ticket w/ support about this yet ?

  • 11. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Yes we have: ISS03950556

  • 12. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Just to keep information going and everybody in the loop on a high level - we are working with bmc support
    to resolve the issue but still no luck.

     

    We investigated the file:

    C:\Program Files\BMC Software\BladeLogic\RSCD\daal\Implementation\BMC_VMware_VirtualInfrastructureManager_win64\win64\vmware.properties

     

    Just a while ago we ran with a fixed jar package for the agent side - it still fails.

     

    I will post what eventually resolves the problem. I will make sure to do that even if we may have to choose to go directly on the vm-api and bypass the bmc vcenter api.

  • 13. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Support has given us a pre-delivery of a fix, that we currently are running with.

    With this fix online we have been been able to see we can create virtual servers using BL
    towards VM.

     

    Reference to the fix, what exactly was the root cause, release date for a verified public fix is
    for bmc to make official.

     

    Thanks everybody that helped out, much apreciated! 

  • 14. Re: BL 8.2 vCenter and WM
    Ulf Karlsson

    Where we are now :  past the intital bug fix we ran into more bugs we also have reported to support

     

    Basically we have despite quite some effort now not yet been able to to create any virtual server inside VM with a customized touch such as naming the server or settig any customized configuration values inside.

     

    While we still expect support to lead us through and see this functioning we had to swith design in current project and are bypassing the virtual Blade Logic parts and instead talking directly with the VM-API to produce the server features we want to automate.

1 2 Previous Next