Perform data collection is failing on Solaris 10 with the error "SemLock::createSem() - semget(0xc1000623) failed, Error = No space left on device" or SMQue::AllocMem shmget() failed, Error = No space left on device

Version 3
    Share This:

    This document contains official content from the BMC Software Knowledge Base. It is automatically updated when the knowledge article is modified.


    PRODUCT:

    BMC Performance Assurance for Servers


    APPLIES TO:

    BMC Performance Assurance for Servers



    PROBLEM:

    The most likely problem is that in Solaris 10 the semaphore settings (or 'setting' since only semsys:seminfo_semmni still needs to be set in Solaris 10) isn't done through the /etc/system file anymore. The setting on Solaris 10 is controlled by the "project.max-sem-ids" parameter via the 'prctl' command. In most environments the system defaults been sufficient for Perform to run on the machine but some environments do require project level changes for Perform to allocate the required number of semaphores.

    Although insufficient semaphores available at the project level is the most common problem, it is also possible for all available shared memory segments to be exhaused. That will result in the following error:

        Perform Agent (672) SMQue::AllocMem shmget() failed, Error = No space left on device  
      Perform Agent (672) BaseQue::ReMap - AllocMem() failed  
      Perform Agent (672) BaseQue::WriteGrab() ReMap() failed  
      Perform Agent (672) CIQ::Insert() - WriteGrab failed  
      Perform Agent (672) MrPoolEntry::Register - ERROR: Failed to insert ([GROUP]) at interval (10) in CIQ (bgscollect:noInstance)  
      Perform Agent (672) UDR History : UDR request terminated - MrPool registration failed - Collector registration failed(:noInstance:'[GROUP]')  
      
       
    • BMC Performance Assurance for Unix  7.4.00,  7.3.00,  7.2.10,  7.2.00
      
       
    • Solaris 10

     


    SOLUTION:

     

    Legacy ID:KA290922

      

    Section I: Increasing the project semaphores defined on the system

      
      Steps to increase the number of project semaphores available on the system.  
     
     
       Step 1   
     
    Find the pid of the running bgsagent process:  
     
     
       > ps -ef | grep bgsagent   
    perform   590     1   0 13:07:44 ?           0:01 /usr/adm/best1_7.3.00/bgs/bin/bgsagent -b /usr/adm/best1_7.3.00 -a 57300 -d 573   
       paska   634   556   0 13:15:02 pts/2       0:00 grep bgsagent   
     
     
     
       Step 2   
     
    Get the project name for the running bgsgent process:  
     
     
       > ps -o project -p 590   
    PROJECT   
    default   
     
     
     
       Step 3   
     
    As root, increase the number of available semaphore sets for that project:  
     
     
       # projmod -a -K "project.max-sem-ids=(priv,1k,deny)" default   
     
     
    In the above example that will increase the number of available semaphore sets to 1000. You could pick a lower or higher value.  
     
    That updates the /etc/project file on the machine.  
     
    My /etc/project file now looks like this:  
     
     
       > cat /etc/project   
    system:0::::   
    user.root:1::::   
    noproject:2::::   
    default:3::::project.max-sem-ids=(priv,1000,deny)   
    group.staff:10::::   
     
     
    So the setting of 1000 semaphore sets has been added to the /etc/project file by the 'projmod' command.  
     
     
       Step 4   
     
    Restart the Perform Agent. A reboot is not necessary for this change to take effect.  
     
     
       Step 5   
     
    You can verify that the change has taken effect using the 'prctl' command.  
     
    So, before making the change, run the following command:  
     
     
       #  prctl -n project.max-sem-ids 590   
    process: 590: /usr/adm/best1_7.3.00/bgs/bin/bgsagent -b /usr/adm/best1_7.3.00 -a 573   
    NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT   
    project.max-sem-ids   
            privileged        128       -   deny                                 -   
            system          16.8M     max   deny   
     
     
    That means there are 128 total semaphore sets available in that project.  
     
    Then, after making the change (and stopped and restarted the Perform Agent) run that command again:  
     
     
       # prctl -n project.max-sem-ids 663   
    process: 663: /usr/adm/best1_7.3.00/bgs/bin/bgsagent -b /usr/adm/best1_7.3.00 -a 573   
    NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT   
    project.max-sem-ids   
            privileged      1.00K       -   deny                                 -   
            system          16.8M     max   deny                                 -   
     
     
    Now you can see there are a total of 1000 semaphore sets available to the process in that project.  
     
     

    Q: Is the 'project' going to be different on every server?

    It depends on your system configuration. Your Solaris system administration team would better be able to answer that question for your specific environment.  
     
    Our expectation is that if you are just doing default OS installs and aren't breaking processes out by project then the project will be the same for all systems and will probably be the "default" project.  
     
     

    Q: What could cause the project for Perform to change after a system reboot?

    The Perform product doesn't attempt to join any specific project on Solaris. It just runs under the project of the starting process. On most Solaris machines this is the 'default' project. On possible way for the Perform Agent to be started under the 'system' group is for an rc startup script to be configured that runs $BEST1_HOME/bgs/scripts/best1_agent_start on the machine to start the Perform Agent. To avoid this possibility use the '$BEST1_HOME/bgs/scripts/best1collect -n localhost -q' to start the Perform Agent if it is being started through an rc startup script (since that would start the agent exactly the same way that the console starts the agent remotely).  
     
    You can see the best1agent_start command being used to start the Perform Agent on a machine via the following messages in the [hostname]-bgsagent_6767.log:  
     
         Perform Agent (450) Started by user 'bgsagent' at host '[local hostname]'   
      Perform Agent (450) Trying to connect to Agent on node localhost using port 6767   
     
     
    When the best1agent_start (or bgsagent_start) script is used to start the Perform Agent it uses the username 'bgsagent' as shown in the above log example.  
     
      

    Section II: Increasing the project shared memory defined on the system

      
      First, check which application is using all of the available IPC semaphores on the machine:  
     
       # ipcs -a   
     
     
    If the application really isn't supposed to be using that much shared memory resolve whatever is causing it to allocate those IPC shared memory resources. If the application is working as designed then the next step is to increase the quantity of shared memory resources defined on the machine.  
     
    To resolve "SMQue::AllocMem shmget() failed, Error = No space left on device" problem it is necessary to increase the quantity of available shared memory segments defined on the machine or   
     
    What needs to change is the "project.max-shm-ids" project setting. This parameter can be changed without a system reboot.  
     
    Search for solution id 1296 and refer to Section I: Increasing the project semaphores defined on the system in this document but anywhere it says, "project.max-sem-ids=" you'd replace that with "project.max-shm-ids=".  
     
    For example, the command to run in Step 3 above is:  
     
     
       # projmod -a -K "project.max-shm-ids=(priv,1k,deny)" default   
     
     
    Assuming that 'default' is the name of the project that the Perform product belongs to.  
     
    For Solaris 10, the default IPC shared memory count available for a project is 128 segments.  
      
    Related Products:  
       
    1. BMC Performance Assurance for Servers

     


    Article Number:

    000302702


    Article Type:

    Solutions to a Product Problem



      Looking for additional information?    Search BMC Support  or  Browse Knowledge Articles