Skip navigation
1 2 3 Previous Next

Discovery

163 posts
Share This:

(This blog posting pertains to any BMC Discovery product: on-premise or Helix Discovery)

 

Some customers rely on Discovery to see the Uptime for hosts and devices.

 

One problem: OOTB, the Uptime is missing for NetworkDevice and SNMPManagedDevice.

 

Why?  I'm not sure.  But, maybe it is because the SNMP sysUpTime isn't always reliable.

We have a discussion here:  does ADDM/Discovery discover network device sysUpTime?

 

According to google results, the maximum number of days that the SNMP sysUpTime will show is 497 days.

After 497 days, the SNMP sysUpTime value starts over again at 0.

 

Here are several articles I found about the topic:

    https://www.zabbix.com/forum/zabbix-help/47245-snmp-uptime-overflow-after-497-days

    https://knowledge.broadcom.com/external/article/167562/why-does-the-snmp-value-for-sysuptime-ov.html

    https://live.paloaltonetworks.com/t5/general-topics/limitation-when-monitoring-uptime-with-snmp/td-p/218504

 

If the device is up for longer than 497 days, then the value will not be correct for the sysUpTime.

 

You will have to decide whether or not that's a problem for you.

Devices should be upgraded (and consequently rebooted) with new firmware at intervals shorter than 497 days, if only for purposes of security, IMO.

 

We can provide the Uptime with a custom pattern.  But, there is nothing that the pattern can do to get around the 497 day limitation. 

The value is either correct, or it isn't correct .. and you won't know if it is or it isn't, unless you investigate further, and/or ask the sysadmin for the device.

 

Having said all that, I am pasting below, and attaching a pattern I wrote to get the Uptime for a NetworkDevice and for an SNMPManagedDevice.

 

FYI: I used this tutorial to see how to find and interpret the sysUpTime from this OID 1.3.6.1.2.1.1.3.0:   https://www.easysnmp.com/tutorial/tutorial-step-2.html

 

Since the Uptime of a Host is found on the HostInfo, my pattern sets the Uptime for a Device on the DeviceInfo.

 

I only added the Uptime in days.  I didn't add an uptime_seconds also exists on the HostInfo.

 

 

Steps to use the attached custom pattern:

1) Upload the attached custom pattern from the Manage>Knowledge page of the Discovery UI

2) Scan your devices

3) You can look for the Uptime using this query:

   search FLAGS(no_segment) NetworkDevice, SNMPManagedDevice show summary, #InferredElement:Inference:Primary:DeviceInfo.uptime as 'Uptime Days'

 

==========================================================================================

tpl 1.13 module device_uptime;

 

metadata

    origin := "Custom";

    tree_path := "Custom", "device uptime";

end metadata;

 

from DiscoveryFunctions import DiscoveryFunctions 1.3;

 

// OID to obtain uptime

table device_uptime_table 1.0

     "1.3.6.1.2.1.1.3.0" -> "device_uptime";

end table;

 

definitions localFunctions 1.0

    '''

    '''

    type := function;

 

    define get_uptime_for_device(device)

        '''

        '''

        uptime_table := discovery.snmpGet(device, device_uptime_table);

        if not uptime_table then

            log.info("Failed to get uptime.  Stopping pattern");

            stop;

        end if;

 

        uptime_ticks := uptime_table.device_uptime;

        log.debug("uptime in ticks = %uptime_ticks%");

 

        //  ((((ticks/60 seconds )/60 minutes )/24 hours )/100 ticks per second)

        uptime_days :=  ((((uptime_ticks /60)/60)/24)/100);

        log.debug("uptime in days = %uptime_days%");

 

        device_infos := search(in device traverse InferredElement:Inference:Primary:DeviceInfo);

 

        if device_infos then

            device_info := device_infos[0];

            device_info.uptime := uptime_days;

            model.addDisplayAttribute(device_info, "uptime");

        end if;

    end define;

end definitions;

 

 

pattern device_uptime_discovery 1.0

'''

add uptime attribute for a NetworkDevice

'''

    overview

        tags Custom;

    end overview;

 

    triggers

        on device := NetworkDevice created, confirmed;

    end triggers;

 

    body

        localFunctions.get_uptime_for_device(device);

    end body;

end pattern;

 

pattern snmpmanageddevice_uptime_discovery 1.0

'''

add uptime attribute for a SNMPManagedDevice

'''

    overview

        tags Custom;

    end overview;

 

    triggers

        on device := SNMPManagedDevice created, confirmed;

    end triggers;

 

    body

        localFunctions.get_uptime_for_device(device);

    end body;

end pattern;

==========================================================================================

 

 

Share This:

Discovery platform scripts are run to obtain certain core information from Unix-like OSes and may be customised for particular environments. The reasons to customise scripts  certainly decreased when patterns were introduced, but there are still some - the main one being the addition of privilege escalation commands (eg "sudo") in the init script to those methods that require it for your environment/requirements.

 

See here for a full list; perhaps the most common might be to run dmidecode on Linux under sudo, in order to ensure the full BIOS ID is obtained, and hence a Linux VM can be linked to its VMware container. See under Administration > Discovery Platforms > Linux, where the init script is highlighted because it has a customisation:

 

 

I have some customers who run with other minor changes to workaround certain edge cases, but those are relatively rare compared to init script changes.

 

Now, as the documentation describes, each new version may make additions to the default scripts to fix bugs or support new features. For example, in 12.1 a new section in the init script was added to support the new Discovery by AWS System Manager feature. For default scripts (no customisations), these will be updated to the new defaults after upgrade. But for any customised scripts, they will not be updated. It is the Discovery Administrator's job to merge the customisations with any changes due to the upgrade.

 

At least that is what is supposed to happen. Unfortunately, we have found that from 11.3 to 12, customisations get lost, and you end up with all defaults for the new version. I have logged defect DRUD1-31023 which I expect will be fixed in the next patch releases, but in the meantime:

  • If you haven't upgraded yet, it's best to manualy save the scripts from the UI before the upgrade
  • If you have already upgraded and not noticed your customisations were lost, then you should be able to find them buried in the file: /usr/tideway/etc/discovery.conf.rpmsave
Share This:

Multi-generational datastore, it sounds very catchy doesn't it?

 

As anyone would expect, with great power comes great responsibility. This is a great new feature introduced in Discovery 20.02 (12.0), where we have the ability to perform tasks like backup and datastore compaction with no downtime.

However it's not a set it and forget it thing!

 

The Discovery Admin (that needs to be at a very good level with Linux and Linux scripting) will need to regularly manage, maintain and protect the datastore. Those tasks need to be performed in the command line (no UI yet) and no out of the box automation/scheduling is provided, so I'll try to cover some basics here on where and how we can achieve this and please keep in mind that this in not intended for a Production system.

 

We're starting from what we state and recommend in our documentation here. Under the section "Recommendations for automation" we layout the steps needed to perform. From Datastore generation creation, to datastore compaction and finally satastore protection. Keep in mind that this will not be deemed as a Discovery "backup". We're simply going to be protecting the read-only generations and the Compacted Generation of the Datastore.

 

We'll use again crontab to schedule the execution of the script that will do.

Some rules to keep in mind before we start:

  • Do not allow more than 7 generations to accumulate without performing a compaction
  • Wait 35 minutes between completing the creation of a new generation and archiving

 

  • New Datastore Generation and Archive/backup
    • Create a new generation, at most once per 1 day. This task is cluster aware so we'll perform it on the Coordinator only.
    • Archive/backup the read-only generations and the compacted generation. This task is not cluster aware so we need to perform it on all the members of the cluster.

 

  • Datastore Compaction and Archive/backup
    • Perform an online compaction of the datastore, once per week. This task is not cluster aware so we need to perform it on all the members of the cluster.
    • Archive/backup the newly created compacted generation. This task is not cluster aware so we need to perform it on all the members of the cluster.

 

The crontab jobs will need to be created on the Coordinator (if this is a cluster) or just a standalone appliance to schedule the execution of the script. We'll place the script on every member of the Cluster and from the Coordinator we will "call" the script execution on the Members through SSH. In my example I'll be running the New Datastore Generation 5 days a week and the Datastore Compaction 2 days a week, keeping in mind the rule of the 7 Generations.

 

  • Crontab for New Datastore Generation and Archive/backup

Create the Discovery cron job file: "/usr/tideway/etc/cron/tw_newgen_backup.cron" and insert the lines below:

# Create a new Datastore Generation and backup the Read-Only and Compacted Generations.

# Run this job every FRI,SAT,SUN,TUE,WED at 11 am

00 11 * * FRI,SAT,SUN,TUE,WED sh $TIDEWAY/scripts/multi-gen-ds/multi-gen-ds.sh -G -U TWUSERNAME -D BACKUPDESTINATION -P PWPATH -F BACKUPPATH

 

  • Crontab for Datastore Compaction and Archive/backup

Create the Discovery cron job file: "/usr/tideway/etc/cron/tw_onlinecompact_backup.cron" and insert the lines below:

# Create a new Compacted Generation and backup the Read-Only and Compacted Generations.

# Run this job every MON and THU at 13 pm

00 13 * * MON,THU sh $TIDEWAY/scripts/multi-gen-ds/multi-gen-ds.sh -C -U TWUSERNAME -D BACKUPDESTINATION -P PWPATH -F BACKUPPATH

 

To add/update the Appliance's crontab with the newly created tw_newgen_backup.cron and tw_onlinecompact_backup.cron jobs we execute the tw_cron_update utility.

 

I really hope you'll find this useful.

Always open to suggestions, feedback or even criticism

 

At the time of posting this, I'm in the process of adding a check that will run a datastore compaction if, for some reason, there are 7 or more generations of the datastore.

 

 

Notes/Requirements:

  • A full tw_backup should be used to protect the Appliance as a whole in regular intervals. The procedure and scripts provided here are examples and will only protect the Datastore and should be used at your own risk.
  • Do not allow more than 7 generations to accumulate without performing a compaction. For example, If you are making a new generation every day, you must compact once per week.
  • The Cluster members need to be able to SSH between them for the remote script execution
  • We'll need a remote Linux server with adequate disk space to hold the datastore backups. Replace references to BACKUPDESTINATION on the cron job above accordingly.
  • A separate enterprise backup solution should/could be used to protect the datastore backups stored on the Linux server
  • We'll need to create SSH RSA keys to allow the script to connect from the Coordinator to the Members
  • We need to create a user for use in the command line. The user needs to be assigned to the Group: [cluster_mgmt]
  • We’ll need to store the password for the compaction command into a file for the command to use without user interaction. The file used on the scripts is $TIDEWAY/scripts/multi-gen-ds/userpass.txt
  • You should consider the difference between the two backup/archive processes described and scheduled in the cron tab jobs. The first one, just after the creation of the new datastore generation, will protect just that and should in theory be small/smaller than the second cron tab job, that runs after the datastore compaction job and will in turn protect the entire (compacted generation) datastore. In sort, see the backup jobs as two differential and one full backup jobs.

 

Notes about the script:

  • We assume the script is located in this directory "$TIDEWAY/scripts/multi-gen-ds/"

 

  • The variables in the command line "-U TWUSERNAME -D BACKUPDESTINATION -P PWPATH -F BACKUPPATH" are:

TWUSERNAME : the username to use

PWPATH : the path where the password of the user exists in a file

BACKUPDESTINATION : the linux server that we'll be storing the datastore files using RSYNC

BACKUPPATH : the path to the root folder where we'll store the datastore

 

  • At the top of the script you'll find (and could update) the variables that are passed through the command line:

BACKUPDESTINATION=linux-backup-server                (in the form : server.domain.local or just an IP)

TWUSERNAME=username-here

PWPATH=$TIDEWAY/scripts/multi-gen-ds/userpass.txt

BACKUPPATH=/usr/tideway/remotebackup                   (note the trailing "/" is not added/used here)

 

  • The usage/options of the script:

      -G: Greate a new Generation and perform backup/rsync of the previous

          read-only and compacted generations.

      -C: Greate a new Compacted Generation and perform backup/rsync of the previous

          read-only and compacted generations.

      -B: Used by the program to invoke a remote backup/rsync of the previous

          read-only and compacted generations.

      -R: Used by the program to invoke a remote Compaction and backup/rsync of the previous

          read-only and compacted generations.

      -v: Print version.

      -h: Print usage (this)

Share This:

BMC Discovery provides a wealth of information out of the box, without having to make any changes. However, in some cases it is necessary to modify the data that is being presented in Discovery or being sent to the CMDB.

 

These modifications are often done with custom patterns. The Pattern Language (TPL) is very powerful and allows for many types of changes to be made to Discovery and its data.

 

Writers of custom patterns may benefit from the following tips and hints:

 

Knowledge Articles (may require a login to a BMC Support account):

 

 

Discovery community blogs:

 

Sample custom patterns on Discovery community forum:

https://communities.bmc.com/community/bmcdn/discovery/pages/patterns

Share This:

Just a reminder, per https://docs.bmc.com/docs/discovery/supported-versions-of-bmc-discovery-531336352.html, Discovery versions 11.1 and 11.2 will go into limited support on September 15, 2020.

 

When a product enters limited support:

  • New enhancements are not made to that version of the product. For reported issues, BMC Software Customer Support directs customers to existing fixes, patches and workarounds.
  • New Technology Knowledge Update (TKU) releases are not available for that version of the product.

 

During the time of limited support, BMC Software strongly encourages users of the respective versions of the product to upgrade to the latest current version.

Nick Smith

Certificate Discovery

Posted by Nick Smith Employee Aug 21, 2020
Share This:

You may have noticed that with TKU 2020-08 we have started to implement a new feature: SSL certificate Discovery:

 

Certificate.png

 

where, for Webservers of types:

"Apache Webserver"

"IBM HTTP Server"

"Oracle HTTP Server"

"HP Apache-based Web Server"

"HP HP-UX Apache-based Web Server"

"Red Hat JBoss Enterprise Web Server"

"Apache HTTPD-based Webserver"

"JBoss Core Services Apache HTTP Server"

"Nginx Webserver"

 

or:

"Apache Tomcat Application Server"

 

we try and connect to connect to TLS port via an openssl command on the target, and thus create a Detail node of type "TLS Certificate". Except, rather embarassingly, for the TKU on 12.0 where we misspell "TLS" as "TSL". Oh dear. I logged defect DRDC1-15692 to address that. The TKU on 11.3 doesn't have this problem.

 

It should also be noted, although not yet in the official docs (logged DRDC1-15728), that this is applicable only for instances running on a Unix-like OS.

 

Query to find near expiry dates

 

You may use a query such as this to show cetificates whose end date is less than 30 days away:

 

search Detail where type matches 'Certificate' and expiry_date < (currentTime() + 30*24*3600*10000000) show start_date, expiry_date, common_name, #Detail:Detail:ElementWithDetail:SoftwareInstance.name as 'Software Instance Name', #Detail:Detail:ElementWithDetail:SoftwareInstance.#RunningSoftware:HostedSoftware:Host:Host.name as 'Host Name'

 

Note that I have used the "type matches 'Certificate'" instead of the more efficient "type = 'TLS Certificate'" until the defect mentioned above is corrected.

 

Assessing Coverage

 

When depending on this information, it is important to know the prerequisites and coverage you are getting.

 

For Apache HTTPDs, we need to have created a "Website" SoftwareComponent off the main SoftwareInstance, and recorded encrypted sockets. This query lists those that have that, but no certificates:

 

search SoftwareInstance where type in ["Apache Webserver", "IBM HTTP Server", "Oracle HTTP Server", "HP Apache-based Web Server", "HP HP-UX Apache-based Web Server", "Red Hat JBoss Enterprise Web Server", "Apache HTTPD-based Webserver", "JBoss Core Services Apache HTTP Server", "Nginx Webserver"] and nodecount(traverse SoftwareContainer:SoftwareContainment:ContainedSoftware:SoftwareComponent where type matches "Website" and listen_ssl_tcp_sockets) and not nodecount(traverse ElementWithDetail:Detail:Detail:Detail where type matches 'Certificate')

 

This set would be good candidates for investigation, perhaps for failing openssl commands.

 

Now, to find Apache HTTPDs where we don't have any Website SoftwareComponents, use:

 

search SoftwareInstance where type in ["Apache Webserver", "IBM HTTP Server", "Oracle HTTP Server", "HP Apache-based Web Server", "HP HP-UX Apache-based Web Server", "Red Hat JBoss Enterprise Web Server", "Apache HTTPD-based Webserver", "JBoss Core Services Apache HTTP Server", "Nginx Webserver"] and not nodecount(traverse SoftwareContainer:SoftwareContainment:ContainedSoftware:SoftwareComponent where type matches "Website")

 

I found my own Fedora webserver is not getting detected; I will report back when I know more.

 

For Apache Tomcat, we need to have recorded some secure sockets. So, we can find all Tomcat instances where we do have these, but no certificates:

 

search SoftwareInstance where type = 'Apache Tomcat Application Server' and listen_ssl_tcp_sockets and not nodecount(traverse ElementWithDetail:Detail:Detail:Detail where type matches 'Certificate')

 

This set would be good candidates for investigation, perhaps for failing openssl commands.

 

This should be only the start of the Certificate Discovery work. While I can't guarentee anything, work is underway to look at other products' certificates, and to map these Detail nodes into the CMDB using the BMC_Document class.

 

But in the meantime, I encourage you to see what coverage you are getting and provide us feedback on what we can improve.

Share This:

In the IT world we regularly try to automate tasks that take up our time. We create procedures that have an input and produce an output.

One of my customer’s had this requirement where they needed to scan multiple subnets (with lots of dark space) and having ping (ICMP) opened through the firewalls was a challenge.

The idea to automate their scanning process became apparent, so I’ve created this custom pattern that will take as an input (trigger upon) the results of a Sweep Scan (that was set not to ping before scan) and it will produce as an output the creation of a Full Discovery in the form of an Open Discovery scan.

 

First step, we trigger on a DiscoveryRun. This seems to be, after asking around, the first time someone has attempted something like that.

 

    constants
        scantype := "Sweep Scan";
    end constants;
    triggers
        // Trigger when DiscoveryRun has ended.
        on scan:= DiscoveryRun modified where scan_level = scantype and endtime exists and discovery_endtime exists;
    end triggers;

 

First thing in the body, we search for all the DiscoveryAccess’s, in that DiscoveryRun, that have resulted with “Success”

 

    endpoints := search(in scan
                                          traverse List:List:Member:DiscoveryAccess where result= "Success"
                                          show endpoint);

 

Following that, we need to make sure that we’re not going to be adding the same endpoints in the same Open Discovery Scan, in case we’re scanning multiple times the same endpoints.

Here, initially we were checking for the DiscoveryAccess’s, but that proven to be inaccurate as the DiscoveryRun could  have been updated at the same time by different ECA engines and thus causing the pattern to trigger multiple time.

Also, on a busy system, the DiscoveryAccess’s could take some time to be created and that could lead to double entries.

So I came up with this solution of using the “valid_ranges” from the DiscoveryRun, but that needed some fixing-up.

 

    if endpoints then
        //verify that the endpoints are not already part/scanned in an active Open Scan
        currentopenrun := search(DiscoveryRun where scan_type = 'Open' and not endtime
                                                     show valid_ranges);
       //cleanup and spit the result to generate a list of endpoints
        currentopenrun := text.strip("%currentopenrun%", "[]'");
        currentopenrun := text.split("%currentopenrun%", ", ");

 

Now we’re getting into the sorting process. Iterating through the endpoints of the Sweep Scan that was just finished and the endpoints that exist in the currently active one (if one exists). This will produce the list of endpoints that we’ll add to an Open Scan later.

 

        endpointlist := [];
        for endpoint in endpoints do
            addendpoint := 'True';
            for currentendpoint in currentopenrun do
                if endpoint = currentendpoint then
                    addendpoint := 'False';
                        break;
                end if;
            end for;
            if addendpoint = 'True' then
                list.append(endpointlist,endpoint);                                        
            end if;                                         
    end for;

 

 

And the final step, initiate the open scan by using the discovery.scan() function.

    endpointsnum := discovery.scan(endpointlist);

 

The result in “endpointsnum” will be the count of the endpoints that were added in the open scan.

 

 

Optional enhancement:

Now if you would like to enhance this pattern, if there is a need to execute the scan on a specific Outpost, you could by forming the command like this:

 

    endpointsnum:= discovery.scan(endpointlist, "BMC", "outpost01");

(BMC is the name of the company and outpost01 is the name of the Outpost)

 

 

From this point the customer simply needed to create Sweep Scan schedules and make sure some prerequisites exist before using the pattern and invoking the discovery.scan() function.

Prerequisites:

  • A scheduled scan defined that includes the address or addresses of the scan targets
  • The Allow patterns to scan in this range check box is checked for that scan
  • The scheduled scan does not need to be in its specified scan time window
  • The scheduled scan needs to be activated

 

Hopefully this is helpful for some.

 

Appendix:

Open discovery run

An open discovery run is a one shot extendable list of endpoints to scan. When an open discovery run is created the scan starts immediately. Unlike snapshot and scheduled discovery runs, an open scan can be extended with additional endpoints.

It is possible to add any number of endpoints to an open discovery run until midnight (appliance local time) on the same day the open discovery run was created.

Adding additional endpoints after midnight creates a new open discovery run. An open discovery run is completed when all the endpoints it contains have been scanned and the time is after midnight. Open discovery runs remove the need for Discovery to have very many small snapshot scans. You can create open discovery runs using the TPL function discovery.scan.

Nick Smith

A Database in every port

Posted by Nick Smith Employee Aug 11, 2020
Share This:

A customer of mine is interested in discovering not only Software Instances that represent running database software, but also the actual Databases within. You wil often see a core pattern that handles the SI, and then an extended one that handles the deeper Database discovery. For example:

  • Microsoft.SQLServer.SQLServer creates the SI nodes
  • Microsoft.SQLServer_Extended.DatabasesAndTables creates the Database nodes

 

For maximum information, credentials are required to login to the database directly over JDBC. However, for SQL Server, the list of Databases can usually be obtained via WMI:

 

 

Now, my customer was interested in Sybase a while back. There is an option in the Sybase.SybaseASE_Extended module "Use log files for deep discovery if direct query of the database fails" which used to be True by default:

 

 

This interrogates the database software logs for entries that reference database names as they are created. This is good, as again you don't need database credentials for this to work. Unfortunately, we found that Sybase doesn't log when a database is removed, so we were accumulating a list of Databases that existed at some point, but that we couldn't guarentee actually existed at any given scan. Hence, we changed the option to default to False; this was way back in TKU 2016-09.

 

Thus we had to live with no Sybase Database discovery until recently, when the customer wanted to test Sybase credentials. The DBA added some for a test server, and they got... nothing. We were able to get Databases created by providing a default port in the credential, like this:

 

 

but in general we shouldn't have had to do this. The Sybase pattern should have populated fields on the SI:

  • bind_address
  • port

that can then be used by the Extended Sybase pattern. This information is obtained from the interfaces file, such as "/opt/sybase/sydb01/interfaces". In order to get this file path, we need to know the installation root path, "install_root" which should be visible on the process command line. In turn, this was missing because the OS was Solaris 10, and we were suffering from truncated command lines. This was solved by adding "sudo" to the PRIV_PS command in the platform init script, in order to gain elevated permissions. This problem is actually highlighted in the UI with a Discovery Condition:

 

https://bmcapps--c.na117.content.force.com/servlet/servlet.FileDownload?file=00P3n00001ZLnqkEAD

So with this change were we getting very close. Some databases appeared, and versioning started to appear on the Software Instances. Now, it is worth noting that in general (not just Sybase) different databases can be configured to listen on different IPs. And this was the case here, so while we scanned on 192.168.50.66, we could not see some databases on that server that only listened on another IP. We allowed Discovery to contact a different IP than scanned (Discovery Configuration):

 

 

and ensured firewall rules to the server were open for all IPs. On rescan, we could observe under the Integrations section of the 192.168.50.66 Discovery Access, that SQL Results were returned for both that IP and 192.168.50.68:

and, all Sybase Databases were created, fully versioned, on all Sybase Software Instances. Phew - time for a sit down and a nice cup of tea.

 

(IP addresses and database names have been changed to protect the innocent)

Share This:

WARNING:  Do not upload the August 2020 TKU to 12.0.0.0 or 12.0.0.1.

Before applying the August 2020 TKU, you must upgrade Discovery to 12.0.0.2.

 

Documentation is here: Technology Knowledge Update TKU 2020-Aug-1 - Configipedia - BMC Documentation

 

"

Important Note for BMC Discovery

 

If you use BMC Discovery 12.0 (20.02), you must install Patch 2 before you install the August TKU. More information on the patch is available here, Patch 2 for version 20.02.

"

 

============================================================================================

 

If you do upload the August 2020 TKU onto 12.0.0.0 or 12.0.0.1, you will get this errors such as these upon the TKU Upload:

"

ErrorChange to module CMDB.OwnedManagementGroup_AdminDomain failed because at line 12 imports pattern module 'CMDB.CloudManagementGroup_AdminDomain' which has Unknown node or relationship kind 'CloudManagementGroup' in mapping traversal at line 25.

 

ErrorChange to module MicrosoftAzure.Storage.ManagedDisks failed because at line 13 imports pattern module 'Cloud.CommonFunctions' which has Unknown node kind 'CloudManagementGroup' in search at line 174.

"

 

Even with the errors, the "product content" in the Manage > Knowledge page will show 08/2020.

 

The CloudManagementGroup referred to in the error message is a new node kind with the August 2020 TKU.

 

To recover from this error, you must upgrade to Discovery version 12.0.02 and then re-apply the August 2020 TKU.

 

See also:  KA 000371956  -

Discovery: Upload of August 2020 core TKU fails with "Unknown node or relationship kind 'CloudManagementGroup'"

 

Share This:

KA 000349819 describes the process for running a device capture and requesting integration support for an unsupported SNMP-enabled device, such as a Network Device or Printer.

 

The device capture produces a zip that contains several files. The most important of these files are:

  • capture.log - Look here for the sysobjectid and sysdescr values, and check for any error messages
  • <ip_address>@public.MVC file - This is the actual formatted device dump.

 

However, problems occasionally occur when running a device capture. What are the most common problems, and how to troubleshoot them?

 

 

1. A device capture fails with "ERROR: SNMP++: SNMP request timeout" or "Device skipped - no SNMP access".

 

Here’s an example from a device capture log:

 

Note that there are two ways to get a device capture (see https://docs.bmc.com/docs/discovery/113/capturing-snmp-devices-788111406.html):

  • from the Discovery Access page
  • from the Device Info page (which is reached from the Discovery Access page)

 

In some cases, doing a capture from a Device Info node may result in a blank screen. When clicking the browser "back" button, it returns to the Device Info page and briefly shows a green banner that says "Device skipped (no SNMP access)", which then fades out after a few seconds:

 

There are many possible reasons for the "timeout / no access" result. Here are some possible causes and things to check:

  • Make sure a SNMP credential is present and that the IP range includes the address being discovered (a valid credential is needed for device capture)
  • Run a credential test. What is the result?
  • Increase the timeout in the SNMP credential to 100 seconds and retry.
  • What is the SNMP version (v1, v2c, v3) on the credential? Is the device configured to respond on that SNMP version?
  • If the device supports SNMP versions other than the one specified in the credential, change the version in the credential and retry.
  • For SNMP v1 and v2c, make sure the community string in the SNMP credential is correct. An invalid community string can cause a "Unable to get the deviceInfo: TRANSIENT" error.
  • Ask the device administrator:
    • if there might be an Access Control List (ACL) or some other configuration on the device that prevents responses to the Discovery appliance.
    • if the device is configured to use the default SNMP port (161). Run nmap to confirm the port is open (see below).
  • On the Discovery Access page, click on any links for "script failure" or "xx Session Results related" to look for clues.
  • In the case of a timeout during a device capture, check the log in /usr/tideway/var/captures for additional clues
  • In the case of a timeout during a scan, turn DEBUG logging on for Discovery, run the scan again, and check the tw_svc_discovery.log for clues. Remember to turn DEBUG logging off!
  • In one case, the customer made corrections to the IP range and mask on the device, then was able to discover the device.

 

  • From the Discovery command line, as user tideway, run the following commands and check the results:

    a) Check connectivity from the appliance to the endpoint:

                             ping [device_ip_address]

                             traceroute [device_ip_address]

 

     b) Check the port status of the device by running nmap. For example:

 

    /usr/bin/nmap --privileged -sT -sU -p T:22,U:161 [device_ip_address]

 

The expected result is that port 161 would have a state of "open" or "open|filtered" :

        

      c) Do a snmpwalk to the device. For example:

 

                         /usr/tideway/snmp++/bin/snmpWalk [device_ip_address] -v2c -cpublic > /usr/tideway/snmpwalk.out

 

Change the SNMP version and community string as needed. If using SNMP v3, other parameters need to be specified. To see the usage notes with a list of available options, run snmpwalk with the "--help" option.

 

If snmpwalk also fails, please consult the device administrator.

 

If the problem persists, please contact Customer Support and provide the results to all the questions / checks above.

 

 

2. A device capture fails with "ERROR: sysObjectID is not an enterprise MIB" :

 

sysDescr: Cisco Secure Access Control System 5.4

sysObjectID: 1.3.6.1.2.1.47.1.1.1.1.13.1

ERROR: sysObjectID is not an enterprise MIB: 1.3.6.1.2.1.47.1.1.1.1.13.1

 

Discovery expects every sysObjectId to begin with 1.3.6.1.4.1 as per the SNMP MIB standard. Each vendor is allocated a subtree under 1.3.6.1.4.1.

 

Because the sysObjectID does not comply with the SNMP MIB RFC published here:

 

https://www.rfc-editor.org/rfc/rfc3418.txt

 

it is currently not possible to discover this device.

 

3. A device capture fails with "ERROR: Loop detected" :

 

Dumping range: Start of MIB to End of MIB

ERROR: Loop detected: current OID is 1.3.6.1.2.1.17.4.3.1.1.0.0.170.199.13.135, WAS 1.3.6.1.2.1.17.4.3.1.1.44.65.56.156.227.187 previous OID, OID ILD is 1.3.6.1.2.1.17.4.3.1.1.44.65.56.156.227.187

 

An snmpwalk shows something like this:

 

SNMP++ snmpWalk to <ip address> SNMPV2 Retries=1 Timeout=1000ms Community=<community>

[...]

1.3.6.1.2.1.1.9.1.4.1 = 0:00:00.00

1.3.6.1.2.1.2.1.0 = 31

1.3.6.1.2.1.2.2.1.1.1 = 1

1.3.6.1.2.1.2.2.1.1.1 = 1

1.3.6.1.2.1.2.2.1.1.1 = 1

[...]

 

The "next oid" of 1.3.6.1.2.1.2.2.1.1.1 is itself (not normal, this indicates a loop).

 

In some cases, it is possible to work around this. Please contact Customer Support and provide the device capture file.

 

4. A device capture for a router device takes hours to complete.

 

If a router device has a huge routing table, device capture may take a long time to complete. In a case where the capture is not completed even after several hours, it can be cancelled and the partial capture can be shared with BMC Support for device integration.

 

In some cases, the partial device capture contains sufficient information for device integration.

 

5. A device capture fails with ERROR: SNMP: Variable does not exist

 

This typically indicates a problem with the device or the SNMP agent running on it. To confirm, run an snmpwalk on the device (see above). If the snmpwalk also fails (or does not produce usable output), please contact the device vendor.

 

6. A device capture fails with ERROR: SNMP: Cannot perform operation, General Error

 

This typically indicates a problem with the device or the SNMP agent running on it. To confirm, run an snmpwalk on the device (see above). If the snmpwalk also fails (or does not produce usable output), please contact the device vendor.

 

In some cases, the device capture in this case will produce a partial capture file that is sufficient to integrate this device. If so, please contact Customer Support and provide the device capture file and snmpwalk output.

Share This:
With the May 2 2020 TKU, there are some "Special Instructions" to perform after applying the update.  The instructions are documented here:
https://docs.bmc.com/docs/display/Configipedia/Technology%20Knowledge%20Update%20TKU%202020-May-2

 

 

Q&A:

Q1)  If a customer skips the May TKU entirely, is it still necessary to perform the special instructions?

 

A1)  Yes. If, for example, a customer updates from April 2020 TKU to July 2020 TKU, the customer needs to perform the special instructions after applying the July 2020 TKU.

 

Q2)  Does the customer need to continue using the Special Instructions from now on after each TKU Update?

A2)  No. These instructions are one-time-only.

 

Q3) The instructions are confusing.  Is there something easier?

A3) Yes.  After updating to the May 2020 TKU, simply reboot the appliance / cluster from the UI:  Administration->Control->'Reboot Appliance' or 'Reboot Cluster'

 

Q4) Why do we have the special instructions?

A4) This TKU 2020-TKU-May-2 re-release includes an important upgrade for AWS Schedule Scans and Microsoft SQL Server containment relationships

       To correct the existing data, the special instructions are necessary.

 

Q5) Why is the May 2020 TKU dated May 2 instead of May 1?

A5) Sometimes a TKU is re-released after the initial release.  The initial TKU, May 1 2020, had some problems. 

       The problems were corrected, and the May TKU was re-released as May 2 2020 TKU.

       On the EPD site, the older problematic TKU is removed when the TKU is re-released.

      

 

Best Practice:

Customers who download a TKU in advance of Uploading the TKU should check back on the EPD site to see if that TKU has been re-released.

 

============================================================================================================

SPECIAL INSTRUCTIONS AFTER APPLYING MAY 2 2020 TKU (or after skipping over May and applying a later TKU):

 

Once the TKU has been applied the upgrade must be applied. There are two ways to do this.

1. Via the appliance command line:

1.1. Log into the appliance command line as the “tideway” user. For clusters, these steps must be done on each cluster member.

1.2. Restart the “cluster” service:

  • For 11.2 and CentOS/RedHat 6 based 11.3 systems: sudo service cluster restart.
  • For CentOS/RedHat 7 based 11.3 systems: sudo systemctl restart cluster.

1.3. Perform a conditional service start: tw_service_control --start --conditional.  You should see a message “Performing model upgrades”.
        Note: this step only needs to be done on the cluster coordinator machine.

--OR--

2. Reboot the appliance (for a cluster:  reboot all members of the cluster). Note that simply restarting system services will NOT apply the update.

=============================================================================================================

 

NOTE:  This blog posting refers only to the on-premise Discovery product.

             Customers on Helix Discovery do not update their TKU's because that is done automatically each month.

 

 

Nick Smith

Where is your boundary?

Posted by Nick Smith Employee Jul 17, 2020
Share This:

In a previous post I asserted that the security boundary of the Outpost should be considered to be the OS. This is given additional weight if you consider the UI authentication mechanism: at install time, a local administrator password is chosen:

 

Outpost Login.png

 

(Aside: the Outpost's password needs to be at least 8 characters, but there doesn't seem to be any configurable options, like there are for appliance users; a 10-level password history is kept)

 

This account information is stored locally to the Outpost - it is not linked to the local OS, Active Directory or an appliance. If you look in the registry under the key:

  • HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\Discovery Outpost

 

Outpost Registry.png

 

you will find settings OutpostAdminSalt and OutpostAdminPassword. The password is stored with a modern digest algorithm with a random salt, so it should be resistant to rainbow table attacks to recover the password.

 

However, any Windows user with Administrator rights can simply replace these values with ones that correspond to a known password (normal Users can only read). This is a useful way to proceed if you have forgotten the Outpost password. Which you should never do, of course, since you will be using a good password manager. Right...?

 

You could obtain the required values by either:

 

1)

Installing another Outpost somewhere, entering your desired password, and grabbing the values from its registry (it doesn't need to be connected to an appliance)

 

2)

Using the values here:

  • OutpostAdminSalt:
    080D4CB1AFD201062FFC6B67141C077DAB479C04D0F5CA535F27A4BB890E8215580448CF3F311610EDCAA9226945756F9BD177F90997B94E2B24A4BA7D804864D1329C5BE63ADE563C98869F457C5F9CBD8341C567DA249D10BC64CBE3233A5906B24EC9281751844855D079790303764844FB40A87BE88AAC53B189F64F9AAB4A95352DDBBB2B856DE9E63AC44F2853F0850F65785C641CAB8EAEACD46803C2CDFA35CEA37791EF5B3EEE1CAE5CDC1E932AD71C123E5B525CBFC0D75A54AE69CA9750B736F1F16F19B2302BA4ACAF567CD8F8ED5CB2816D4624BC87F81ACA43A7EE53F48E1C44B52DF03FF2EF30DA8D7C206489BC9586F3BCE5FE5B54FD1332
  • OutpostAdminPassword:
    48E1984F27E9C4F3AF9F24C04209070D875FB6C8EC5A009762ABA23566F9B16C0B6712FD7F1D9294ED114A00F7FED6A4175AF75A356EF1065AD4A204CE7A3CD2

 

After restarting of the Outpost service, the password is set to "secretpass" (see - I can do irony). You should then login and immediately change the password in the Outpost UI to something of your choosing:

 

Change Passowrd.png

 

In addition to the password hash, the salt is also updated when you change the password in the UI.

 

So, is this a security problem? Well, only if you take misguided actions based on an incomplete understanding of how the system works. Again, it's about where the boundary is. Perhaps it would be nice to be able to have more flexibiliy on how to authenticate to the local OS or external systems, but it's really no different from:

  • Unux/Linux root being able to copy entries in /etc/shadow file
  • Tomcat user passwords being configured in the tomcat-users.xml file
  • OpenLDAP storing the Manager DN password hash in olcRootPW field in an .ldif file
  • The tideway CLI user controlling Discovery UI users with the tw_passwd command.

 

Where if you have control of the OS, you control the applications that run on it.

 

Let me know your thoughts on how this architecture fits into your security policies and controls, and if you agree or disagree with my thinking.

Share This:

The Discovery Outpost is one of the major new features of the Discovery 12 (20.02) release that provides a lightweight Windows-based component that can replace proxies and consolidation scanner appliances. It has its own HTTPS-based UI, and after installation generates its own self-signed certificates - hence your browser will have to be told to trust it, as there is no chain of trust to a known CA. The certificate will look something like this in your browser:

Def Cert.png

The auto-generated key and certificate files are stored in C:\Program Files\BMC Software\Discovery Outpost\etc\https (unless you have changed the install directory from the default) as

  • server.key
  • server.crt

You will find it has a 4k RSA public key, with 10 year lifetime:

cert1.png

It is standard practice in an organisation, at least for production systems, to mandate the use of certificates that are signed by a known and trusted in-house CA. Perhaps you want to reduce the certificate lifetime; this is certainly the trend for Internet-facing certificates.

 

So, is there an easy way to generate your own Outpost keys and certificates, like you can for the appliance? Alas no. The documentation mentions how you can, but the files have to be copied manually (and bounce the Outpost service). You might notice that the server.key file is encrypted (PKCS#8 standard):

so the obvious question is with what password is this encrypted? The answer is that it is the Outposts ID, that can be found in the Discovery Outpost\etc\machine.uuid file, or the tw_svc_outpost.log:

I don't know if there is a native way on Windows to process these PEM-format files, so I use openssl commands on Linux:

Here you can see it asks for the password, to which I entered the Outpost ID. You don't *have* to encrypt the key - the service will read a non-encrypted key, that looks like:

Moreover, I maintain that encrypting the key here is futile and can lead to a false sense of security. By design, the Outpost has to know how to decrypt at startup and as we have seen, that information is held in a file. It would only make sense if user interaction was required, like during Apache Websever startup or a Discovery passphrase-protected-vault. I recommend you consider the Windows OS as the security boundary to the HTTPS key.

 

I expect you woudl request your CA to generate a key/certificate pair; they will need to know the X.509 CN (Common Name) - which will be the DNS name the browser would use to connect to the Outpost. If for some reason you want to generate your own key and CSR to send to your CA you can, using any openssl installation.

 

This is what I used to generate the (4 k bit) key:

 

[nicsmith@npgs-fedora int2]$ openssl genrsa -out private/clm-tlv-uch418.bmc.com.pem 4096

Generating RSA private key, 4096 bit long modulus (2 primes)

......++++

....................................................................................................++++

e is 65537 (0x010001)

[nicsmith@npgs-fedora int2]$ chmod 400 private/clm-tlv-uch418.bmc.com.pem

 

where the clm-tlv-uch418.bmc.com.pem file gets renamed to server.key at the Outpost.

 

This is what I used to generate a CSR:

 

[nicsmith@npgs-fedora int2]$ openssl req -config openssl.conf -key private/clm-tlv-uch418.bmc.com.pem -new -sha256 -out csr/clm-tlv-uch418.bmc.com.csr.pem

Enter pass phrase for private/clm-tlv-uch418.bmc.com.pem:

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [GB]:

State or Province Name [England]:

Locality Name [London]:

Organization Name [BMC Software Ltd]:

Organizational Unit Name [Discovery Support Certification Authority]:

Common Name []:clm-tlv-uch418.bmc.com

Email Address [nick_smith@bmc.com]:

[nicsmith@npgs-fedora int2]$

 

I have my own Root and Intermediate CAs setup, with their certificates trusted by my browser, so I could sign this generated CSR and generate my own server.pem file.

 

Finally, then, you will have a nice HTTPS connection, using a certificate to a browser-trusted CA. This is how it looks for me, in Firefox:

 

If something goes wrong, and SSL library won't initialise, you'll probably get something like this dumped into the outpost.out file:

This happened to me when I initially tried to use an encrypted key with the "-aes256" option to the "openssl genrsa" command. That resulted in this format:

which the Outpost didn't like. If you really feel you want to have an encrypted key, despite my earlier opinions, an acceptable PKCS#8 encrypted file can be generated by:

  • first generate an unencrypted key file
  • then encrypt like this:  openssl pkcs8 -topk8 -in server-unencrypted.key -out server-encrypted.key

 

See here for an old post with some other useful openssl commands.

Share This:

Back in the day, we used to be quite happy to run out protocols over the wire unencrypted. Checking email using elm over telnet or rlogin,  sharing files over NFS v3 and user authentication using NIS. Things move on, however, and now it's no longer acceptable to move data around without protection on the wire. Over the wider Internet it's absolutely critical, but even within an organisation's network it is rapidly becoming mandatory too.

 

So, this brings us to the topic of the connection between Discovery and Atrium CMDB. While the data should not contain information such as passwords or credit card details, information about systems configuration, software and patches (or lack thereof), IPs and so on would certainly be a valuable resource for someone with malicious intent. Thus, many organisations would want to secure this connection.

 

Historically, the legacy CMDB/Remedy API has been unencrypted by default. An extra-cost option has been available: the Premium Security add-on, but this has not been officially validated/supported from the Discovery end. Thankfully, when the REST API option for CMDB sync was introduced, we have a no-extra-cost encrypted option using industry-standard TLS. The REST option is now the default for new connections:

 

 

It should be noted that you don't *have* to use encryption: an "http" connection could be specified too. The REST API method is recommended for other reasons; we have found it more reliable than the legacy API, and I have seen cases where error reporting is more helpful too.

 

If you have had a Discovery/CMDB connection for a long time, and upgraded, chances are you still have the legacy API (it won't get changed during an upgrade automatically). It might be worth updating this - especially during the next change window you might have for another patch/upgrade.

 

Now, I realised there is a gap on the HTTPS connection configuration. While the connection will certainly be encrypted, there is no facility to confirm the authenticity of the server you connect to by checking the presented certificate against a trusted CA. Compare to the LDAPS UI configuration where you can upload a CA:

 

 

Some customers may be OK with this, on the assumption that the risk to MITM attack within an enterprise are of a low risk. But we should provide this facility, so I have logged defect DRUD1-30177. Let me know how important you think this is.

 

Regarding Helix Discovery (DaaS), the docs do not describe the option to push data to an on-prem CMDB, but it is an option if you can expose your CMDB's REST API port to a public address that Helix Discovery can connect to. I have logged a docs bug DRUD1-30294 to describe this possibility.

 

While simple in principle, there would be many network and security considerations to be understood. Connections should probably be limited to our egress points. Special firewall/load balancer configurations may be needed. Extra intrusion detection/monitoring might need to be applied. It would seem to me that the inability to check certificates becomes more important in this case. Moreover, it would be better to offer even stricter authentication of the client (Discovery), such as presenting a client certificate that needs to be validated. To this end, there is an improvement DRUD1-27874 under consideration.

Share This:

I wonder how many of you know what these are, at the bottom of (say) a Host node:

or like this, on a SoftwareInstance node:

Well, NIST maintains a National Vulnerability Database that provides a common classification of software in the form of those CPR Strings (Common Platform Enummeration is a structured naming scheme in URI format) and CPE ID.

 

If you have the Extended Data Pack installed (used to be a separately license entity), you will have the National Vulnerability Database patterns that generate this data. We currently conform to the still-supported CPE version 2.2, although we are looking to update in a future TKU as part of DRDC1-12249.

 

For my Postgres SI example, you get an link:

which takes you to this list at NIST.

 

I would be interested to know if anyone is actively using this data, and how. Are you feeding other security analysis tools?

Filter Blog

By date:
By tag: