Hello and thank you for reading. First let me apologize for the highly granular technical details in this discussion.
This information has been provided to you by the official BMC Software AtriumCore Support organization in cooperation with the AtriumCore Customer Engineering team in Austin, TX. and San Jose, CA. We guarantee its accuracy and apologize for any typos that were not identified during publishing. Other publications on this topic are to be considered inferior unless otherwise specified. By inferior we mean that this article has the highest precedence weight in value and supersedes any previous discussions on this topic.
Additional references are available and recommended for review:
The following is an explanation of the CMDB Common Data Model design and its reaction to being introduced to overlays. Let’s begin by making the following general statement:
“Forms owned by the BMC AtriumCore CMDB application should never be overlaid, customized or otherwise altered outside of the CMDB Class Manager. AR Developer Studio does not have any role in managing CMDB data structures and its use to modify and overlay CMDB schemas can have and does have severe impact on functionality of the CMDB and can lead to data loss.“
Making the right and informed decisions while making modification to CMDB data structures can make all the difference to prevent long term impact on performance and even AR Server stability. I’ll begin by explaining the CMDB design a little.
CDMB Application - BMC.CORE
CMDB Forms are created in the same fashion as inner joins where each joined form has a common field used to find specific results. Querying these tables as a "joined view" will return only the records where both tables match exactly the same value in one column of the row (field). It's like pinpointing a GPS location on the map by its coordinates. In respect to BMC's CMDB these inner joins are then using the exact InstanceId value to find a unique match. To clarify this point the T(#) tables are indeed "physical" tables in whatever RDBMS you have deployed on. I double quoted the physical, because technically it's just electromagnetic charge where no one is serving dinner on that table. By physical I mean a difference between a database table and a joined view, where the view is a window that allows us to present the data in a controlled way. ARSCHEMA table has records that include schema Id's that make up the AR metadata and "BMC.CORE:BMC_BaseElement", "BMC.CORE:BMC_ComputerSystem_" and "BMC.CORE:BMC_ComputerSystem" are listed there as tables or joins, but they are different DB objects ( views) created to be used as part of the CMDB data present in a ARSYSTEM database.
For example consider this "joining" of
T503 (BMC_CORE_BMC_BaseElement) and T522 (BMC_CORE_BMC_ComputerSystem_)
gives the result of the intersection of both tables where InstanceId (179) matches a record with the same value in both tables . This then gives the results as defined by the View with values end users understand as Configuration Item (CI). Here views show the joining of two or more tables that have common data and are used for data presentation for end users. With this design we can manage all aspects of data with extreme precision.
NOTE: Please note that the schemaids used here, id number 503 and 522, are not static id's from one database to another and can change. They are not exclusively identified as BaseElement or ComputerSystem table ids. Deployments in your environment would likely have different ID. Conversely all attributes will have the same Column ID from one database to another. For example Status will always be C7. There are some exceptions to this for column IDs and attribute names, but in general this will apply to most attributes.
I am going to add some SQL statements here to illustrate it. Running two queries to find a CI record with the same InstanceId in two different tables may seem to be impossible because we understand a BMC.CORE:<CLASS> schema to have a unique value for field InstanceId(179). It has a unique index on it and finding only one record should not be possible. That is true, but not if you separate the inner join into two tables.
Consider the following query that looks in each table:
select * from T522 where C179 = 'OI-306843f7d118455882a5847bf9200c8f'
select * from T503 where C179 = 'OI-306843f7d118455882a5847bf9200c8f'
Where in my example table T522 is BMC_CORE_BMC_ComputerSystem_ and T503 is the BMC_CORE_BaseElement. Note here that both are actual tables. We are more familiar with name BMC.CORE:BMC_ComputerSystem which is a view of the two tables where the joined data is displayed.
select * from T503 INNER JOIN T522 on T503.C179 = T522.C179 and T503.C179 = 'OI-306843f7d118455882a5847bf9200c8f'
This is essentially the same thing as running:
select * from BMC_CORE_BMC_ComputerSystem where InstanceId = 'OI-306843f7d118455882a5847bf9200c8f'
Here is an illustration showing these results in MSSQL Studio:
Note that the last result where the RequestId (Field C1) results in a composite record from both tables.
This is just to illustrate how most CMDB data is stored and looked up.
If either table is altered outside the inner join (view) then the view will show the result as a whole record. We do not recommend doing this in practice, however some data manipulation can be performed with a certified CDMB Data Administrator. I can give two reasons for not doing it for you to consider. For one, making changes at the database level bypasses workflow. Secondly, altering and specifically deleting data through SQL query will impact only one table and can make ‘half’ of the record disappear. Cases where BMC_BaseElement part of joined record exists while BMC_ComputerSystem half is not found have been reported and are direct result of manipulation of data through the SQL back end.
For example if you run this SQL:
"delete from BMC_ComputerSystem_ where MarkAsDeleted = 1"
then you're only clearing out the BaseElement side while leaving the ComputerSystem_ table still full of that data. If you think you've done something like that in the past then you can check it by running this type of SQL query:
select count(*), ClassId from BMC_CORE_BMC_BASEELEMENT where classid = 'BMC_COMPUTERSYSTEM' and instanceid not in (select instanceid from BMC_CORE_bmc_COMPUTERSYSTEM) group by ClassId
Any results from the above query mean that there are records in BMC_BaseElement table that are not found in BMC.CORE:BMC_ComputerSystem class container. You can substitute the COMPUTERSYSTEM ClassId in the above query with other class id values like BMC_IPENDPOINT and so on.
Please refrain from such practice. Use CMDBDiag, and Reconciliation Delete or Purge activity instead. These tools can help you delete data without violating the data consistency.
Next layer is the AR Metadata with set of tables that have references to schema ID of the actual table structures.
- Table structures use prefixes like T, H and B
- AR Metadata is comprised of the various tables, but for the CMDB the ones that matter the most are:
- arschema - references tables by ID (schemaid)
- field - references columns in tables by fieldid and associations with schemaid.
There are other tables like “escalation” - references all escalations, “filter” - references all filters and others. These are not relevant for this topic, so I will not list them all here.
ARSCHEMA is usually the naming convention for "out of the box" name or ID of the database, however this name can be different if AR Server was installed into a database that was created before the installer was started. For the context of this posting just note that the name of this database can have different names but the "arschema" table itself can only have "arschema" as name. It is a table inside the ARSCHEMA database. So, when I mention the "arschema" in the following paragraphs I mean the "arschema table" rather than the database.
Records in "arschema" table are mostly pointers to tables that will use schemaid to look up the data. For example SchemaId 456 is referring to a table T456. Unfortunately these structures have no restrictions set on them and can be modified via BMC Developer Studio and become Overlays with "Best Practice Mode" and Customizations if "Development Mode" is used. The important thing to understand is that even CMDB data has a T table. This discussion is not about the design of AR Schema tables as it is about the CMDB data structures, so please forgive me if I am leaving out some details related to AR Schema database.
CMDB Metadata and the Common Data Model (CDM)
CMDB application is made of several components. Some API's, minor workflow, database tables and CDMB metadata. The last one is built in memory when the CMDB API is loaded by AR Monitor. It mainly queries forms like "OBJSTR:AttributeDefinitions" and "OBJSTR:Class" as the two main components for the hash table (arrays) where the data will be processed. The main difference between CMDB data structures and "arschema" data structures is that CMDB uses class hierarchy which "inherits" fields from its parent tables. The inheritance is simulated and for the data structure of a CI or Relationship this just means that tables have common purpose for data that is distributed. There is no other structure to date that uses this approach to storing data within the ARSCHEMA. Only the CMDB does this.
If you like to know more about the data management structures then please see DMTF (Data Management Task Force) design (http://www.dmtf.org).
In this design all common attributes of objects are stored at the Base Parent class and get more specialized with their child classes. A parent class that has a child class with specialized attributes will not be aware of its child attributes. For example attribute "MACAddress(C260140111)" will only be found in BMC_LANEndPoint that could not be associated with a BMC_Person record where it has no meaning. The attribute that is appropriate to define an object will be owned by a class that closely matches the attributes of real world object. Sending a SQL query to BMC_BaseElement class where the MACAddress is included in the where clause will result in error: "Invalid column name 'MACAddress'."
These tables were designed to show data for various reasons, including data visualization. Volume of data is to reflect the data size of discovered items of various types and hence these tables were intended to be trimmed down for faster data processing. Typically the more data is in the table the longer it will take to look it up. Our expectation was somewhere in the millions and some of the largest deployments seen in our experience is in the range of ~80,000,000 records. But those are just the CI's. There are also Relationships, which also use inheritance. Total size for the CMDB data can be in the billions, although such data volumes would require specialized hardware that can handle processing it.
All CMDB tables typically begin with BMC.CORE, also known as the NameSpace, and the Object has human understandable ClassId that adds meaning. ComputerSystem, Person, NetworkPort, these all mean something to us within the context of ITIL. Each class has attributes that increase in number and the more the class is removed from its parent the more specialized it will be. For example all attributes of BaseElement class are inherited by all CI classes, but BaseElement is not aware of any attributes of its children. Editing BMC.CORE forms in Dev Studio only edits that one table, but does not push the change to any other table. For this you need the Class Manager exclusively.
BMC AR Developer studio has the ability to create overlays, which is the "modern" way to introduce customizations to ARSCHEMA. However, these are still customizations that are outside of the CMDB. Unfortunately we cannot prevent AR Administrators from adding customizations to the CMDB due to the nature of features the Dev Studio offers, but it does allow us to easily separate customizations compared to what was available in the past.
All CMDB data structure changes need to be done with Class Manager that then builds the set of instructions for CMDBDRIVER that then does the work of data structure changes in the CMDB. Changes to a parent class will always propagate to its children. If you are using the BMC Developer Studio to add overlays to BMC.CORE:BMC_BaseElement for example, then that change will only be set on the BMC.CORE:BMC_BaseElement class and not propagate to its child classes.
Asset Management (AM) as an example is an application that takes advantage of the CMDB data store, but it does not use the inheritance model of the CMDB. Since Version 8.0 AM stores its data in AST:Attributes and qualifies CMDB classes by records in AST:ClassAttributes. Please refer to Updates to CI lifecycle data attributes - BMC Remedy IT Service Management Suite 8.0 - BMC Documentation for more details of this change.
In AM you then have two halves of a composite record that is then presented to the Asset Operator in one interface. They see the data through AST:<CLASS> views as one record. One half of the data is in AST:Attributes and the other half is stored in the BMC.CORE:<CLASS> and both are joined into one visual interface. In order to join the data into a view that offers Asset Operators the access they need we need to run CMDB to Asset Synch. In this synch all classes that are selected to have a UI in Asset are processed by a triggered synch job. Updates to individual fields (attributes) are also done this way by altering the tables in Asset. Adding overlays to AST:Attributes put locks on these Asset views and this often prevents the CMDB to Asset (SynchUI) functionality to complete successfully .
AST:Attributes must be edited in Base Development mode so that these changes can be added during the synch. CMDB2Asset Synch is Overlay agnostic. It has no idea that overlays exist. The expectations by end users is that CMDB2Asset Synch will update overlays is unrealized. Unfortunately this leads to a significant amount of extra work that is done before reaching out to BMC Support for help.
In conclusion use overlays with caution, or rather not at all with the CMDB data structure and please set your expectations that you'll be likely asked to undo all overlays related to CMDB and joins related to CMDB data structures. Upgrading to AtriumCore version 9.0 will require all previously created overlays of the CMDB forms to be removed.