Developing AR System® Plug-Ins - Filter API, AREA, ARDBC

This week's theme:  Cool Tech Tips
This topic was recommended by a member of the AR System Developer Community.

 

Developing AR System® Plug-Ins – Filter API, AREA, ARDBC

The plug-ins are an advanced feature of AR System. They allow for a tight and efficient integration between AR System and external programs and environments. Employing an AR System plug-in involves:

  • Configuring the AR System Server and plug-in server
  • Developing a plug-in
  • Installing/registering a plug-in (developed by you, BMC, a partner)

Developing a plug-in involves writing code to interact with external programs and environments. This article defines the plug-in types that can be built, documents the development steps, and provides few development guidelines.

 

Plug-in Architecture

First, let's quickly review the plug-in service architecture. A plug-in service is a companion application to the AR System server. It acts as the central location to load plug-ins, and as a channel for plug-in interaction between AR System and the plug-in code.

Plug-in interaction is typically initiated from within AR System, including: user authentication, filter processing, or user actions that require access into external data sources. In each scenario, the AR System server calls into the plug-in service with the necessary information. The plug-in service invokes the appropriate callback method in a suitable plug-in. Once the plug-in method processes the request and the function call returns the output, the plug-in service sends the status and response information back to the AR System server. In essence, an individual plug-in is responsible for the implementation of a pre-defined set of callback functions, as defined by the API. The plug-in service's responsibility is to dynamically load the pre-configured plug-ins and manage the interaction between the AR System server the plug-ins.

 

Plug-in Types

Currently, there are three AR System-supported plug-in types.

  • AR System Filter API Plug-in (ARF): The Filter API is provided to allow for efficient interaction with an external environment during filter processing. This mechanism acts as an efficient alternative to a RunProcess.
  • AR System External Authentication Plug-in (a.k.a. AREA): An AREA plug-in is designed to allow the usage of an authentication mechanism outside of AR System to validate AR System users.
  • AR System Database Connectivity Plug-in (ARDBC): An ARDBC plug-in is designed to enable external data store interaction as if it were a table within the AR System environment.
 

Developing an AR System Plug-in

Now let's discuss AR System plug-in development. The following steps get you started:

  1. During your AR System server installation, choose the option to install the API component. Doing so, installs the plug-in development related APIs: header files, library files, and a few sample skeleton programs.
  2. Install a C/C++ development environment, e.g. Microsoft Visual Studio, or compiler/linker, etc., plus your favorite text editor.

Developing a plug-in involves creating a C DLL or, a shared library, that uses the above APIs. It also uses the dependencies to interface with external data sources or to perform the plug-in logic.

Note: An easy way to get plug-in development kick-started is to borrow the skeleton source files provided, make the necessary file name changes, and use the file set (source files plus project/makefile) as a starter set for your plug-in implementation.

  1. Implement your logic with the appropriate callback functions.

As mentioned previously, a plug-in is responsible for supplying the logic for a set of callback functions. At a very high-level, all types of plug-ins provide some common set of functionality and conform to the same set of rules. Additionally, they will also provide some information specific to implementing that plug-in type.

 

Implementing Common Callback Functions

The following are the common callback functions that a plug-in can implement.

  • ARPluginIdentify – By implementing this function, the plug-in author can supply the plug-in's identity information to the plug-in service. This information includes: the type, name, and version. At run-time, this is the information used by the system to refer to the plug-in. The plug-in service calls this function once, when initially loading the plug-in.
  • ARPluginSetProperties – (OPTIONAL) The plug-in service calls this function to provide useful configuration information. Although this function is a callback function, its main purpose is to receive information in the plug-in context from the plug-in service and use that information later. Most callback functions are called by the plug-in service to ask the plug-in to do some thing in the current call context, but in the current implementation, this function is used by the plug-in service to give the "logger" function handle to the plug-in. After getting this call, the plug-in can store the log function pointer, and later on, as the plug-in intends to log some of its activity, can simply piggy back on this logger (i.e. C function pointer), instead of re-inventing it's own logging mechanism. If you don't intend to take advantage of this option, you can skip implementing it or provide an empty implementation.
  • ARPluginEvent – (OPTIONAL) If a plug-in implements this callback method, the Plug-in Service calls this function whenever specific events occur in the system. For example, whenever there are any configuration changes within the AR System Server, the Plug-in Service calls this function in each plug-in that implement this method. Thus, if your plug-in needs to react to specific events in the system. then your plug-in must implement this method.
  • ARPluginInitialization – (OPTIONAL) The plug-in service calls this function when it first loads the plug-in and gives the plug-in an opportunity to initialize itself and acquire any resources that the plug-in intends to use for the life of this plug-in's instance.
  • ARPluginCreateInstance – (OPTIONAL) The plug-in service is designed to employ multiple threads to call the methods of any loaded plug-in as needed. This architecture allows the plug-in service to support better scalability/throughput capabilities. For example, the plug-in service may call multiple ARDBCCreateEntry function calls into a specific plug-in concurrently, if multiple AR System users were to request access to the external data source pointed to by this plug-in at the same time. While this design requires an individual plug-in to manage thread safety, it doesn't diminish plug-in flexibility however. Specifically, this callback function lets a plug-in implementation establish a "thread specific storage" data structure that can be shared across plug-in calls within the same thread. The point here is to give opportunity to plug-in authors to establish shared resources that can be reused across plug-in calls in a thread-safe manner. This shared resource is referred to as an "instance" which is an opaque pointer to whatever custom data structure the plug-in author may create in this function. Later on, the plug-in service would hand this instance pointer to subsequent callback functions.
  • ARPluginDeleteInstance – (OPTIONAL) This is an opportunity to release any shared resources, i.e. instance data structure that was previously established within ARPluginCreateInstance. The plug-in service pairs up the Create/Delete Instance calls and places any calls between them that can re-use the instance data without each call going through a costly initialization of the data structure.
  • ARPluginTermination – (OPTIONAL) This is an opportunity for the plug-in to release any global resources acquired during ARPluginInitialization.

Generally, all these plug-in callback functions follow the same structure as other AR System API functions. The plug-in service calls these functions, which typically are expected to return a success code, and fill in a status list parameter with additional status information related to plug-in processing. In addition, the plug-in service supplies the necessary input data, specific to a given callback function, and the plug-in accepts the data, processes it, and returns the appropriate response data to the plug-in service.

Warning:  Plug-in operations run synchronously, i.e. AR System waits for a plug-in to complete its processing before the server will continue its processing. Thus, a badly written plug-in may adversely impact AR System performance and stability. A plug-in developer must pay particular attention to the following areas: thread safety, minimal processing logic for optimal code execution, only implement callback methods that are required or have custom logic for a given plug-in, allocate/free resources, as appropriate, to prevent resource leaks and, finally, optimize any costly operation that must be performed (or data structures that must be built) by employing a meaningful caching strategy.

Now let's talk about the specific callback functions that need to be implemented by plug-ins depending on their type.

 

AR Filter API (ARF) specific callback functions

The Filter API enables efficient interaction with an external environment during filter processing. In short, it lets you place a "custom function call" that can be invoked from within filters. When the filter workflow is executed, AR System Server fills the necessary input parameters, comprised of the contents of the current transaction and initiates a call into this Filter API callback method as needed. It waits until the custom function finishes its processing, receives output from this custom function call, and proceeds further with rest of the workflow processing. When compared to a RunProcess, which involve starting and stopping of processes frequently, the filter API provides a more efficient mechanism to perform a custom operation. From the implementation standpoint, this is the easiest kind of a plug-in to develop because you only need to implement only one callback method.

  • ARFilterApiCall – This callback function has an value list as input argument. It is expected to fill an output value list. Thus, this function is slated to receive arbitrary number of input values and to return an arbitrary number of response values. Implementing this function is fairly trivial. The plug-in author must document the plug-in's input/output arguments – so that a workflow developer can suitably map the necessary input parameters and define the response assignments within the Set Fields action of a filter.
 

AR External Authentication (AREA) specific callback functions

The sole purpose of an AREA plug-in is to provide an external authentication mechanism, so that AR System can defer authentication responsibility to a configured external mechanism when needed. For this to work, an AREA plug-in must provide the following callback functions:

  • AREAVerifyLoginCallback – Simply put, this is the heart of an AREA plug-in, where all the interesting action happens. When the AR System server determines that it needs to defer authentication of a user to a configured plug-in, a call is initiated into the plug-in service, which calls this function with the appropriate parameters. The plug-in implementation must use the specified input data, such as user name, password, and IP address of the client machine, where the user initiated this request, authentication string parameters and determine whether the user can be authenticated with supplied information. If a user is valid, the user/password match in the external system. Then, the plug-in must respond back with a successful login status (AREA_LOGIN_SUCCESS). In fact, there is a response data structure (AREAResponseStruct) that holds this status value. The plug-in can fill it in and return it to the plug-in service. The details that can be filled are:
  • login attempt status
  • email address of user per the external source
  • last modified time of the user in the external source
  • preferred notification mechanism for AR System notification
  • semicolon-delimited group names that this user be considered part of .
  • if a floating license is consumed by the user
  • what string value (if any) can be logged into the aruser.log right after this authentication attempt.

This information acts as an override for whatever corresponding information is present for this specific user in the User form. So, an AREA plug-in can achieve different user information and authorization characteristics for a given user depending on the values it returns in VerifyLoginCallback implementation.

  • AREANeedToSyncCallback –  If there were any changes in the user information, such as a group membership change for a specific user, one would typically expect the AR Server to react and honor those changes in near real-time. If there were no external authentication in picture, AR System server can fulfill this requirement fairly easily, as it owns the user information and manages the user/group information manipulation API requests. In the case of external authentication, however, the user information changes occur in the external user repository, which is, most likely, outside AR System.
    Let's step back and see how AR Server could honor user information changes that may have happened in an external user repository, over which it has no control. The existing Plug-in Service architecture employs a polling mechanism to periodically ask the AREA plug-in whether there were any changes within the external user repository. This callback is precisely for that purpose. In essence, the AR Server periodically polls the Plug-in Service which calls this callback method and asks the plug-in to tell if any of the external user information has changed since it was last asked. Based on the plug-in's response, the AR System server may re-authenticate the user to reflect any recent user information changes in the external user repository.
    How can you implement this callback method? Ideally, a plug-in implementation should be able to fetch the latest modification time of any relevant user information within its external user repository. It is not really critical what the specific change was nor which user's information has changed; what is important is the time when a modification last happened. In a typical NeedToSyncCallback implementation, the plug-in must fetch this last modification timestamp (say, curModifyTime), compare it against previously fetched modification time (say prevModifyTime), and store it for later use (say as prevModifyTime). If the timestamps match, there were no changes in the external user repository, thus the plug-in must return 0. This tells AR System Server that its cached user information is still good. But if the timestamps do not match, then its clear that some thing has changed in the external user repository since the plug-in queried the repository previously. In such situations, the plug-in should return a non-zero value. This tells the AR System Server to invalidate its cached user information. There is a performance implication here. Once a plug-in returns a non-zero value, it causes the AR System Server to re-build its user cache over time, which is a performance intensive operation. So, a plug-in implementer should pay attention to when the plug-in should return a non-zero value.
    Alternatively, depending on your situation, simply returning 0 in this callback implementation, is an acceptable implementation choice. The only down-side is that the server will use already authenticated users information as-is without picking up external user information changes. This would apply only to currently logged in users whose information might have changed after they logged into AR System, and, only for the duration prior to these user sessions are timed out. If you consider the external user repository changes are infrequent, or, AR Server does not have to honor such changes for the currently logged in user sessions; then you can safely return 0 always without more complex processing in this callback implementation. Depending on the nature of your plug-in, external user repository and requirements around how AR System Server should react to user information changes, decide how you should implement this callback method.
  • AREAFreeCallback – In the VerifyLoginCallback function, an AREAResponseStruct is instantiated, filled by plug-in and given back to the Plug-in service. Instead of the plug-in service freeing this memory by itself, it calls AREAFreeCallback right after the VerifyLogin call giving the plug-in author a chance to not free and re-use the same data structure (if one chooses to) or do some other optimization.

Note:  Although this method gives an efficiency opportunity, this can be a potential source of memory leak error prone conditions. To avoid such problems, always allocate a response data structure in VerifyLogin that you return to plug-in service and when plug-in service hands a response structure in this FreeCallback function, you simply de-allocate it.

 

AR Database Connectivity (ARDBC) specific callback functions

Implementing an ARDBC plug-in is more complex than the other plug-ins. Yet creating a simple ARDBC plug-in is just as straight forward as the other two kinds. But to fully implement or support complex features such as transactions, BLOB support, statistics, extensive qualification processing, etc. – significant development effort is required.

Before jumping into ARDBC implementation details, let us step back and see how the ARDBC plug-in works within AR System. To an ARDBC plug-in there is a complementary concept called "Vendor Form" in AR System. A form in AR System is construct that represents an entity including its entry storage structure (database definition), its layout definition (VUIs) and the actual underlying table(s) that would store its instances/entries. For regular/join/display forms, both the definition aspects as well as the underlying storage infrastructure are completely managed by AR System. Whereas in the case of Vendor form, AR System manages only the definition aspects and defers any storage aspects to an external logic (i.e. ARDBC plug-in). From a high-level, a vendor form can be seen as a form, without AR System managed storage, hence relying on some externally managed table (access to which is provided by a specific ARDBC plug-in) with a defined mapping of its AR System fields to suitable columns of the external table.

To facilitate creation of a Vendor Form against a specific plug-in, AR System must offer a workflow author the ability to browse ARDBC plug-ins, their supported tables and type information of columns from any of those tables etc. To fulfill this, AR System expects ARDBC plug-ins to implement the following metadata querying methods:

  • ARDBCGetListSchemas – When a plug-in service calls this function, the plug-in should return names of all tables it provides access to. Note: the external data source may or may not be a database; these "tables" need not even be real tables, instead a plug-in simply wants mimic a table like access to some storage; there need not be a 1:1 mapping between the actual tables within the external data source and what is exposed out of the plug-in.
  • ARDBCGetMultipleFields – This function is used by AR System to query available fields and their type information for a given table of a plug-in. AR System seeks such information from the plug-ins to provide meaningful visual cues in the Admin Tool environment when some one attempts a vendor form creation and also to ensure suitably typed fields are mapped back to the external table columns.

Note: The above functions use somewhat verbose or nested data structures, but keep your cool, all you need to do is simply fill your table names for the GetListSchemas and field names, data types, limit values for fields of your tables in the GetMutlipleFields call.

As noted previously, for vendor form entries storage, AR System server defers that responsibility to the corresponding ARDBC plug-in. Thus, each of the ARDBC plug-ins is expected to support entry-oriented API calls, akin to AR System C API functions, as follows:

  • ARDBCCreateEntry– Plug-in service calls this callback function when AR System requests an entry to be stored into the external source. Plug-in receives input, such as table into which the entry be created, and list of field values represented as a list of (numeric field id, ARValueStruct) pairs. Using these values, the plug-in is expected to create a new entry in its external data source and return an AR System style Request Id of the entry just created.

There are a couple of subtle issues going on here – first, the fields are referred to by numeric ids; second, its expected that the external data source has a unique key that conforms to AR System Request Id notion. These two issues may confuse plug-in developers in the beginning, so let's discuss them a bit here. The external table columns may be identified by string names or could even be numeric ids. However, these external names may or may not align well as AR System field id substitutes. Considering this and the fact that AR System fields are predominantly referred to by their ids (ARInternalId), the plug-in service architecture calls for external column names (designated by plug-in implementation) mapping to AR System internal ids (that are assigned by AR System to Vendor Form fields). Further, to keep entry operations consistent, irrespective of form types, AR System chooses to refer to fields by ids. Thus, it sends any field value data, in terms of (field id, value) pairs, expects plug-in logic to map these ids to external column names and to use corresponding values for further processing. The id-name lookup information ARVendorFieldList parameter for this purpose.

A pretty similar explanation follows for Request Id requirement, with a slight twist. In AR System, entries are identified by their request id. It may or may not be feasible for an external table to have a primary key that is structurally the same as request id (i.e. besides being unique – its numeric or a character string of 15 characters or less). The plug-in service architecture should accommodate some "mapping" for external primary keys to internal Request ids, if the previous (column name – field id) mapping analogy was applied.

Here is the twist, plug-in architects chose to solve this issue by simply requiring the external source's (or, at least the plug-in authors) to provide a primary key that conforms to Request Id rules. Although it sounds like a limitation, in fact, this requirement simplifies the plug-in architecture quite a bit. If a true (requests id – external key) mapping were implemented, then some/all of the following aspects are to be addressed by the plug-in service or individual plug-in. AR System needs to know the external key definition; AR System needs to assign its own Request Id, while few key field values comprise unique key from plug-in standpoint; AR System needs to qualify and map entries by searching them by Requests Ids or by external keys. All of these are simply eliminated by requiring external sources to have a Request Id-like key column in the external table.

This explanation is an attempt to articulate reasons behind certain design choices. With this requirement in place, the Request id always needs to be in the external data source. If it's not possible to alter the external data source, then the plug-in must take the responsibility of maintaining Request Id like column somewhere else and implement appropriate mapping in the plug-in implementation. Also such Request Id value needs to be assigned by plug-in or the external data source itself. Thus, AR System is completely relieved key mapping and entry id creation responsibilities.

  • ARDBCGetEntry – Plug-in service calls this callback function, when AR System requests an entry to be retrieved from the external source. The input parameters include table name, entry id, list of field ids for which values be retrieved. Plug-in is expected to look up the external data source for the row with this entry id as the key. If found, it returns values from this row for all the fields requested.
  • ARDBCSetEntry– Similar to the CreateEntry call, except it updates an existing entry, thus includes an entry Id input parameter indicating which entry to modify.
  • ARDBCDeleteEntry– Plug-in service calls this callback function to request an entry deletion in the external source.
  • ARDBCGetListEntryWithFields– This method is to support querying in the external source. Critical input parameters include:
  • an AR System style qualification structure, which is a recursive tree data structure representing a SQL-like qualification syntax specific to AR System
  • a sort hint
  • field values to retrieve

Depending on the sophistication of the query capabilities desired, the implementation of this routine can be complex. You need to implement AR System qualification processing and translate the qualification to the applicable external source. The rest is fairly simple. The plug-in implementation issues the translated query to the external source, receives the data, and packages it back into the output EntryListFieldValueList data structure. Additional logic is required if the plug-in will sort and chunck search requests.

  • ARDBCGetEntryBLOB– If the plug-in supports a column of AR_DATA_TYPE_ATTACH in a table, then when a user accesses an attachment value for an entry from the corresponding vendor form, the AR System initiates this method into the plug-in. Using the input parameters such as table name, entry id and id of the attachment type field, the plug-in is expected to retrieve the BLOB value from its external source and return it back to AR System in the form of a ARLocStruct structure. Unless your plug-in indicates that it has AR_DATA_TYPE_ATTACH type fields, implementing this method is not needed.
  • ARDBCGetEntryStatistics– (OPTIONAL) A plug-in may choose to support entry statistics, such as count, computed off of a result set matching a given qualification.
  • ARDBCCommitTransaction– (OPTIONAL) The idea here is that the plug-in service may call into a plug-in with an arbitrary number of entry operations with some transaction id. Based on successful workflow processes or failures, AR System may tell the plug-in to commit all these operations or rollback. An advanced plug-in that intends to support transactional capability must implement both Commit/RollbackTransaction callback methods. If the external source behind the plug-in is capable of supporting transactions, implementing the Commit/Rollbacks is fairly trivial. Both these methods should simply issue a commit/rollback respectively to the external source and then begin a new transaction. Thus, subsequent operations all fall into the next transaction that would eventually be committed or rolled back. If the external source is not capable of transactions, yet the plug-in needs to support some transactional notion, a possible implementation choice would be, each of the entry operations queues up the operations into a command list data structure. The CommitTransaction callback iterates over this data structure to issue the actual operations to the external data source and it cleans up the data structure. The RollbackTransaction cleans up the data structure for subsequent transactions to pool its operations. NOTE: The Commit/Rollback callback methods also aid as an implicit indicators for beginning the next transaction. So their typical implementation should be to perform commit/rollback on the current transaction and simply initiate the next transaction.
  • ARDBCRollbackTransaction– (OPTIONAL) See CommitTransaction above.

Note: An ARDBC plug-in developer may choose to support only partial capabilities, in which case some of the above calls may be considered optional. For example, if one were to support a read-only source – then Create/Set/Delete methods are optional.

 

Summary

Plug-ins are custom pieces of logic that can be plugged into the AR System plug-in service, enabling tighter AR System integrations. There are three types of plug-ins – Filter API, AREA, ARDBC. There is a common set of rules for implementing plug-ins. With some effort, an AR System knowledgeable C programmer, can easily create plug-ins to augment AR System with external logic or access to external data sources. Depending on the desired capabilities and scope, the plug-in developer can choose to implement more or less functionality within a plug-in implementation.

~ Appajee
Lead Product Developer, Service Management R&D
Joined BMC Remedy in 1997