SOAP Web Services Best Practices

Version 23
    Share This:

    << Other Best Practices

       << AR System APIs and Integration Interfaces Overview

     

    Context and Issue Statement

     

    BMC provides a synthetic and straight-forward description of web services and their implementation in AR System. See the “Integration” documentation for your version (7.6.04 or 8.1) for :

    • A short introduction to web services
    • A description of the web service architecture in AR System
      • You can also see this document for a quick dive-in (the document is old but still valid)
    • Consuming (from a filter set fields action)
    • Publishing (from Mid-Tier)
    • Registering (UDDI)

     

    Trending in Support: AR server, the SOAP client (really) by Justin Bakker is a great introduction to SOAP web services in the AR System context.

     

    This document adds some practical information gathered from experience and discussions on these boards.

     

    • Performance concerns
    • ITSM Web Services
    • Designing robust implementations
    • Complex web services
    • Web Services and SSL
    • Debugging tools and techniques
    • Most common error messages
    • Known limitations and issues

     

     

    Consuming vs Publishing

     

    In the following paragraphs, we consider that we are on the AR System side of an interface.

    - Consuming means reaching for an external web service from AR System via the Web Service ARF Plugin, in order to push data to or get data from another application.

    - Publishing means exposing methods via the Mid-Tier for external applications to push data to or get data from AR System.

     

    Performance concerns

     

    When allowing 3rd Party consumption of your published Web Service, the Mid Tier is used as the interface platform.  As the Mid Tier serves User access also, dependant on the perceived load by the Web Service, additional dedicated Mid Tiers should be considered for Web Service integrations.  With the standard scalability of AR System, this is a cost effective way to build a robust integration platform.

     

    When consuming a 3rd Party Web Service, the AR System Web Service plugin is utilized.  Dependant on the consumption load, additional performance tuning may be required on the AR side.

     

     

     

    ITSM Web Services [publishing]

     

    Using BMC ITSM OOB Web Services : How to test the ITSM Incident Interface based Web Services. There is also some example .NET (VB) code showing how to Create, Query and Modify using the Incident based Web Services. (Carl Wilson)

     

     

    Designing robust web service interfaces

     

    If you are designing a simple query interface to allow your partner to search for items or get some details for them, then you should only need to publish a web service with a getEntry and a getEntryList methods directly on your form, provided you keep the "performance concerns" in mind.

     

    Now if you are going to design a full AR System WS interface, with bidirectional objects updates, there are a few questions you need to answer and points of attention  :

    - consider that the partner application might be unpredictably unavailable, or your request could be rejected for an unanticipated reason : you should handle the errors without blocking your users

    - consider that your application might be unpredictably unavailable (MT or AR System down) : if your partner sends his requests in bulk and not necessarily in the right order, you should be able to handle them without loosing consistency

    - consider that the partner application or yours can have scheduled downtimes : an admin should be able to deactivate the interface temporarily, and resume it without loosing data

     

    For all these reasons, keeping all transactions in a staging form is strongly recommended.

    The diagram below shows a sample design for a bidirectional interface for incidents.

    AR+System+WS+Interface.png

     

    Remember that if the partner web service is unavailable, you will receive an error from the web services plugin. If you consume the WS "on modify" without handling the errors on the HPD:Help Desk form, the user simply won't be able to save his changes.

    You have two options here :

    - call the WS synchronously when the staging form entry is created/modified but catch exceptions on your staging form with an error handler

    - use an escalation to call the WS asynchronously after an entry in the staging form has been flagged; handle exceptions too.

     

    If you can bear with a few minutes of delay, the second option is the way to go. If you choose the first option the WS call will be done quickly most of the time, but if the partner is malfunctioning and causes a timeout, your users will be blocked until the timeout is reached.

     

    A staging form comes with a few perks too :

    - you keep track of every transactions and their status, see the error numbers and labels

    - you can add a mechanism to "replay" an incoming or outgoing transaction after an error

    - you can make statistics about your interface

    - you can build a user friendly console for the administrators, empowering them with the options listed above

     

     

    Complex Web Services

     

    A lot of discussions around web services in AR System community are about consuming or publishing complex web services. We call web services "complex" when they include more than one occurence of an element. For example, you may want to send or receive informations on an update of an incident or change along with its worklog entries and its related CI.

     

    To either build a complex web service or to consume one, you will need to define the following elements :

    • The main form on which the web service will be based (ex: HPD:Help Desk)
    • One or more child forms that contain the multiple elements (ex: HPD:Worklog)
    • A distinguishing key (DK) : the unique key (for this transaction) that will differentiate the children records (ex: Worklog's ID)
    • A foreign key (FK) : the child form field that will contain the parent form's Primary Key (ex: Incident's ID)

     

     

    [publishing] a complex web service

     

    Let's say you have 2 forms :
    - P, the parent form (ex: HPD:Help Desk)
    - C, the child form (ex: HPD:Worklog)

     

    You want to build a "create" operation.
    For P, the Primary Key is the Request ID by default. This primary key is obviously unique.
    In C, you need two keys because of the sequence of which AR System creates the records. First, it needs a key to link the records of C to the records of P. This is the foreign key. Given that you can have multiple records in C for a record in P, it is not unique. When AR System create the records, it pushes P's primary key into the field you defined as foreign key. You must not map this field in your input mapping.

    The second key will identify, uniquely in the context of this request, the record of C. This is the distinguishing Key. This field you have to set with the input mapping. So this value doesn't have to be unique table-wide, but request-wide.


    Here is an example with values. You want to push a request with 1 parent record and 2 child records.

     

    <P>
         <...>
         <C>
              <id>C1</id>
              <...>
         </C>
         <C>
              <id>C2</id>
              <...>
         </C>
    </P>
    
    
    
    
    
    

     

    When you send that request to your WS, here's what you get :
    - record in P reserves a request ID, for example "1"
    - records in C are created
      - AR System sets the FK field to "1"
      - You mapped the "id" field from the input request to your DK field, so the first one gets C1 and the second gets C2.

    You can rerun this multiple times. The only thing that has to be unique in your tables is your request ID, which AR System is taking care of.

     

     

    Web Services and SSL

     

    This part is focused on the configuration of AR System and its arplugin. The commands for handling certificates and truststores depend on you server's OS and Java version. This doc by Karl Miller provides a lot of details about how to deal with certificates with Remedy 8.

    Please post your question in the AR System community forum if you have one on the subject.

     

    [publishing] a web service with server-side certificate


    Server-side certificate, to make people access your web service via https instead of http, can be configured on the web server (Apache, Apache Tomcat if stand-alone, IIS, Web Logic, ...) or on the reverse proxy gateway.

    Either way, this is not an AR System concern.

     

    [consuming] a web service via https

     

    To encrypt the http stream and authenticate the server hosting the WS, if it's private key has not been compromised, you can use https instead of http. For this, the server needs a valid certificate that has been approved by an authority that is trusted by your server.

     

    Java comes with many world wide trusted authorities, but big IT departments often use their own authority for internal certificates:

      • it's cheaper, if you have many certificates to handle
      • it's more secure (search for DigiNotar being compromised for example)

    Most of the time, servers are already configured to accept the internal authority's certificates.

     

    To make the plugin validate the server's certificate, you need :

      • a truststore (if the authority is not already trusted)
      • to add the authority's certificate to your trustore, or simply the WS server's certificate if it's self-signed
      • to configure arplugin to use the truststore
        • edit the armonitor.conf file
        • locate the line for java arpluginsvr**.jar
        • add the following parameters to the command line

    -Djavax.net.ssl.trustStore=<path to your truststore> \

    -Djavax.net.ssl.trustStorePassword=<password> \

     

     

    [consuming] a web service using a client-side certificate

     

    After it has authenticated itself, the server that publishes the web service can require that your server authenticates itself. If your private key has not been compromised, it pretty much guarantees that no rogue server will access the web service by impersonating your server.

    This is a good idea if anonymous login has been configured, for example.

     

    To make the plugin show a client certificate, you need :

      • if you need a signed certificate :
        • a .csr file ("Certificate Signing Request") to send to the certification authority
        • the .cer file the authority will return to you
      • else :
        • the .cer of your self-signed certificate
      • a PKCS keystore
      • to store your certificate in the keystore
      • to configure arplugin to use the keystore and to send the client certificate when required
        • edit the armonitor.conf file
        • locate the line for java arpluginsvr**.jar
        • add the following parameters to the command line

    -Djavax.net.ssl.keyStore=<path to your keystore> \

    -Djavax.net.ssl.keyStorePassword=<password> \

    -Djavax.net.ssl.keyStoreType=pkcs12 \

     

     

    Debugging tools and techniques

     

     

    Usually, when you develop applications and interfaces based on AR System, your best debugging friend is the server-side logging functionality (escalations, filters, SQL, arplugin, API). For web services it will help, but sometimes it won't help enough.

    Use these log files for debugging:

    It might be enough for the occasional debugging. For the conception and development of an interface, you will need a SOAP development tool to forge your sample requests, do some unit testing for both inbound and outbound SOAP requests.

    There are a few toolsout there, so take some time to check them out :

     

    SOAP UI, which comes in a free version with all we need for most situations:

    • Creates sample requests from WSDL file ([publishing] and [consuming])
    • Creates unit tests ([publishing])
    • Creates mock web services from a WSDL file ([consuming])

     

     

    Most common error messages

    1. In DevStudio, while loading a WSDL ([consuming])

         When an error occurs while loading a WSDL within a "Set Fields" action in a filter, the most obvious symptom is that nothing happens. Clicking the "reload" button leaves the "ports" and "operations" menu desperatly empty. Remember that only your client is used at this stage, not your AR System server. The first thing to do is to make sure the "Messages" window of DevStudio is open. The messages displayed there are often obnoxious, but you need to bite the bullet and take some time to read the details.

     

      • PKIX path building failed

    ERROR: URI not found https://mydummyserver:1337/?wsdl

    Additional Details: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

     

         You are trying to consume a web service hosted on a server that requires https. The certificate provided by the server is not considered as trustworthy. What you need is to add the public key of the host of the web service to a local truststore. If the publisher didn't provide it, you can get it from your browser. On Firefox, click the lock icon next to the URL, then "more informations", "show certificate", "details", "export". Save it into a crt file. Now you can add it to a local truststore using the java "keytool", and change your devstudio.ini file to add :

    -Djavax.net.ssl.trustStore=<path to your truststore>

    -Djavax.net.ssl.trustStorePassword=<password>

    Restart DevStudio, and if the settings are correct, you should not see this error.


      • An error occurred trying to resolve schema referenced at [...]

    The WSDL references a schema (xsd) that cannot be located/accessed. Make sure the URL is correct, and that your workstation can resolve it. You can test this with a simple browser. If you can, check if your browser is accessing it via a proxy server. If it does, make sure your DevStudio is also configured to use this proxy (see below for the "proxySet" option).

    If your workstation cannot access to file by any mean, you have to download the file or request it from the publisher. Save the wsdl and the xsd files in a directory and edit the wsdl file. Replace the "http://..." adress with "yourfilename.xsd", save and load this wsdl.

     

     

      • Unable to locate or load the specified file {0} [...]

         This one is the most puzzling error you can get. It seems like quite generic and can be seen for a number of reasons. But the most common cause for it is that the WSDL is not understandable by DevStudio in its current state. See "known limitations and issues" below.

     

         It can also be raised when the WSDL has some references to URL that cannot be resolved, either because they are just dummy URL (like "http://tempuri.org/...") or because your workstation itself cannot resolve them (not on the same network, no access to Internet, or access through a proxy server). To solve this, there are two solutions :

        • modify your hosts file to map a dummy IP address with the FQDN (Fully Qualified Domain Name), ie: 127.0.0.1  tempuri.org
        • if you have access to Internet via a proxy server, modify your devstudio.ini to let it know :

     

    -Dhttp.proxySet=true
    -Dhttp.proxyHost=proxy_address
    -Dhttp.proxyPort=8080
    
    
    

     

     

     

    Known limitations and issues

     

     

    • [Consuming] Documented limitations regarding specific XML schema and WSDL structures render some W3C-compliant web services impossible to consume by AR System plugin.

     

    The supported structures depend on your version of AR System.  See your version’s Integration document for more details, or here for version 8.1.

    So how do you handle this? Well, get ready for some nasty XML tinkering. You are going to forge your own WSDL to make AR System understand it.

    Sometimes it will not be possible. E.g. replacing <s:any> if the sequence never has the same structure.
    Sometimes it will be approximate.  E.g. replacing an unbound limit to 10, assuming that you will never receive more than 10.
    Sometimes you will brilliantly reproduce the original WSDL with a supported structure, after hours of pain. E.g. merging an external unsupported xsd file into the wsdl.

     

    Either way, you know you will have a lot of work if your interface nemesis changes. This is also a major risk you have to consider when charging for the conception of a web service interface. Out of 6 web service-based interfaces I created, 4 required a locally edited WSDL.

     

     

    • [Consuming] the appropriate web service URL according to the current environment.

     

              During your development process, chances are you're not consuming the production webservice of your interface mate. So how do you move to Q&A or production or other environment?

              Starting from versions 7.6.04, you can query the Registry to overwrite the WS location at run time, but you can't set the endpoint directly from a field. See "Querying the registry" form more details.

              One solution is to modify the wsdl locally, set an alias for the “location” tag and configure the “hosts” file on your server to target to correct environment. Of course, it supposes that all the URL are the same except for the host name (same port, same path).

     

     

     

    Frequent terms and abbreviation

     

    FQDN : Fully Qualified Domain Name

    SOAP : Simple Object Access Protocol

    WS: Web Service

     

    External links

     

    Jose Huerta posted a good article here