Integration with 3rd party systems is, of course, always important. After all, we need ways to get data in and out of Remedy, we can hardly keep it to ourselves. Data shared is value gained - you can get more business value out of each application of they know how to work together. So how do you go about this? There are several ways: running reports is one, exporting to an application like Microsoft Excel is another. But if you want a more integrated option, something automatic, what are the alternatives? There is the Java API, which offers a rather complex way to connect to the system on programming level, but that’s not everyone’s cup of tea.
A common way to integrate with Remedy is via web services, because it is a method supported by many applications. It’s pretty straightforward: you offer an interface, connect via a client and request data. You can both consume and publish web services with AR server so it can go both ways – you can even publish and then consume them again on the same server. One is the server, the other is the client, or the other way around, just make sure you map everything accordingly, keep the WSDL in mind, and … Right, it’s not that straight forward after all is it?
That’s the problem with Remedy’s web service integration: it’s versatile but it’s not very easy to get your head around. The coming two months I’m going to talk about Remedy’s web service integration and to start we’ll have a look at how AR server consumes external web services.
I like web services; they are a clean and reasonably easy way to get data from one system to a completely different system. You don’t need to write a lot of code to do this, as long as you adhere to the web service standard it should be quite straight forward. But what is a web service, and how does this work? Let’s have a look at an example first.
A book publisher registers the data about their book (price, ISBN, publication date, etc) with a company dedicated to distributing it to the book trade. Besides book data, this company also keeps statistics on sales through bookshops registered with them. So how can a bookseller or library get access to this data to integrate it in their own systems? They could perhaps do a search on the company’s database or call the provider to get the information, but that is not integration. How can you get the data from the book data system to the bookseller’s system in a hassle-free manner?
One way of doing it is by offering a data service: third parties such as libraries and booksellers can lookup data provider to their own system. They simply key in an ISBN or book title, and the system returns the full book data: title, description, author, statistics, etc. The key here is not to expose the underlying database. You keep it as simple as possible: send the ISBN, get back the book data.
That’s what a web service does: it’s a way of communication between a provider (the book data company) and the requester (the book seller). You can do this whatever way you want, but the easiest way is to agree on a common standard: the XML web service. This standard describes how the two systems should communicate, how the provider should present the data, what the interface should look like and how the requester should make its request. It’s a common standard, so as long as you adhere to it, both parties can communicate.
The first thing the provider needs to offer is a machine readable description of the operations offered, in an interface that we call WSDL – short for Web Services Definition Language. It specifically describes the functionality offered: how the service can be called, what parameters are expected, what data it returns and what that data will look like. WSDL is a standard and is XML-based. It can be quite complex, but keep in mind that it’s a description of the services on offer. In our example it could describe an operation called getBookData which has an input for ISBN and returns fields like title, author, description, cover, etc.
The bookseller has its own application on the client side to connect to the web service on the server side. This program can read the WSDL file to determine what operations are available on the server and what its request should look like. The client can then send a request to the web service and get the data it requested. This is an XML web service so the requests and responses are once again sent as XML. The format is called SOAP, which stands for Simple Object Access Protocol. It’s another XML based protocol which describes how the data should be formatted when it’s sent between computers (the requestor and the provider in this case).
That’s about it really: there’s a service provider which provides an interface (the WSDL) to its system. It doesn’t really matter to us at the moment how the service provider handles it from there, as the service requester we’re only dealing with the WSDL. The requester reads the WSDL, determines what services are on offer and what the communication should look like, requests are made in SOAP and the responses are sent back (again in SOAP). Once the data is in, the requester can integrate it in its own system.
Right, hope that was clear. Like I said, Remedy supports web services, but we do it both ways. It can act as the provider and make data stored in Remedy available via a web service interface, but it’s also possible to consume a web service stored by an external provider. In that case Remedy is the service requester and the external provider is the service provider. In the client server architecture we’re the client and the provider is the server.
So let’s have a look at a web service and discuss how we can integrate this into Remedy. I’m using a free web service. Not the most useful one, but it will do to illustrate the example. This web service fronts a system which stores data on German cities. The web service allows us to retrieve city names. Our interface with the system is the WSDL which we can access via a URL. It’s all XML so I can just open it in a web browser and read it.
You need to know what you’re looking at, as WSDLs are rather complex. If you’d read it with an explanation of the format it makes more sense. You can, for example, find the port, the connection to the web service:
Or the Types, which describe the data:
You don’t really need to know all the ins and outs of the web service. A web site like wsdl-analyzer.com can do it for you if you want to know what it looks like.
What you need to do is get Remedy to connect to the web service via the URL provided. You can do this during design time in BMC Remedy Developer Studio. Remedy filters allow you to consume a web service. It’s up to Developer Studio to interpret the WSDL and determine what ports are available, what fields can be mapped, etc.
Developer Studio’s interpretation of the web service is based on the WSDL. It finds two operators which can be mapped to the form I base my web service on. As soon as the web service is triggered, BMC Remedy AR System (ARS)should take care of the rest: send out the SOAP request to the web service, wait for the answer, put the received data in the fields of the form. If everything goes okay, there’s nothing else you have to do. But what do you do when things go wrong? How does ARS actually handle the communication?
Let’s go back to our sample web service. Communication between the service requestor and the service provider is done via SOAP. Our sample web service provides an interface to request German city names based on the prefix. So if I want to know all the cities that start with ‘Ber’, I need to send a request in SOAP format which looks like this:
It’s XML, I know what port to use because I interpreted the WSDL. Since I’m communicating using SOAP I know I need to stick to a certain format: I start with an element called Envelope, I use a Header, and a Body. I also make sure I get all my namespace information correct: soapenv for the SOAP elements, ort for the data elements (again as specified in WSDL). It’s good to know the basics of how SOAP fits together, but if I’m unsure I usually just use a SOAP consumer to do it for me.
These kinds of applications interpret the WSDL, inform me what operators are available, and build the XML requests based on the SOAP and WSDL specification, unless I need to work my way back to a problem in the WSDL, I don’t do this myself. My tool-of-choice is a freeware application called SoapUI.
Next the request is sent off. It’s wrapped in a HTTP POST request and sent to the web server (on the service provider’s side) like a normal HTTP request. The web server handles it and feeds back the answer, again in SOAP:
I already know it’s using this format since it’s described in the WSDL: it will start with GetPrefixedEntriesResponse followed by an element called GetPrefixedEntriesResult. In this case the value is: Beratzhausen;Berching;Berchtesgaden;Berg (Pfalz);Berg b.Neumarkt i.d.OPf.;Berg im Gau;Berg, Kr Ahrweiler;Berg, Kr Ravensburg;Berg, Oberfr;Berg, Starnberger See.
Remedy does the same thing. As soon as that filter is triggered, a SOAP request is put together and sent to the service provider. That’s a bit of a simplification actually; this is what the real data flow looks like:
During run time the filter will set everything in motion. AR server can’t communicate with the web service directly so it generates XML with the requests it needs to make. This XML contains mapping information (what goes where) and the request information (what do we need from the web service). This then goes to the Plugin server, which is a separate component residing on the server where AR server is installed. This is responsible for formatting the XML as a SOAP request. We already have the XML data, we just need to wrap everything in SOAP and send it off. We use an external component for this called AXIS.
Notice Mid-Tier is not involved here, this is all AR server side. What we’re looking at is AR server talking to the external web service, and that goes via WSDL (design time) and SOAP (run time). You don’t need Mid-Tier for this, the only web server involved is on the web service side, and that’s not our concern.
You can visualise most of this traffic via various logging options, so it will become a lot clearer. I am going to consume the same web service we’ve been using so far and walk you through the logs during run time. I have a simple form with two character fields and a filter which maps the two fields. This is where I defined my filter.
Developer Studio interpreted the web service based on the WSDL; it found the endpoint, operation, etc. and populated the required fields. I then mapped this to fields on my form (input and output) and told the filter to execute on Submit. This is all on AR server of course, so what happens when my filter is executed? AR server’s job is to handle the mapping and pass on the values we filled in (via the form). AR server doesn’t have the capability to send SOAP itself, it relies on a component called the plugin server to do this. It does make sure the data is in the right XML format already before passing it on. There’s a way to visualise this by setting an environment variable (ARXMLLOG) but to be honest this doesn’t always go very smoothly, I rarely use it. I’ve included a knowledge article if you’re interested. Instead I use the Java plugin logging.
This is the second component that’s involved; it’s entirely written in Java so it comes with Java logging options. There’s a lot of stuff logged, but this is how I read the communication between AR server and the plugin server: I first look for the entry ‘soapOperationDoc’ which contains AR server’s interpretation of the operation we’re going to use. It gets this information from the WSDL:
So the operation is ‘orteStartWith’, which corresponds with our Filter definition. We expect back ‘OrteStartWithResponse’, which it determined by looking at the Filter output mapping and the WSDL. The next are the mappings AR server expects: the form (webservicetest), the form elements (536870913) and the corresponding field in the web service (prefix). This is all coming from AR server.
From here the plugin server takes over. It knows what to do (the first XML bit) and what fields to map (the second XML bit), but there’s a third bit of information that’s recorded slightly later. I removed the timestamp and a few other parts to make it a bit more readable, but this is what the plugin server does:
It records the service, method and URL it’s using. The input document again comes from AR server which has the value I filled in (Ber). Next, it’s SOAP wrapping, with the end result logged as the ‘Input Env’. This is the actual SOAP message the plugin server plans to send over. ‘About to Invoke’ is the moment the request goes out and from there on we’re dealing with the result: ‘Output Envelope’ is the result of the web service followed by the piece of XML the plugin server is planning to send back. From that moment it’s up to AR server again to get the information back to the form.
So remember that AR server and the plugin server are communicating via XML: AR server sends operations, mapping information and the values (the input document). The plugin server extracts what it needs, wraps everything in a SOAP envelope and sends it off. It then processes the response, creates an XML file AR server can understand and sends it back.
I’m going to leave you with one last trick I quite like to use. You see, the plugin server is a Java application, so the usual Java options apply here. I can specify heap settings, GC settings, etc. I can also tell Java to redirect the traffic to a proxy rather than straight to the web service, so I could for example set up my own proxy and send traffic there. We’ve talked before about Fiddler, and as Fiddler is effectively a proxy server, I could do just that: redirect traffic from the plugin server to Fiddler, log it here, and send it to the web service. This is helpful, as you don’t have to work your way through the plugin logs and you see the actual HTTP communication between the two servers. This is what I add in armonitor.conf:
I tell Java there’s a proxy to be used here and tell it to send it traffic to localhost:8888, which is where Fiddler runs. I’ve already configured Fiddler to act as a proxy, so if I repeat my request, this is what I see:
You recognise the HTTP traffic containing the SOAP request and the SOAP response.
I hope this gives you some idea of how Remedy integrates web services and how logging works here. Since we’re dealing with 3rd party components things can go wrong, but if you know where to look and have a basic understanding of the process you should hopefully be able to tackle any problems that come your way. And if not, well you can always give us a shout in BMC Remedy Customer Support.
Right, that's all for now. Until next time,