Use this material as a basis for discussion, for the implementation of BMC Atrium Orchestrator in widely distributed network and if Web Services / SOAP integration is of concern, during a "BAO Workshop". If you like more information, see the links at the end of the document.
The information presented cover only a subset of the items involved in planning a high available infrastructure for your business transactions.
Let's focus first on SOAP.
Simple Object Access Protocol
SOAP is transport neutral and supports multiple transports like Http, ftp, JMS, SMTP, TCP, MQ, etc. and is platform agnostic. They implement WS* specifications like WS-Atomic Transaction, WS-Security, WS-Interoperability, WS-Reliability, etc and hence are highly secure, reliable, and transactional.
HTTP or HTTPS can be viewed as the current de-facto standard transport binding for Web services. It is frequently said, however, that HTTP is inherently unreliable and not appropriate in situations where guaranteed delivery and other Quality of Service (QoS) characteristics are required.
SOAP over HTTP is the easiest and the most interoperable way to implement Web services today. So how much can we trust SOAP/HTTP Web services and should HTTP even be considered in an enterprise setting where QoS is almost always a requirement?
We need to remember that HTTP as well as virtually any other communication protocol today is based on TCP. The whole purpose of TCP is to be able to transfer data reliable.
TCP & UDP
Differences in Data Transfer Features
TCP ensures a reliable and ordered delivery of a stream of bytes from user to server or vice versa. UDP is not dedicated to end to end connections and communication does not check readiness of receiver.
TCP is more reliable since it manages message acknowledgment and retransmissions in case of lost parts. Thus there is absolutely no missing data. UDP does not ensure that communication has reached receiver since concepts of acknowledgment, time out and retransmission are not present.
What does it mean in terms of Web services? Say, we have a one-way service. If we invoked a service and received a successful reply (no SOAP faults or HTTP errors), we can be confident that our SOAP message reached its destination. But what if the connection went down in the middle of the call and we received a timeout error? Our message exchange is now in an ambiguous state. If the message has not been received, we need to invoke the service again, but, on the other hand, if it has been received, the duplicate message may lead to an inconsistent state if the service provider is not able to handle duplicates correctly.
QoS & WS-RM
QoS provided by messaging systems or WS-RM helps us in this situation by ensuring that the message will only be delivered once; messaging systems can also handle re-delivery of messages and "store and forward". Messaging systems also provide another important benefit by allowing a Web services call to participate in an atomic transaction. This allows service consumers to keep in synch multiple resources thus improving data integrity and reliability.
So where does it leave SOAP/HTTP services? Based on the explanation above, SOAP/HTTP is not the best fit when:
- Service providers can't handle message duplicates (in other words, an operation performed by the service is not idempotent).
- Different data resources owned by service provider and service consumer must be in synch at all times.
However, we still might be able to use SOAP/HTTP if we make our service consumers a little smarter. Specifically, service consumers must be able to meet the following requirements:
- Consumers must be able to retry a Web service call in case of a failure due to a connectivity error. In the simplest case, a consumer's application may choose to show the error message to an end user and let the user press "submit" button again. However, most consumers will probably choose to automate the retry logic and at least log unsuccessful attempts and potentially alert an application administrator.
- Consumers must be able to uniquely identify the data that they are sending to providers. Suppose we have "addEmployee" service that takes an employee XML message as an input. The employee message must have unique ID set by the consumer of the service, so that when the invocation is retried, the service will be able to detect that the employee was already added as part of the previous call. This means that popular techniques using database sequences or autoincrement fields for generating unique IDs do not work with SOAP/HTTP Web services. Service consumers must implement their own way of creating unique IDs (perhaps relying on some kind of a natural key or using UUID).
- Consumers must be able to handle certain application-specific errors (SOAP faults). For example, "addEmployee" service may return "Employee with this ID already exists" error after "addEmployee" call was retried in response to a connectivity failure. The consumer's application will have to stop retrying after catching this error. This situation may also require an end user (or an administrator) involvement to verify that the employee was indeed added.
As an example, let's take a look at "Create/Read/Update/Delete" (CRUD) operations. While real-life services oftentimes do much more than just "CRUD", it is a valid simplification for our purposes.
|Operation||Is Indempotent?||Required Service Consumer Behavior|
Must be able to retry the call
Must be able to handle "record already exists" error.
|Update||Yes||Must be able to retry the call|
Must be able to retry the call
Must be able to handle "record does not exist" error.
So what is the takeaway? SOAP/HTTP can be used in many (if not most) situations, however, implications of this decision must be fully understood. All service consumers must be made fully aware of these implications. Most importantly, service consumers must implement proper logic for handling connectivity failures and application errors. In some cases, users of service consuming application may need to be instructed about how to react to certain application errors.
Representational State Transfer
REST leverages the already built web infrastructure like Web which is mainly built on http Programming model. REST is built completely on http and leverages all the attributes of http like Uri, http methods, caching, chunking, http error codes and so on.
Unlike SOAP, which only uses POST operation, REST uses all http operations like GET, POST, PUT, and DELETE. REST services are easily discoverable through Web URLs as against the Service Discovery in SOAP. It uses http error codes to communicate response messages to the user.
Reliability and Transaction
Reliability can be achieved using http Idempotent put operation and Transactions can be achieved by treating Transaction itself as a resource.
- Create Transaction (use POST)
- Commit transaction using PUT
- Rollback using DELETE
- Retrieve transaction using GET
SOAP vs REST?
- Use SOAP when you are developing mission critical applications which needs high security, reliability, needs multiple transports, and are highly transactional in nature.
- Use REST when interoperability of the application is highly needed, performance is critical, apps. Need to be accessed from multiple devices, needs high scalability and faster time to market with moderate security, reliability and transaction requirements.
Areas where SOAP based WebServices is a great solution:
- Asynchronous processing and invocation: If application needs a guaranteed level of reliability and security then SOAP 1.2 offers additional standards to ensure this type of operation. Things like WSRM – WS-Reliable Messaging etc.
- Formal contracts: If both sides (provider and consumer) have to agree on the exchange format then SOAP 1.2 gives the rigid specifications for this type of interaction.
- Stateful operations: If the application needs contextual information and conversational state management then SOAP 1.2 has the additional specification in the WS* structure to support those things (Security, Transactions, Coordination, etc). Comparatively, the REST approach would make the developers build this custom plumbing.
Areas where RESTful WebServices are a great choice:
- Limited bandwidth and resources: Remember the return structure is really in any format (developer defined). Plus, any browser can be used because the REST approach uses the standard GET, PUT, POST, and DELETE verbs. Again, remember that REST can also use the XMLHttpRequest object that most modern browsers support today, which adds an extra bonus of AJAX.
- Totally stateless operations: If an operation needs to be continued, then REST is not the best approach and SOAP may fit it better. However, if you need stateless CRUD (Create, Read, Update, and Delete) operations, then REST is suitable.
- Caching situations: If the information can be cached because of the totally stateless operation of the REST approach, this is perfect.
Web service QoS requirements
The major requirements for supporting QoS in Web services are as follows:
- Availability is the quality aspect of whether the Web service is present or ready for immediate use. Availability represents the probability that a service is available. Larger values represent that the service is always ready to use while smaller values indicate unpredictability of whether the service will be available at a particular time. Also associated with availability is time-to-repair (TTR). TTR represents the time it takes to repair a service that has failed. Ideally smaller values of TTR are desirable.
- Accessibility is the quality aspect of a service that represents the degree it is capable of serving a Web service request. It may be expressed as a probability measure denoting the success rate or chance of a successful service instantiation at a point in time. There could be situations when a Web service is available but not accessible. High accessibility of Web services can be achieved by building highly scalable systems. Scalability refers to the ability to consistently serve the requests despite variations in the volume of requests.
- Integrity is the quality aspect of how the Web service maintains the correctness of the interaction in respect to the source. Proper execution of Web service transactions will provide the correctness of interaction. A transaction refers to a sequence of activities to be treated as a single unit of work. All the activities have to be completed to make the transaction successful. When a transaction does not complete, all the changes made are rolled back.
- Performance is the quality aspect of Web service, which is measured in terms of throughput and latency. Higher throughput and lower latency values represent good performance of a Web service. Throughput represents the number of Web service requests served at a given time period. Latency is the round-trip time between sending a request and receiving the response.
- Reliability is the quality aspect of a Web service that represents the degree of being capable of maintaining the service and service quality. The number of failures per month or year represents a measure of reliability of a Web service. In another sense, reliability refers to the assured and ordered delivery for messages being sent and received by service requestors and service providers.
- Regulatory is the quality aspect of the Web service in conformance with the rules, the law, compliance with standards, and the established service level agreement. Web services use a lot of standards such as SOAP, UDDI, and WSDL. Strict adherence to correct versions of standards (for example, SOAP version 1.2) by service providers is necessary for proper invocation of Web services by service requestors.
- Security is the quality aspect of the Web service of providing confidentiality and non-repudiation by authenticating the parties involved, encrypting messages, and providing access control. Security has added importance because Web service invocation occurs over the public Internet. The service provider can have different approaches and levels of providing security depending on the service requestor.
Transactional QoS refers to the level of reliability and consistency at which the transactions are executed. Transactional QoS is crucial for maintaining the integrity of a Web service. Transactions are very important for business processes to guarantee that a set of related activities are treated and completed as a single unit of work. If an exception occurs while executing a transaction, the transaction has to be recovered back to a consistent state. This property is called the "atomicity" of a transaction. Other than property of atomicity, transactions in a stricter sense should satisfy consistency, isolation and durability properties. All these four properties are together called "ACID" properties
ACID properties of Transactions
- A transaction is an atomic unit of processing; it is either performed in its entirety or not at all.
- A correct execution of the transaction must take the system from one consistent state to another.
- A transaction should not make its updates visible to other transactions until it is committed. That is, it should run as if no other transaction is running.
- Once a transaction commits, the committed changes must never be lost in the event of any failure.
There are new complications when we are thinking of transactions involving Web services. The Web services used by a particular application or Web service are often distributed remotely over the Web as well as owned by different parties. Having a central transaction coordinator, which dictates the rules and performs the commits and rollbacks, in a Web services environment is very tedious to implement considering the transaction coordinator does not have full control over all the resources. Also, two-phase commit protocol involves one or other form of resource locking. Longer periods of resource locking will result in serious scalability issues. Therefore even though it is possible to use, extreme care should be taken to make sure that resources are not locked for long periods of time.
- Transmission Control Protocol - Wikipedia, the free encyclopedia
- Reliable Message Delivery in a Web Services World: A Proposed Architecture and Roadmap
- Web Services Reliable Messaging Protocol
- SOAP vs. REST
- Reliable XML Web Services
- Understanding quality of service for Web services
Please share comments on how this worked for you.
Please “like” this post if it solved a Business issue for you.
Regards, "Smooth Orchestrator"