Integration Technologies

Integration has always been high on the list of concerns for IT professionals, and over the years there have been a number of models and technologies that tried to address these concerns; to ease the pain of integration. The introduction of the Service Oriented Architecture (SOA) model, along with associated technologies such as the emergence of the Enterprise Service Bus (ESB) and Business Process Management (BPM) solutions has both helped create the possibility of more open infrastructures, but also muddied the waters. The problems that now exist are, in the main, due to the coming together of several technologies, and the uncertainty as to where precisely each technology might fit in a larger integration scenario. One clear example of this concerns the aforementioned SOA and the older technology of Web services. To many, there is little (if any) distinction between the two. While there is a great deal of synergy between SOA and Web services, one is not dependent upon the other. Integration has always (or should always) have addressed the issue of why integration is necessary rather than how to carry out integration; at least as the starting point.

The 'why' of integration is concerned with the ability for organisations to operate in a more open and flexible manner. The creation of a total organisational infrastructure; of which the technical infrastructure is only one element (albeit an important one). Past integration solutions have more closely addressed the issues of data and information rather than that of process. Whilst the former two are still clearly important, any integration strategy should not focus upon them to the exclusion of the latter. Integration has now become a much wider concern than in the past. However, opening up the integration requirements to a larger audience has created the possibility of making a reactive and proactive organisation a real possibility.

Another positive aspect of the newer models of integration (such as SOA) is that it is not a 'rip and replace' model; it is firmly entrenched in the re-use of existing assets. SOA is a conceptual architecture, as opposed to an installable technology, and as such fits well with more traditional methods of integration; all of which have relevance in creating the required organisational environment.

Business Issues

As mentioned previously, the key to integration is the question 'why integrate?' If this question is not asked, and not answered in a structured manner, then any integration is bound to fail to a greater or lesser degree. There are several business drivers that make integration increasingly important. Globalisation means that competition could come from anywhere. Integration, both internal and along the supply chain, could make the difference between staying in business and losing out. For example, in many vertical markets, supply chains today tend to be global, with design carried out in the west, but production undertaken in the east - and there can be strong competition at both ends of this chain. There has to be integration, not just of data and information, but also of process to make a global organisation most effective. Legislation is also a strong driver for integration, particularly where there is a need to consolidate information from across many different applications in a very short timescale. This type of integration is often more data and information related rather than application related.

Every IT professional knows that integration is not a facile undertaking, and it is not just technology that puts barriers in the way of successful integration. There are also business aspects that hinder integration. At the root of this is likely to be a lack of understanding of exactly what applications really are in use, and what function they perform. This may seem a surprising statement to make, but often even when there is a good understanding of what applications are in use, it is not always known exactly what function they really perform, as the application itself may have moved on significantly from any documentation that may exist. One clear example of this is when the issue of host systems enter into the picture. In many organisations they have formed the backbone of the transactional systems, and the thought of attempting to integrate them into a larger framework always raises concerns regarding future operability. Butler Group believes there is an issue here that organisations really need to come to terms with. The older host systems appear to have some mythical presence; with complete reliance on the veracity of information within those systems and produced by those systems. These systems have been modified over a number of years, and there is little guarantee that the faith placed in them is now well founded. There is a simple but effective answer to those who would leave host systems alone with the phrase 'if it ain't broke don't fix it'. That answer is 'the chances are it is broken, and it is only when it is brought into the mainstream of an organisational operational environment will the extent of its failings become apparent. Mentioned previously, the importance of integration from a process viewpoint is a clear driver. With many organisations responding positively to questions about the importance to them of streamlining business processes it is perhaps still surprising that relatively few have jumped wholesale into integration projects. An IntegrationWeek survey in the first quarter of 2004 indicated that 90% of financial services executives surveyed considered optimising business processes a key target for 2004. Yet only 49% of them said that application integration projects were on their to do lists for the same year. A disparity that is hard to explain. However, it might be put down to a fear of cost; something that could be overcome with a clearer understanding of modern techniques, such as the implementation of a SOA infrastructure. The idea of a Service Oriented Architecture (SOA) is gaining ground, and is reaching the heights of hyperbole at the moment. What this means from a business perspective is that each application needs to be understood from the point of view of what services it provides to the organisation. Applications then need to be 'deconstructed' into their component services - each of which performs a specific business function. Examples of business services are address lookup, customer credit check, and so on. Within an overall SOA framework, such services can then be re-assembled in such as way as to be much more flexible, futureproofing the organisation. Whilst the evolution of Web services has undoubtedly supported the concept of SOA, it has been around for a number of years, but has only recently become a high profile idea. Integration Technologies www.butlergroup.com 10 Section 1: Management Summary June 2005
Creating an integration strategy based on process as the primary driver (whether or not one wants to call it SOA) will allow organisations to better understand the benefits of spending money on integration. For those more forward looking organisations, this cost element becomes subservient to the fact that the organisation will be future-proofed both in terms of technology advancements and organisation change. Clearly, whatever approach is taken there are prerequisites that need to be taken into account. Firstly, the emphasis has to be on re-use. The cost and upheaval of a 'rip-and-replace' mentality has no place in today's organisations. Secondly, it is imperative to have take a strategic view of integration. From a business perspective this will have to include an understanding of the relationships that are held with external partners, and how they will affect both the internal and external strategy. Lastly, integration has always to be considered as an ongoing task. To this end a framework for integration has to be built. Even without the SOA model, there is a strong driver to couple systems as loosely as possible and this is an obtainable end if the framework approach is taken.

Technology Issues

There are multiple technology issues concerned with integration, and the introduction of newer processbased models should not lead to ignoring some of the more traditional approaches. There is a place within an integration strategy for multiple models to exist side-by-side.

However, as the technology that is highest on the radar at the moment the joint issues of SOAs and ESBs need to be addressed; not least because understanding of these will radically affect the total strategy. A SOA is an architecture model designed to achieve loose coupling between process-based components. The key here is in the word 'Service'. In order to understand a SOA, the definition of a service has to be created, and in order to do that the concept of a process has to be codified.

A process is a series of explicit events that are joined together by a set of rules in a logical fashion in order to either fulfil an expectation (provide a mortgage for a customer, as one example) or to change the state of something (modify information in a structured fashion held within a repository). Each of these explicit events carry out a specific task or function. In the case of a mortgage application, one task would be to retrieve financial information about the prospective client. Another related task would be to take that information and from it apply in-house rules to create a credit-rating; the result of which would then determine the next task or action to be taken.

Loose coupling, which is the underpinning basis of a SOA, leads to concerns; not least of which is that of security. A security model based on tight integration is easier to implement than one based on loosely coupled components. However, there are methods and technologies available to bring security into this loosely coupled and highly distributed world.

Closely allied to SOAs are the concept of the ESB. This is an extension to the bus messaging model, and has subsumed some of the technology that existed as a node on the old bus model into a piece of core functionality. One prime example of this is the use of an integration server that in the past would exist as such a node and would use adapters to carry out required transformations. This function can now exist as part of the ESB. This leads to the issue of precisely what an ESB is. Different vendors would put forward different ideas. There is a basic split between those that promote ESB as an installable product, and those that treat it as an architectural model. There are arguments for both ways of thinking, but at the end of the day, if one considers the functionality of an ESB and applies that to an integration strategy then the arguments become more of an irrelevance.

The functionality of an ESB (whether productised or as an architectural style) has a set of core characteristics, which defines it. These characteristics include:

• Messaging.
• Transformation services.
• Content-based routing.
• Basic connectivity to defined standards (Web services, J2EE Connector Architecture, Java Message Service, etc.).
• Support for highly distributed environments.

Although there are other functional elements that can be implemented within an ESB, the preceding would be considered a minimum 'must have'. The ESB, or at least the idea of an ESB, is so central to many people's understanding of a SOA, we have devoted the Technology Evaluation and Comparison Section of this Report over to that topic. Similarly, we have extended this to look at vendors who also have an ESB product or who talk about an ESB style within the Vendor Profile Section. Although the ESB is not the be-all and end-all of integration, as this Report demonstrates, it is a factor whose inclusion needs to be examined closely. Bringing the ESB (product or model) into the total infrastructure has an effect on host systems, and this needs to be clearly understood. The ESB exists as part of a highly distributed system model, which is the complete antithesis to the basic understanding of the role of host systems. The more traditional role of middleware and associated technologies cannot be ignored. Although there may be those who talk about service or process-based architectures to the exclusion of everything else, a more pragmatic approach is to realise that these architectures will, at least for some time, have to co-exist with other already implemented models. The synergy between emerging technologies and existing ones has been mentioned previously, and nowhere is this more important than when one looks at the role XML has to play. Once promoted as the answer to all the integration problems, it has now found its correct place within the grand scheme of things. This is an important technology, but, as with many other technologies, it is only one part of the complex jigsaw. Much mention has been made of an integration strategy and this needs expanding to provide context. The route chosen for integration will be highly dependent upon both the current implemented infrastructure and the purpose for the integration. Purpose being the keyword; integration should never be carried out for integration's sake. If one is attempting to integrate thousands of applications in a large-scale integration project, it is likely that a different architecture will be selected to that chosen for a small-scale integration project that is attempting to link three or four diverse systems together. Selecting the right topology can make a difference in terms of integration performance, management of unforeseen events, and maintenance costs into the future. Making the decision about which is best for a particular environment will depend on many aspects, such as the number of systems to be connected, the volumes of data to be shared, and whether integration has to be real-time or is time-independent.Again, it is important to understand the total integration scenarios available. It might be considered forward thinking to only consider service-based integration, but there is still a need for some of the more traditional technologies, such as point-to-point integration, data integration, or the creation of a hub-and-spoke model. Although we have stated that integration has to be considered from the point of view of process, this does not deny the requirements for other integration necessities. As the vast majority of organisations can be seen as information based, then the creation of an integrated information model has also to come under serious consideration. The total integrated architecture built will incorporate elements of multiple technologies, each one working within a specific area. This is the major change in integration that has happened in the past few years. No longer are organisations faced with implementing a single model, which to a large extent forced a lowest common denominator architecture onto the organisation.

Market Issues

The integration market is now relatively mature, with some well-established players competing for frequently large budgets. However, there has been much consolidation over recent years, and there is still a lot of pressure on vendors. The first few years of this century have proved very difficult for a number of the well-known players, and only the last year has shown some slight improvement. For many of the newer entrants, there have been some good customer wins, but profits are still in the future. The ESB market has allowed some of the more niche vendors to gain a presence, but how sustainable this will be in the long term is debatable. Given the duplicity of view as to the precise nature of an ESB, it is likely that vendors in this space who do not extend into the larger integration market will fall by the wayside. The market size for integration today is some US$5 billion, according to figures from Datamonitor. Enterprise integration software spend is likely to show modest growth during 2005 of 4.23%, and will then rise over the next three years at a compound average growth of 4.9%. North America is still the most significant market segment geographically. Within Europe, Germany and the UK are the biggest spenders on integration technology. Datamonitor has also investigated revenues in the various different vertical sectors, and an interesting picture is revealed when examining data in this way. Contrasting 2003 with 2008, we see that the highest spending area, financial services, which is certainly one in which many integration vendors place a significant emphasis, actually sees falling revenue projections. The largest likely growth market is manufacturing, and the reason for this is likely to be related to the large spending hike in that sector for Y2K-related issues. This led to a number of years when that sector did not spend significantly on technology, and it is gradually starting to see the benefits of integration and thus expenditure may rise. As with all markets, there will be an 'ebb and flow' of investment in integration. To a large extent this measurement will depend upon the definition one gives to integration. Just as the Business Process Management (BPM) space is being attacked by pure-play integration vendors, so will BPM vendors start to encroach on the 'integration market', especially as organisations move towards the process-centric approach to infrastructure implementation. Butler Group Integration Technologies Market Lifecycle Ratings Butler Group's vendor ranking and assessment model groups suppliers into Outperform, Perform, and Under-perform categories, and shows the predicted progress through the three major market phases of Early Adopter, Market Adoption, and Market Maturity.

Key Findings