SOA Governance

CATALYST

Service Oriented Architecture (SOA) is becoming widely deployed as a means of constructing an agile IT environment that can keep up with the accelerating rate of change demanded by the business world. It supports several higher-level initiatives such as Business Process Management and a number of upcoming forms of Web 2.0 collaboration. As such, it has (or should have) high business visibility, and should be seen as a collaborative business and IT issue rather than just another IT platform. In order to fully exploit the potential of SOA and avoid it falling into disrepair over years of iterative deployments, governance of the environment needs to be implemented early with the understanding and endorsement of business stakeholders as well as IT executives.

ANALYSIS

Introduction

The purpose of SOA governance is to ensure that the investments in SOA deployments made by the organisation deliver the anticipated benefits both in the short term and throughout their entire lifetime. SOA is intended to be a long-term architecture, so it is reasonable to expect that some of the resources being deployed now will be in use for decades. The essence of SOA is that there should be just one way of carrying out each automated business activity, and that this should be provided by a service that delivers the appropriate functionality. The service will be used in different contexts and by different service consumers, but will remain autonomous from outside influences, and therefore unaffected by most changes. However, it is to be expected that over time, services will be required to accommodate new functionality or to exhibit altered behaviour as business needs change. It is important that these changes are managed to avoid the proliferation of multiple versions of the same service, and to avoid the creation of functionally-similar services. SOA governance therefore entails design-time procedures and best practices that deliver services with the best business fit, and with due consideration to the reuse of services by subsequent projects. It also requires governance of the SOA environment at runtime to ensure operational requirements are being met, and to impose security, compliance, and other constraints on the activities being conducted. It additionally provides change management procedures that ensure quality is retained across many iterations of altered requirements. Most importantly, SOA governance includes the initial considerations of establishing an appropriate corporate structure and placing the right individuals in the right roles to ensure that the governance effort is implemented and enforced in a cost-effective manner. While many of these tasks are procedural and more concerned with the activities of humans, there is also a need for technology to enable, simplify, and enforce the procedures and policies that have been put in place.

Business Issues

SOA exists because traditional IT architectures have proved to be inadequate in terms of responding to business change. The industries that have led the migration to SOA tend to be those that are currently experiencing the most extreme change, such as the adoption of Third Generation (3G) mobile phone technology in the telecommunications industry, and the convergence across the financial services industry. The greater the rate of change, the greater is the demand for SOA. It should not be thought, however, that an organisation that is not planning any fundamental changes need not consider SOA. External factors that can have an unplanned but very real impact on change include the change of regulatory requirements, changing expectations of business partners and customers, and the potential of an aggressive acquisition.

Even when no immediate changes are planned, a wise organisation will recognise this as an interlude in which to prepare to the next period of change. This is a good time to invest in the infrastructure that will allow the organisation to prosper when future changes catch its competitors unready. It follows that the implementation of SOA needs to be carried out as a collaborative project between IT and the business that it supports. While most of what has been written about SOA discusses the technology aspects, in fact it is far more important that the architectural design of the services receives the investment it needs, not just in monetary terms, but in terms of business executives taking the time to make themselves and their staff available to ensure that the service design aggregates functionality in the way that will provide the best reuse of the development work by subsequent phases of deployment. For each logical component of the architecture there must be a recognised owner. For the active components (such as processes, composite applications, business rules, policies, and the services themselves) it is preferable that the owner should bridge the divide between the IT and business worlds, and preferably would be a business-focused person with good knowledge of IT.

The issue of ownership can be a major stumbling-block. Because services are designed to be reused in many situations, SOA is an uncomfortable fit with organisations that hang onto a 'silo' structure where there is significant functional redundancy across the silos. The general trend in an organisation towards the specialisation of business functionality and the abandonment of the rigid and hierarchical business silos is a good indication that it is ready for SOA. Unfortunately, many organisations (in fact the majority) first conduct an IT-driven pilot project as a proof of concept. As a general rule, these projects are successful and lead to a larger deployment. While this is understandable, certain expectations should be set. The IT-led project is unlikely to achieve the degree of reuse that should be expected of SOA because the service design and overall architecture will have been designed to meet IT requirements rather than business requirements. A typical scenario is the use of SOA to meet the requirement for application integration. The 'services' in this case are likely to be the result of bottom-up, functional analysis rather than top-down, requirements analysis, and will probably have an inappropriate level of granularity. This might be a suitable testing ground for the supporting technology, but organisations should be prepared to scrap this early IT-led work if the service design is inappropriate.

A further complication that often arises in large enterprises is for different business units to embark on SOA through separate initiatives, often being unaware of other work being carried out from which they could benefit. The ability to 'retrofit' governance across a number of established SOA projects can be a test of executive skills. The one common factor across the majority of organisations that have non-trivial SOA deployments is a wish that they had embarked on SOA governance at an earlier stage. Most organisations find that it is helpful to establish a 'SOA Centre of Excellence' that is made up of staff from all of the business units, and with representatives for each of the different disciplines required. This serves both to give a degree of ownership to the business units and to create a pool of knowledge that can be leveraged by further projects. A final business-related point is that the governance initiative will put in place a number of procedures that will help to ensure adherence to all the best practices. However, these procedures must be enforced. We hear of several instances where the procedures have been put in place, but have been ignored or shortcut (often because of the traditional pressures to deliver on-time and on-budget). This has resulted in missing documentation, with service functionality deployed that no longer matches the service description - with inevitable confusion being the outcome. In the worst case, ignoring the procedures could result in services being deployed that have not been rigorously tested, resulting in poor service levels, unhappy customers, and an understandable but inappropriate loss of faith in SOA.

Technology Issues

Technology plays a strong supporting role in SOA governance, and some technology will be an absolute requirement. However, it is undesirable to select and implement SOA governance technology until there is a good understanding of the SOA strategy and the specific requirements of SOA governance. A possible exception to this general rule is in the situation where SOA has been deployed with little or no governance effort. In this situation a first activity should be to implement runtime monitoring software that can discover what services are being used and how they are being consumed. This can take place alongside the evolution of the initial governance plans, since little can be done unless there is a sound understanding of what services need to be managed and what the current usage patterns are. Once the service discovery is in progress there will be a requirement to establish a registry of services and a repository for all of the metadata that will be generated. Depending on the solution chosen, these might be two different products or a single, combined registry/repository.

Governance in the design and development phase will be simplified if modelling and development tools are selected that integrate closely with the registry and repository, automatically storing new and changed service details that impact the way in which the service is used in the registry, and maintaining all the 'where used' information in the repository. Technology could also be implemented that imposes a customisable change approval process, to avoid the problems associated with short-cutting the established procedures.

At face value, IT will be most comfortable with the runtime governance of SOA, since it is a logical continuation from business service management and IT service management requirements. However, the manner of implementation is different. Since the SOA runtime environment is inherently message-based, runtime governance is applied primarily at the message level. Particular attention needs to be applied to the security implications of SOA, since in addition to the security requirements of the applications underlying the services, there is rich value contained in the messages themselves. The problem of security is highlighted in the case of the use of external services or the provision of services to customers or partners. However, even 'within the firewall' there is a requirement to evaluate the risk to the business and adopt appropriate security technology solutions.

It is not a good idea to implement maximum security solutions unless there is a strong business need, since there is a performance implication of providing strong encryption and authentication. Performance issues can be ameliorated by good design and by the use of appropriate technologies, including the use of network appliances as an alternative to server-based software solutions. Runtime governance technologies support the fundamental requirement to abstract variable logic from more stable business logic that can be hard-coded into services. Business policies that can be abstracted include Service Level Agreements (SLAs), security policies, legislative compliance rules, and industry best practices. By removing the implementation of these policies from hard-coded services into a more dynamic environment, the long-term stability of the SOA deployment is enhanced along with the ability for the business to alter its policies in a more cost-effective manner.

Market Issues

The SOA technology market is evolving rapidly, with major full-platform vendors acquiring niche solution specialists in order to build-out a complete offering and to be able to derive additional income from the existing user base. However, it is a fact of life that most large organisations will be faced with applying governance across a heterogeneous SOA environment, and so most governance products try to avoid becoming too locked-in to a specific platform. Any full SOA governance solution is going to involve multiple products, which may or may not be bundled into a single suite. The product types that have the greatest significance in this context can be summarised as: Service Registry: This provides the system of record for determining what services exist, what functionality they enable, where they can be found, and how to interact with them. The registry is used in both the design and development phase, and the runtime phase.

Metadata Repository: This provides a vehicle for storing all of the metadata describing the many artefacts in the SOA environment. This will include version information, cross-dependencies between services, processes, rules, routing, and message transformations. Governance of SOA entails keeping metadata complete, current, and easily available. Over time this will raise the profile of the repository to a 'must have'. 'Passive' Runtime Governance: This category provides visibility into what is going on in the SOA environment, both in real time and as aggregated historical information. It includes usage analysis and performance analysis that can be used in planning the physical environment to cater for growth. These products generally also provide threshold alerting capabilities so that any deviation of runtime performance or usage can be investigated and corrected. 'Active' Runtime Governance: These products have the ability to dynamically alter the execution environment so as to enforce business policies. Most widely deployed are security-related products, which can reject malformed messages, encrypt sensitive information, ensure the authentication of service requestors, and redirect messages to alternative services. Other policies can be implemented in a similar way, such as the use of SLA policies to identify the most important transactions and to change the message priorities, reroute messages to high-priority services, or to throttle back the resources given to low-priority work.

Conclusions

Any organisation that is sufficiently mature to have a formal initiative for business governance and/or IT governance will discover that SOA comes attached to considerations that require a deliberate approach to the governance of SOA also. Early adopters of SOA have tended to implement SOA governance as a subset of IT governance; but as described later in this Report, SOA governance really provides a linkage between the governance efforts of business and IT. As such, it should be resourced in a collaborative manner. As SOA deployments become more mature the initial technology focus will inevitably change towards a business focus. This will be a more natural transition if business participation in SOA can be encouraged from the outset.

The reality is that most organisations will be faced with the situation where a SOA pilot project followed by several iterations of small-scale SOA deployments will have been driven largely by IT, and with little consideration for governance in the long term. Retrofitting governance to this established environment can be a painful process that might involve discarding some of the investments already made. At the same time, it is not recommended that an organisation should embark on a large-scale SOA governance implementation before any real benefits have been achieved.

The solution has to be something of a compromise. An organisation should be realistic about all the requirements and costs of SOA governance, and the risks that are associated with the lack of governance. It should then determine the appropriate level of investment in governance commensurate with the benefits and risks of the current projects. The objective should be to keep the investment in SOA governance just ahead of the demands of the current level of SOA deployment. To achieve this needs a holistic view of the ultimate structure, and a planned approach to investment and deployment.