Architecture

From Mnjiwiki

This page provides an overview of considerations for establishing a technical architecture for message based business integration solutions at an enterprise level.

Contents 1 Architectural Goals 2 Integration Styles/Patterns 2.1 Request/Response 2.2 Notifications 3 XML 3.1 XML Schema 3.2 Message Content 3.3 SOAP 4 Solution Components 4.1 Application Interfaces, Adapters, Services 4.1.1 Event Enabled 4.2 Hub/ESB/Broker 4.3 Transport 4.3.1 MOM 4.3.2 HTTP 4.3.3 MOM vs. HTTP 4.4 Transformation 4.4.1 XSLT 4.5 Routing/Distribution 5 Security 5.1 User Authentication 5.2 Application Authentication 5.3 Network Security 5.4 Transport Security 6 Support 6.1 Exception Handling 6.2 System Management 6.3 Documentation

Architectural Goals

An Integration Architecture should be:

• Service Oriented – Services are well-defined, self-contained, and do not depend on the context or state of other services. A Web Service is an example of a service oriented technology.
• Loosely Coupled - Participating applications are able to perform their functions without intimate knowledge or dependencies on other applications.
• Recoverable – Failures within the integration architecture cannot result in loss of data.
• Scalable – Expanding capacity should not require application changes.
• Extendable - Reusable components and layered design are required to support new requirements and applications without extensive rework or expense.
• Secure – Ensure that information exchanges are available only to authorized applications and protected from deviant behavior.
• Supportable – Proper documentation and monitoring processes must exist so maintenance, debugging and problem analysis and resolution efforts are efficient.

Integration Styles/Patterns

There are two primary integration styles or patterns used for message based integration.

1. Request/Response
2. Notifications

For all styles, a service provider and at least one consumer are required to complete the exchange of information.

Request/Response

A Request/Response integration process is sometimes described as a pull. An application initiates a request for information and waits for a response. Often, a user is waiting for the response to be displayed on a user interface so they can act on the information.

Request/Response figure

Notifications

Notifications are sometimes described as a push. Publish/Subscribe is another common categorization. Notifications are generated by a service provider in response to business events. In some situations, the service provider may not even be aware of the consuming application(s).

Below is a diagram illustrating notification routing/distribution. This diagram provides a simple illustration of notification routing. Distribution rules can be static, or they can be more dynamic using other services, tables or subscription lists to determine routing requirements. True publish/subscribe implementations provide consumers the ability to activate or de-active subscriptions as needed.

Notification figure

XML

XML messages provide context to data being exchanged as part of an integration process. The service provider generating a given message is responsible for defining and publishing the message format. As examples, the Court is responsible for CourtXML and CriMNet is responsible for the messages associated with the statute service.

XML Schema

XML messages should be supported by an XML schema. Namespaces should be leveraged to distinguish common element names that may be included in more than one message domain. As an example, it is likely that "PersonName" could exist in more than one schema. Namespaces provide a way to distinguish between PersonName from Corrections and PersonName from the Prosecutor.

Namespaces, schema versioning and related XML standards should be established early during an integration project or program.

• More on namespaces
• State Court schema versioning

While XML is the preferred message format, there may be some instances where the effort is prohibitive to format messages as XML. Those situations should be reviewed on a cases by case basis to arrive at an appropriate solution.

Message Content

Consideration should be given to designing messages so that a single message satisfies a superset of consumers. In other words, more is better. As an example, if there are three consumers of a given service, a single message should be designed to satisfy all three consumers instead of designing three different messages.

This design approach may not always be possible due to security and other considerations. When it is not possible to design a single message format to satisfy all consumers, other design options should be explored. As an example, it may be possible to implement an additional layer of software to aggregate multiple small messages into something that represents a complete business transaction for the consuming applications.

In addition to providing data, messages should provide context about the business event that resulted in the message. A standardized message header should be defined and include the action that generated the message. Actions in CRUD terms include:

• Create – a new event/person was created
• Update – something existing was updated
• Delete – something existing was deleted

SOAP

Messages conforming to SOAP standards should be used, especially for Web Services.

Solution Components

Application Interfaces, Adapters, Services

Historically, an interface was a program used to exchange data between two or more systems or programs. Sometimes a single program provided the interface between systems and sometimes multiple interface programs were required. This description can also be applied to the terms adapter and service, depending on the context.

Coinciding with the introduction of EAI (Enterprise Application Integration) products, the term adapter earned favor over interface, most likely due to promotion by software vendors.

Many vendor product offerings categorize adapters as technology adapters and application adapters. Technology adapters are offered, as an example, to provide connectivity to a certain database product or transaction monitor such as CICS. Application adapters are generally only offered for larger commercial software packages, such as SAP and PeopleSoft.

The term service was introduced more recently and has become popular with the broad adoption of Web Service technologies.

While the terms interface, adapter and service are often used synonomously, integration purest's are likely to offer the following distinctions:

• Interface is a dated term associated with batch oriented processes. Files or database tables are used as the container for packaging data. FTP is a common transport, or data is exchanged through a common database or file directory.
• Adapters are associated with event enabled processes or a connector for a specific technology. Most adapters do not leverage standards such as SOAP, WSDL, etc..
• Services are event enabled and leverage industry standards. Services are well-defined, self-contained, and do not depend on the context or state of other services. A Web Service is an example of a service oriented technology.

Regardless of your terminolgy preference, a key requirement of most current integration projects is that data exchanges are completed in a timely fashion, close to real-time. Providing timely integration solutions require that integration services are event enabled, meaning they are initiated in response to completion of a certain business event.

Event Enabled

Event enabling integration processes can be a significant challenge for certain applications. Four primary approaches to event enable applications are discussed below.

1. Database Triggers
2. User interface controls (Send button)
3. Transaction logs
4. Configurable event logs

All offer advantages and disadvantages.

• Database triggers, if implemented properly, are generally considered non-intrusive. However, it can sometimes be difficult to map a database event to a business event.

• User interface controls require a screen level change to the application and business process changes. They are, however, much easier to correspond to completion of a business event.

• Leveraging existing transaction logs provides a low cost mechanism for identifying completion of system events. Many applications offer some level of transaction logging, but few offer the flexibility to configure which events are logged and which are not.

• The ability to configure when and which events are logged for visibility outside the application is the preferred design. This generally requires significant redesign of an application or at minimum, a significant level of modification to the data access layer of an application, if one exists.

Hub/ESB/Broker

Hub, ESB (Enterprise Service Bus) and Message Broker type products are designed to address the specialized needs of integration projects. Most products provide transformation and distribution services, as well as support multiple transports, platforms and languages. Web Service and related standards have been embraced by leading vendors. Some products provide tools to monitor and identify problems, including multi-step and long running processes that may include both system and human interfaces.

While these specialized products can be very powerful and offer many advantages, they generally require a significant commitment of time, resources and funding before promised returns are recognized. Return on investment is not likely until the product has been widely adopted within an organization and standardized implementation patterns are established and have evolved to maturity, including utility functions for auditing, logging and exception handling. It should also be recognized that all tools have limitations. Learning how to exploit product strengths and work around weaknesses can be a long process, but is essential to long term success.

Despite the long learning curve and early expenses, Hub, ESB and Message Broker products should be considered when establishing technology standards. Organizations that are flexible and committed to standardized solution platforms will see a payoff in the long run with reduced development cycles and quality deliverables.

Transport

A transport technology is required to deliver messages between applications. Because our focus is message based integration, FTP is not addressed. Instead we will address:

• Message Oriented Middleware (MOM)
• HTTP

MOM

Message-oriented middleware (MOM) is software that resides on both consumer and provider portions of an integration process and typically supports asynchronous calls between consumer and provider applications. Message queues provide temporary storage when the destination program is busy or not connected. MOM increases the flexibility of an architecture by enabling applications to exchange messages with other programs without having to know what platform or processor the other application resides on within the network.

MOM transport services are provided through proprietary product offerings such as WebSphere MQ, MSMQ and SonicMQ, to name a few.

The JMS standard (Java Messaging Service) should be considered for Java applications using MOM. JMS offers an interface standard that is independent of the transport product. In other words, leveraging JMS would make it easier if an organization using WebSphere MQ decided to implement a different MOM product.

HTTP

HTTP is free and available to anyone with internet access. It provides a reliable transport for the synchronous exchange of messages.

MOM vs. HTTP

General guidelines for when to use MOM vs. HTTP are:

• Use MOM for notifications.
• Use HTTP for request/response.

These guidelines leverage the strength of each transport, i.e., notifications being asynchronous and request/response generally being synchronous.

MOM is generally considered more secure than HTTP, but both offer higher levels of security if needed. The more secure options for both transports leverage Secure Socket Layers (SSL), which encrypt the message content so it cannot be read without the proper security key.

Transformation

Transformations are necessary when the data from a service provider cannot be consumed as provided. A simple example is if a gender value of Male needs to be transformed to the value 1.

There are a variety of options, opinions and preferences for how to approach transformation logic and where it should reside. Two of the more popular approaches are:

1. Centralize transformations in a Broker product.
2. Transformations should be completed by adapters/services.

Both options offer advantages and disadvantages.

XSLT

Extensible Style sheet Language Transformations should be considered for implementing transformations either within a Broker product or within an adapter/service. XSLT provides a powerful language for transformations that is standards based, vendor neutral and portable between products and platforms.

Routing/Distribution

As indicated previously, delivery of notification messages can be accomplished using a dynamic, topic based publish/subscribe product. Static routing rules are also an option. In both models, the consuming application is responsible for determining if a specific message is of interest. The provider simply makes the data available through in a published format, through an established propotcol and conforming to security requirements.

Security

Security for message based integration should be addressed through several layers:

• User Authentication
• Application Authentication
• Network Security
• Transport Security

User Authentication

Application systems are responsible for user authentication. Integration processes should trust the user authentication completed by partner applications. As an example, assume a user has logged in to application X and retrieves data from application Y using a Web Service. Application Y should trust the authentication by application X and not require a second user authentication process.

Application Authentication

Integration processes should authenticate at the application level. Security credentials should be assigned to application systems and validated as part of the integration process. This is simlar to common database security models.

Network Security

Firewalls and other established security measures also protect integration processes. Limiting access points through network firewalls can be accomplished by using proxy type devices/services.

Transport Security

Use of Secure Socket Layers (SSL), which encrypts the content of a message so it cannot be read without the proper security key, should be required when a firewall is crossed. Use of SSL inside an organizations core network is also an option.

Message Oriented Middleware products, such as MQ, provide another layer of transport security.

Support

Exception Handling

Exception handling processes are very important when developing software solutions. Most applications are designed to include a file or table that is used as a means of troubleshooting problems and potential problems.

Integration between applications introduces another layer of complexity to debugging, particularly for notifications where the existence of a problem may not be identified for an extended period of time. Without a consistent exception handling process that extends from end-to-end of an integration process, it is necessary for support staff to understand the exception handling processes of each application involved in an integration solution. This can result in a cumbersome, time consuming process of checking log files and tables on various platforms.

In addition to infrastructure monitoring tools, consideration should be given to archiving critical messages to aid in debugging and allow for manual reprocessing in emergency suituations.

System Management

In addition to a consistent exception handing process to assist in problem resolution, it is important to implement tools that help identify problems. Ideally, problems should be identified before they are reported by a user. This is particulary important with notification processes and processes that update outside applications. At minimum, a dash-board showing the health of each integration process should be checked on a periodic basis. Automation to identify infrastructure and application problems and contact support staffed is suggested. While complex to implement, system management processes should strive to provide end-to-end traceability for integration processes.

Documentation

Integration infrastructure and services should be documented at the appropriate level. Documentation formats should be consistent and should satisfy the following needs to provide reuse and supportability.

General Requirements

• Written for the appropriate audience
• Searchable – easy to navigate and find what you are looking for
• Intuitive to use – no training required to find documentation
• Versioning without clutter – archive drafts and outdated versions
• Secure – production data may be included
• Documentation may need to be accessible outside of the organization
• Free – no purchase/licensing required

Definition/Design

• A Registry or Catalog of services should be established that will allow analysts, designers and developers to research existing services that are available.
• Specifications, schemas, WSDL files, etc. should be available for download and review to any authorized staff.
• Procedures and contacts for potential service consumers should be documented and easily accessible.

Infrastructure

• The target audience is those staff responsible for supporting the production environment.
• Infrastructure documentation should address not only software services and debugging tools, but also the physical environment; hardware, URLs, ports, queues, channels, etc.