Rob Brooks-Bilson

This post started out as an email response to a question Steve Rittler asked me about integrating ColdFusion with an Enterprise Service Bus (ESB). Since it's a fairly interesting topic, I figured it might benefit the community so I asked him if he minded if I posted my response as a series of blog entries. Before I speak to the specifics of connecting ColdFusion to an ESB, I think it might be useful to talk about what an ESB is, how it came about, and what sorts of problems it helps solve.

To give you some background, I've been working for the past two years now as the architect for a project to transform the way our company does integration. We're going from point-to-point integration using mainly shared databases, flat files, journals, and FTP to a service oriented architecture that uses XML messaging and an Enterprise Service Bus as the foundation. To understand how we arrived where we are today, it's useful to understand the evolution of integration patterns. As I said, my company had been doing the majority of its integration point-to-point. Each system was integrated with every other system directly – through database calls, sending flat files over FTP, etc. A typical point-to-point architecture looks something like this:

As you can see, it looks kind of messy – and it is for a large enterprise like mine. If you are a small company, though, chances are this is how your integration looks as well, and that might be fine given your situation. Just like the procedural vs. OOP debate or the framework debate, the integration architecture debate is discussed in similar fashion. That said, the main drawback to the point-to-point architecture is that it's an n² problem. That means for each new system you add to your architecture that needs to be integrated, you end up with a total of n² integration points. If you only have a handful of systems, you may be able to support this, but it can become unmanageable fast. It just doesn't scale well in most cases. Point-to-point integration is popular because it's so easy to do and can usually be implemented fairly quickly. "I'll just write data to this staging file" or "I'll just grab data from this table over here" probably sounds familiar.

This brings us to the next evolution in integration architectures – hub and spoke integration. Hub and spoke integration attempted to address the scalability and management problems of point-to-point architectures by introducing the idea of a central hub or broker. You'll often hear this referred to as EAI or Enterprise Application Integration as well. The idea is that all systems in an organization would connect to and send their data through a central hub. This way, connectivity went from n² connections to n:

Integration is usually performed via adapters for each specific application. These adapters can be as generic as file drops and database queries or API based adapters for connecting to packaged software such as SAP.

In order to integrate a new system in this architecture, all you needed to do is connect it to the central hub and it is then capable of sending data to or receiving data from any other system on the hub. Hub and spoke architectures make for more loosely coupled systems.

While the hub and spoke architecture was great at reducing dependencies as well as the number of connectivity points, it introduced several new problems as well. First, hub and spoke architectures tend to be proprietary in nature. They also introduce a single point of failure that can be difficult to mitigate. Additionally, they have historically have had scalability problems because of an inability to support large transaction volumes.

Because nothing in IT is ever the be-all and end-all solution, integration architecture continued to evolve. Seeking to take the best parts of EAI (centralized management, loose coupling, n lines of connectivity), a new integration pattern known as the Enterprise Service Bus or ESB emerged. Initially, the ESB was only an integration pattern – there were no commercial applications you could buy and implement. Of course, that has all changed. Today, you can find ESB product offerings from pretty much every major integration software vendor out there as well as a growing list of open source alternatives.

There are a number of features and characteristics that make an ESB different from traditional EAI/hub and spoke integration. It's difficult to get everyone to agree 100% on a universal definition of an ESB, but for my purposes, I've used this high level definition from David Chappell's book Enterprise Service Bus: Theory in Practice (O'Reilly):

• A distributed services architecture, which includes a lightweight container model for hosting integration components as remote services
• An enterprise messaging backbone that provides reliable delivery of messages between applications and services
• XML Data transformation
• Service orchestration and intelligent routing of messages based on their content
• A flexible security framework
• A management infrastructure that lets you configure, deploy, monitor, and manage your remote services

The implementation of the ESB looks something like this:

One of the most important aspects here and one that people often don't fully get at first is the distributed nature of the ESB. In hub and spoke EAI, all data has to be brought to the central hub in order to be processed. In an ESB, you actually deploy ESB end-points where your applications are located. If your data center is all in one building, it's as simple as pushing the processing out to other servers located in the same room. In a distributed enterprise, the remote processing means you can physically locate processing components where the systems you need to integrate reside. This makes the ESB much, much more scalable than hub and spoke architectures.

Another thing to note is that the distributed services architecture is implemented using lightweight containers. This means that you don't need to have an application server installed everywhere you want to deploy the ESB containers. That can easily lower your overall integration costs, especially if you use commercial application servers in your environment. All that's required to deploy the remote ESB containers is (usually) a JVM.

A third difference between hub and spoke and the ESB is that the ESB is built on a SOA foundation. Not only does the ESB act as the underlying backbone for SOA, but it is itself SOA based – all of its parts are loosely coupled and can generally be deployed separately as necessary. One benefit of this is that you can deploy only what functionality you need where and when you need it. If you have an intensive transformation process that occurs in one of the applications you are integrating with the bus, you can deploy the necessary horsepower there to deal with the transformation, without having to have the same configuration everywhere.

Applications can connect to ESB endpoints using a variety of mechanisms including but not limited to Web Services, JMS, JCA, HTTP, email, FTP, JDBC, adapters, and many more. How you connect will depend on what you are trying to connect, your overall architecture, and other factors. The bottom line is that there's no one way to do ESB connectivity, although your choice can impact ease of implementation, performance, scalability, etc.

I hope you've enjoyed this general overview of integration architectures. If you have questions or comments, I'd love to hear them. Stay tuned for the next post in this series in which I answer the question "how to get ColdFusion on the Bus".

This one has been a long time coming. Back in 2003, I blogged about how it would be really cool now that ColdFusion was built on top of Java for ColdFusion to support the portlet specifications outlined in JSR 168 & 170. I was really hoping they would release something for ColdFusion that made this as easy as wrapping code with a custom tag, or invoking a CFC. I also heard rumors that Macromedia had created a proof of concept for this at the request of a customer, but nothing was ever released.

As time went on, I shifted away from looking at a commercial portal (WebSphere) as a solution for some problems we were trying to solve, but I still keep an eye on developments in that space, as I never know when things may change and I could find myself looking at portals again.

All of this brings me to the point of this post. It looks like Adobe has released a proof of concept Portlet Toolkit for IBM Websphere and ColdFusion!

From the Adobe Labs website:

"The ColdFusion Portlet Toolkit is a proof-of-concept library that allows you to use ColdFusion as a Portal Application within an IBM WebSphere Portal Server (WSPS version 5 and greater). A Portal Application can serve several different portlets. Every portlet you deploy will run inside the same ColdFusion Portal Application. This toolkit provides one concrete Portlet called CFCPortlet. This Portlet simply passes the requests on to the ColdFusion server. The ColdFusion then invokes CFC methods to render the Portlet, and to perform actions. You can deploy several instances of this Portlet on your Portal Server. The setting cfcName defines which CFC will be used to marshal the Portlet events."

I think this opens up a lot of cool possibilities for ColdFusion developers working at companies who have also deployed Websphere portals. Although this release is Websphere specific, I would think it wouldn't be that difficult for Adobe to port the library for other portal implementations (like Apache Jetspeed). Now I just wish I had a Websphere portal so I could play with this thing!

For the past year now, I've been involved in a large Enterprise project to overhaul the integration infrastructure at my company. We're moving from a legacy based architecture where almost all integration is done point-to-point via custom programming (AS400/RPG) with no real middleware.

The new architecture is about as cutting edge as you can get these days. We've switched from custom programmed batch/ftp/etl to XML messaging that's mainly handled via configuration. We've implemented an Enterprise Service Bus (ESB), and XML database for persistence, JMS to guarantee message delivery, and RosettaNet's RNIF for supplier/customer B2B. All very interesting stuff to work with.

Two books I've found absolutely key to helping understand and architect the solution are Theory In Practice: Enterprise Service Bus (O'Reilly) by David A. Chappell and Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions (Addison Wesley) by Gregor Hohpe, Bobby Wolf, et. al. I simply can't say enough about these books. Regardless of the language(s) you work in, these two books offer clear and concise advice in a very practical format.

Although we haven't done any ColdFusion integration with our ESB yet, it is on the table for next year. I'll be sure to post more info as things progress.