The Evolution of Integration Architecture: An Introduction to the Enterprise Service Bus (ESB)

This post started out as an email response to a question Steve Rittler asked me about integrating ColdFusion with an Enterprise Service Bus (ESB). Since it's a fairly interesting topic, I figured it might benefit the community so I asked him if he minded if I posted my response as a series of blog entries. Before I speak to the specifics of connecting ColdFusion to an ESB, I think it might be useful to talk about what an ESB is, how it came about, and what sorts of problems it helps solve.

To give you some background, I've been working for the past two years now as the architect for a project to transform the way our company does integration. We're going from point-to-point integration using mainly shared databases, flat files, journals, and FTP to a service oriented architecture that uses XML messaging and an Enterprise Service Bus as the foundation. To understand how we arrived where we are today, it's useful to understand the evolution of integration patterns. As I said, my company had been doing the majority of its integration point-to-point. Each system was integrated with every other system directly – through database calls, sending flat files over FTP, etc. A typical point-to-point architecture looks something like this:

As you can see, it looks kind of messy – and it is for a large enterprise like mine. If you are a small company, though, chances are this is how your integration looks as well, and that might be fine given your situation. Just like the procedural vs. OOP debate or the framework debate, the integration architecture debate is discussed in similar fashion. That said, the main drawback to the point-to-point architecture is that it's an n2 problem. That means for each new system you add to your architecture that needs to be integrated, you end up with a total of n2 integration points. If you only have a handful of systems, you may be able to support this, but it can become unmanageable fast. It just doesn't scale well in most cases. Point-to-point integration is popular because it's so easy to do and can usually be implemented fairly quickly. "I'll just write data to this staging file" or "I'll just grab data from this table over here" probably sounds familiar.

This brings us to the next evolution in integration architectures – hub and spoke integration. Hub and spoke integration attempted to address the scalability and management problems of point-to-point architectures by introducing the idea of a central hub or broker. You'll often hear this referred to as EAI or Enterprise Application Integration as well. The idea is that all systems in an organization would connect to and send their data through a central hub. This way, connectivity went from n2 connections to n:

Integration is usually performed via adapters for each specific application. These adapters can be as generic as file drops and database queries or API based adapters for connecting to packaged software such as SAP.

In order to integrate a new system in this architecture, all you needed to do is connect it to the central hub and it is then capable of sending data to or receiving data from any other system on the hub. Hub and spoke architectures make for more loosely coupled systems.

While the hub and spoke architecture was great at reducing dependencies as well as the number of connectivity points, it introduced several new problems as well. First, hub and spoke architectures tend to be proprietary in nature. They also introduce a single point of failure that can be difficult to mitigate. Additionally, they have historically have had scalability problems because of an inability to support large transaction volumes.

Because nothing in IT is ever the be-all and end-all solution, integration architecture continued to evolve. Seeking to take the best parts of EAI (centralized management, loose coupling, n lines of connectivity), a new integration pattern known as the Enterprise Service Bus or ESB emerged. Initially, the ESB was only an integration pattern – there were no commercial applications you could buy and implement. Of course, that has all changed. Today, you can find ESB product offerings from pretty much every major integration software vendor out there as well as a growing list of open source alternatives.

There are a number of features and characteristics that make an ESB different from traditional EAI/hub and spoke integration. It's difficult to get everyone to agree 100% on a universal definition of an ESB, but for my purposes, I've used this high level definition from David Chappell's book Enterprise Service Bus: Theory in Practice (O'Reilly):

  • A distributed services architecture, which includes a lightweight container model for hosting integration components as remote services
  • An enterprise messaging backbone that provides reliable delivery of messages between applications and services
  • XML Data transformation
  • Service orchestration and intelligent routing of messages based on their content
  • A flexible security framework
  • A management infrastructure that lets you configure, deploy, monitor, and manage your remote services

The implementation of the ESB looks something like this:

One of the most important aspects here and one that people often don't fully get at first is the distributed nature of the ESB. In hub and spoke EAI, all data has to be brought to the central hub in order to be processed. In an ESB, you actually deploy ESB end-points where your applications are located. If your data center is all in one building, it's as simple as pushing the processing out to other servers located in the same room. In a distributed enterprise, the remote processing means you can physically locate processing components where the systems you need to integrate reside. This makes the ESB much, much more scalable than hub and spoke architectures.

Another thing to note is that the distributed services architecture is implemented using lightweight containers. This means that you don't need to have an application server installed everywhere you want to deploy the ESB containers. That can easily lower your overall integration costs, especially if you use commercial application servers in your environment. All that's required to deploy the remote ESB containers is (usually) a JVM.

A third difference between hub and spoke and the ESB is that the ESB is built on a SOA foundation. Not only does the ESB act as the underlying backbone for SOA, but it is itself SOA based – all of its parts are loosely coupled and can generally be deployed separately as necessary. One benefit of this is that you can deploy only what functionality you need where and when you need it. If you have an intensive transformation process that occurs in one of the applications you are integrating with the bus, you can deploy the necessary horsepower there to deal with the transformation, without having to have the same configuration everywhere.

Applications can connect to ESB endpoints using a variety of mechanisms including but not limited to Web Services, JMS, JCA, HTTP, email, FTP, JDBC, adapters, and many more. How you connect will depend on what you are trying to connect, your overall architecture, and other factors. The bottom line is that there's no one way to do ESB connectivity, although your choice can impact ease of implementation, performance, scalability, etc.

I hope you've enjoyed this general overview of integration architectures. If you have questions or comments, I'd love to hear them. Stay tuned for the next post in this series in which I answer the question "how to get ColdFusion on the Bus".

Jim Priest on CFEclipse at the IBM Developer Works Site

Kudos to Jim Priest for his latest CFEclipse article published on the IBM Developer Works website.

Jim's an avid blogger and regular contributor to the ColdFusion community. Getting an article on the IBM Developer Works site is no small feat, and an article there about ColdFusion is just one more feather in the cap of of CF developers. Cheers to Jim! This is the kind of exposure outside of the CF mainstream that we can all benefit from.

Pete Thompson on Microsoft's Surface Computing

I'm out in San Jose this week at the Microsoft Global High Tech Summit (I was part of a team that authored a whitepaper on secure collaboration for the conference - more on that later). I'm currently sitting in on a session by Pete Thompson, the GM for Microsoft's Surface Computing platform. I had a chance to talk with Pete a bit last night at a cocktail reception and have to say that Microsoft has some fascinating work going on around the space. The team is currently up to 170 people, with plans to grow it to 250. That's a lot of people working on a very innovative platform.

If you aren't familiar with surface computing, it's essentially using sensing and display technology embedded in everyday surfaces (like tables) to interact with digital content. Microsoft's first product is called appropriately enough, Surface. It has some distinct characteristics, as defined by Microsoft:

  • direct interaction
  • multi-touch (currently can handle 52 simultaneous - think poker)
  • multi-user
  • object recognition

Pete feels that the killer feature is object recognition - things like painting on the table with a brush, placing a camera or cellphone on the table and having it recognized and interacting with it.

Essentially it works like a rear projection TV flipped on it's back with an embedded (custom) version of Vista. It has several cameras mounted underneath along with an IR light source. There were 80+ prototypes before the final design was finalized.

Right now, Surface is being rolled out at hotels and restaurants around the country. The idea is to get it out there for people to start interacting with before it's fully commercialized (like plasma tv's were). Microsoft is working with an Harrah's in Las Vegas on a Surface project (I'm wondering if it's an interactive card game)??

Microsoft is treating Surface as a platform, and plans to release an SDK for developing custom solutions (late '07 / early '08). There's already a lot of interest in both the platform and product, with a consumer version of the product eventually to be developed.

Having a chance to have played with the Surface table being demonstrated at the conference, I have to say that the technology really is exciting, and I can see a lot of potential for the platform. The demo apps they have really only scratch the surface of what's going to be possible once the SDK is out there and people start developing for the platform. Interaction via touch, visual recognition, RFID, wireless, and more.

Right now, Surface is built into a table. Wait until it's built into other objects like walls, floors, refridgerators, etc. Sure it's going to take time to get the components down to a size to make this possible, but they will get there, and when they do, look out.

A Quick Note for Canvas Wiki Users: Handling Large Text Extries

This is just a quick note to Canvas Wiki users regarding large text entries. I was putting our internal coding guidelines into our dev wiki today when I ran into an issue with the entry being truncated. My initial thought was either a validation rule in the app was truncating the text, or the DSN configuration wasn't set to allow CLOBS. After a quick peek at the code, I confirmed the issue was with the DSN config. You'll never run into this issue unless you have a fairly large wiki entry.

To correct the issue, edit your data source in the ColdFusion Administrator and click on the Show Advanced Settings button. You should see a checkbox for CLOB -- Enable long text retrieval (CLOB). Make sure this is checked and you should be good to go.

Some Notes on Using Epoch Time in ColdFusion

I've seen a lot of postings on blogs and mailing lists lately showing various functions for calculating/displaying epoch time using ColdFusion.

There are actually two popular variants of epoch - UNIX epoch and Java epoch. UNIX epoch or "UNIX time" is defined by Wikipedia as:

"Unix time, or POSIX time, is a system for describing points in time: it is the number of seconds elapsed since midnight UTC of January 1, 1970, not counting leap seconds. It is widely used not only on Unix-like operating systems but also in many other computing systems. It is neither a linear representation of time nor a true representation of UTC (though it is frequently mistaken for both) as the times it represents are UTC but it has no way of representing UTC leap seconds (e.g. 1998-12-31 23:59:60)."

As you can see, UNIX epoch is expressed as the number of seconds since midnight UTC on January 1, 1970. Java epoch is still based on the same origination date, however, it is expressed in milliseconds.

There are several ways you can get the UNIX/Java epoch time in ColdFusion including using Java, the getTickCount() function, or using dateDiff().

What's important to note here is that the number of seconds/milliseconds returned depends on the method you use and your server's timezone. Both the Java method and getTickCount() return values that are relative to your server's local time, not UTC, which may or may not be how you need it within your application. The dateDiff() function returns the number of seconds relative to UCT time, not your server's local time.

Let's take a look at a few examples to help clarify my point. The following code uses the three methods above. I've written a udf to wrap the dateDiff() function because I want a reusable way to package things and I also want to account for the server's UTC offset using the dateConvert() function:

view plain print about
1<cfscript>
2/**
3 * Returns the number of seconds since UTC January 1, 1970, 00:00:00 (Epoch time).
4 *
5 * @param DateTime      Date/time object you want converted to Epoch time. (Required)
6 * @return Returns a numeric value.
7 * @author Rob Brooks-Bilson (rbils@amkor.com)
8 * @version 1, June 21, 2002
9 */

10function GetEpochTimeFromLocal() {
11 var datetime = 0;
12 if ArrayLenn(Arguments) eq 0) {
13 datetime = Now();
14 }
15 else {
16 datetime = arguments[1];
17 }
18 return DateDiff("s", DateConvert("utc2Local", "January 1 1970 00:00"), datetime);
19}
20
</cfscript>
21
22<!--- get epoch using getTickCount(). Note that result is in milliseconds --->
23<cfset x=getTickCount()>
24
25<!--- get epoch using java. Note that result is in milliseconds --->
26<cfset javaEpoch=now()>
27
28<table>
29<cfoutput>
30<tr>
31    <td>Java:</td>
32    <td>#int(javaEpoch.getTime()/1000)#</td>
33</tr>
34<tr>
35    <td>getTickCount():</td>
36    <td>#int(x/1000)#</td>
37</tr>    
38<tr>
39    <td>udf::</td>
40    <td> #GetEpochTimeFromLocal()#</td>
41</tr>
42</cfoutput>
43</table>

If you want to go from epoch to local ColdFusion server date/time, add the following code to the example:

view plain print about
1<cfset epochToLocal = DateAdd("s",getTick,DateConvert("utc2Local", "January 1 1970 00:00"))>
2
3<cfoutput>
4From epoch to date/time: #dateFormat(epochToLocal, 'mm/dd/yyyy')# #timeFormat(epochToLocal)#
5</cfoutput>

You should note that the getTickCount() only started returning Java epoch time as of ColdFusion MX 6.1. Prior to that, it returned a sequential number, but it wasn't epoch.

Here are two links to articles I wrote for the ColdFusion Cookbook that give further clarification:

Convert a date/time to epoch
convert epoch to a CF date/time object

Additionally, there are four UDFs on cflib for converting to/from epoch:

epochTimeToDate()
epochTimeToLocalDate()
getEpochTime()
getEpochTimeFromLocal
millisecondsToDate()
convertActiveDirectoryTime

CFSnippets Snip-a-Thon 2007: Get your Entries in Now

This one has been making the rounds around the ColdFusion blogosphere, so I thought I'd add to the mix and help get the word out. Todd Sharp is running a nice contest over at the new CFSnippets website designed to get developers working with the new Snipex features in CFEclipse. If you haven't been to CFSnippets, it's a public Snipex server with several libraries of open source snippets you can use within CFEclipse to speed up development of common tasks.

The contest runs through November 30th and includes some pretty cool prizes. For more information, click the banner below.

Update: ColdFusion 8 BSOD Issue - Resolution!

In a previous entry I talked about a problem I was having with my laptop blue screening every time I tried to install/run ColdFusion 8 with the default 1.6 jvm. At the time, I believed the issue might be with java 1.6 and some drivers on my machine. It turns out I was only partially right.

I though others in my group had been successful at installing CF8 on their machines, but it turns out that they all had the same BSOD problem I did. After a little more digging, Adam Crump was able to determine that the fault was taking place with a file called fslx.sys. It turns out that this file is used by the Altiris Software Virtualization System that was installed on all of our laptops. We use Altiris for lifecycle management, helpdesk, patch management, and more. The software virtualization wasn't being actively used, but something about it is causing the BSOD. Removing the Altiris virtualization program and reverting back to the 1.6 jvm solved the problem.

We're planning to submit the issue to Symantec (they recently bought Altiris) so they can hopefully issue a fix. Their virtualization software is actually pretty cool, so I'd like to be able to have it reinstalled on my machine again at some point.

Adobe MAX 2007 - Sneak Peeks Part 3

I missed the Indesign and Flex sneak. Apologies to all you Indesign fans out there.

Ok, that's it for tonight. It's off to the MAX special event. This year's event is supposed to be the best ever. Hope to see you there.

Adobe MAX 2007 - Sneak Peeks Part 2 (ColdFusion Offline Apps Here!)

Back from break. The next 4 presenters are getting ready to go...

Danielle Beaumont is showing off custom skinning Flex and Air components using Fireworks. I'm not really familiar with all of the challenges associated with custom skinning (other than the fact that most Flex apps have a very similar look and feel), but I can see that what she's doing here is making a lot of people happy (loud applause).

Next up is Hemant Khandelwal talking about building online/offline apps with AIR & ColdFusion. This is huge. He's showing a typical application built using CF with the new layout and AJAX tags. Next he adds a new attribute to the cfgrid tag to tell it operate in off-line mode. ColdFusion then generates an AIR package containing the JavaScript, HTML, and other files necessary to make the application work. These applications can be run in both offline and online mode! There's automatic synchronization of data when an offline application goes online. He's now showing a mail application that's in offline mode sending data to the server. He clicks submit and data moves to an unsent folder. When the application goes back online, the data is sent out. This one is what a lot of ColdFusion developers have been asking for.

I have to post this now. Be right back for part 3...

Adobe MAX 2007 - Sneak Peeks Part 1

Live blogging from the Sneak Peeks session at Adobe MAX 2007. Sneak Peeks is one of my favorite sessions. It's always interesting to see what's on the minds of Adobe engineers, and what might be coming down the line in future releases.

They're pulling a Blues Brothers skit with Mike Downey and Marc Eaman dressed up like Dan Aykroyd and John Belushi. The standard disclaimer applies - the tech previews here may or may not ever make it into products...

There's going to be a vote for best sneak at the end of the session (built with CF and Flex, of course).

First up is Karl Miller and Karl Soule from the visual communicator (Vc) team. They're showing off a new tool (version 3) for doing video production that's supposed to be easy enough for a third grader to use. It has a teleprompter interface for script reading with the ability to place content at various places throughout the timeline. Pretty simple and powerful. It can also communicate with Flash Media Server for live streaming. It can control up to 3 cameras and has both blue and green screen built in. This tool looks like it makes it very simple to do tv quality production on a laptop.

Up now is Danielle Diebler who is showing off VoIP in the Flash Player. She's also mention that P2P in the Flash Player as well as extended codecs are in the works. She's demoing an application now that built using the technology called CoCoNiki. It's an online diary that uses voice and presence awareness. The VoIP quality seems pretty good so far. She's moving on to an Air application now that also does VoIP (she's making a live call to a cell phone.

Ken Sundermeyert is up from the Flash Home for Mobile group. He says Flash Home will let you replace the home screen on your phone (has to be a Flash enabled phone, of course) with a customized Flash based screen. The market for personalization is huge (2+ billion last year, excluding ringtones). Flash Home is capable of getting web data, including from Flash Cast. It is also integrated with the device. No sandbox, so it can access phone functionality natively (address book, sms, call logs, etc.). He's demoing a custom home screen that he built that pulls up a person's location when they call by getting their phone number from the call log, checking the area code, and displaying a graphic of the city. Live demo of someone in the audience calling his phone. The wallpaper on the phone switches from blue background to the Statue of Liberty. You'll also be able to get home screens from a catalog over the air.

Geoff Baum is showing off Photoshop Express, a new online version of Photoshop built using Flex. It's not meant to replace the full Photoshop, but is supposed to provide a "consumer" alternative for performing common image editing tasks. The interface is pretty intuitive. Lots of impressive editing capabilities. It's got a timeline for edits, so you can move backward/forward in edit sequences. All of the editing is non-destructive. Very impressive stuff!

They're setting up for part 2 now, so I'm going to cut this post of here, grab a beer, then come back for more. Stay tuned...