I went to the excellent CloudCamp gathering in San Francisco Tuesday night.
CloudCamp was organized by Reuven Cohen, Jesse Silver, Dave Nielsen, and others, and there were
about 350 people there (SnapLogic also sponsored). The timing was good,
since it overlapped with a number of other conferences, which attracted a lot
of folks from out of town. Overall, I declare this a success, and I’m looking
forward to future CloudCamp events.

Cloud Computing is a broad concept, and that was reinforced by the nunber of
talks and discussions that seemed to always converge on the topic of ‘What is
cloud computing ?’

With an evolving concept like cloud computing, it’s hard to really nail down a
definition of what it is, since there really isn’t an it yet. (It reminds
me of the early days of the Internet, and the early days of the World Wide
Web…how could anyone put a definition on either of those in the early days?)

I’m not going to even attempt to define cloud computing. I’m simply treating it
as the general trend toward virtualization of computing resources. (Others have
done a good job of clarifying the space. See Peter Laird’s post on the cloud
market
, or Reuven Cohen’s description.)

As of now, the cloud space can be divided into several broad market areas,
primarily:

  • Software as a Service (Saas)
  • Platform as a Services (PaaS)
  • Infrastructure as a Service (IaaS)

There is a lot of current activity in all of these areas among vendors and
adopters. However, cloud computing will not happen overnight – it will be
a gradual transition and, current hype aside, that transition is ongoing in all
of these areas.

In the meantime, there are still a lot of existing software applications that
will not move to the ‘cloud’ any time soon. Looking at the trends, this raises
a lot of issues about how we are going to integrate all of these into a
cohesive functional unit. As a result, I chaired a session at CloudCamp to
promote some discussion on this topic, and we had 25 or 30 people in the room.

The session didn’t go as smoothly as I hoped, since there was one dissenting
voice, who insisted the solution to all these problems was local storage on the
client side. That aside, we did cover some good topics I’m summarizing here,
and adding some of my other thoughts.

The integration problem

Integration is a very broad term, that encompasses multiple levels, from low
level data interfaces to workflow and higher level business process automation.
Despite the various levels and interests in the room, there was general
consensus that integration is a real issue with cloud computing.

Some participants pointed that we haven’t fully solved the integration problem
locally. Despite the vision of SOA, and the existence of messaging, there’s
actually a lot of integration done today with Duct Tape and Paper Clips (or,
in hard currency, Perl and PL/SQL.) The larger enterprises have the
resources and skills to build their own integration capability, but smaller
business (the earliest adopters of SaaS), simply don’t have the infrastructure
in place. 65% of the enterprises surveyed by Forrested in late 2007 indicated
integration issues as their reason for not considering SaaS.

The reality is that integration matrix is still too complex, with it’s
permutation of protocols, access methods, and data formats.

Ray Wang of Forrester also notes this integration barrier is ‘more fallacy than
fact’, although I don’t agree with that latter statement – cloud raises some
significant new integration challenges, and they shouldn’t be underestimated.

Why the cloud makes integration harder

If we have difficulty with local integration, the cloud wave only makes integration
more difficult. There are a number of reasons why:

  • New integration scenarios
  • Access to the cloud may be limited
  • Dynamic resources
  • Performance

New integration scenarios

Before the cloud model, we had to tie local systems together. With the shift
to a cloud model, we now have to connect local applications to the cloud, and
we also have to connect cloud applications to each other, which adds new
permutations to the matrix.

Its unlikely that everything will move to a cloud model all at once, so even
the simplest scenarios require some form of local / remote integration. It’s
also likely that we will have applications that never leave the building, due
to regulatory constraints like HIPPA, GLBA, and general security and NPPI
issues. All of this means integration must cross a firewall somwehere.

Cloud to Cloud also raises issues. Do we rely on the (competing) vendors to
provide integration with each other ? Where is the integration hosted ? Does
the integration live in the cloud as well? And if so, how does it connect to
those local applications ?

Access may be limited

Access to cloud resources, either SaaS, PaaS, or pure infrastructure, is more
limited than local applications.

With local applications, we usually had complete access to the application,
even when there were no good integration points in the original design. With
custom applications, adding integration hooks was possible. Even with
commercial applications, it was always possible to slip in database triggers to
raise events and provide hooks for integration access.

Once applications move to the cloud, custom applications must be designed to
support integration because we no longer have that low level of access. For
SaaS, we are dependent on the vendor to provide the integration hooks and
API’s. For example, the SalesForce.com Web Services API doesn’t support
transactions against multiple records, which means integration code has to
handle that logic. For PaaS, the platform might support integration for
aplications on the platform. Platform to Platform is still an open question.

Some would argue that a limited set of APIs will improve the situation, since
backdoor access is what led integration into the ‘Duct Tape’ mess in the first
place. They have a valid point. But those API’s must be able to handle the
integration required, and the popularity of BeautifulSoup tells me screen
scraping as a workaround is alive and well.

Dynamic Resources

The true cloud model abstracts away most of our notions of physical hardware,
and everything becomes a service in some form. This is one of the benefits of
cloud, but it also means integration models change. In a world where
applications and infrastructure move and change dynamically, traditional
notions of tightly coupled integration are no longer valid. Add to this the
issues of application versioning (no longer under our control in a SaaS
environment), or PaaS platform changes (also no longer under our control) and
tight coupling becomes a dead end.

It clear from the SaaS vendors that the Web is the way to go when it comes to
client access. It seems that lower level interfaces will follow the
same REST route.

Performance

Cloud or not, we still can’t get away from physical limitations. Although we
may see better application scalability and performance, the network distances
between elements in the cloud are no longer under our control. Bandwidth isn’t
the limiting factor in most integration scenarios, round trip latency is. On
the LAN, we can optimize. In the cloud, we lose that ability, and have to live
with longer latency in combination with SLA’s from multiple vendors.

These performance constraints change the integration model. Integration can no
longer depend on high performance local access – anyone that does complex
analysis on, say, SalesForce leads, or anyone trying to pull SaaS data into
a warehouse will tell you performance is already an issue.

What this means for integration

There are a lot of implications as a result of the shift to a cloud model, and
I originally hoped we could get into the deeper issues during CloudCamp, but
time wasn’t on our side.

Cloud vendors understand that integration is an issue, particularly at the
application and platform level. We are increasingly seeing both
integration as a service, and integration as part of the platform.

The question is what form will that integration capability take ? Will it be
open and standards based like Open Social? Or will it be vendor and platform specific ?

Theres a lot at stake here,
because there a strong possibility that integration is becoming part of the
vendor
lock-in strategy.

One Response to “Connecting Clouds – Integration and Cloud Computing”

  1. Joseph A. di Paolantonio

    Chris,

    I regretted not being able to make CloudCamp. One area I would have liked to discuss is the focus of integration. Too often, integration projects, whether stemming from ETL, EAI or SOA progrms, focus on application integration and not data integration. I think that the perspective that the applications must be integrated is what has led to the duct tape and paperclip routine. For most integration, and especially for integration among the clouds, the architecture and metadata must be formulated to provide for secure sharing of clean data.

    The real opportunity will be in Data as a Service – just to add another *aaS to the mix. ;-)

Leave a Reply

  • (will not be published)