I went to the excellent CloudCamp gathering in San Francisco Tuesday night. CloudCamp was organized by Reuven Cohen, Jesse Silver, Dave Nielsen, and others, and there were about 350 people there (SnapLogic also sponsored). The timing was good, since it overlapped with a number of other conferences, which attracted a lot of folks from out of town. Overall, I declare this a success, and I’m looking forward to future CloudCamp events.

Cloud Computing is a broad concept, and that was reinforced by the nunber of talks and discussions that seemed to always converge on the topic of ‘What is cloud computing ?’

With an evolving concept like cloud computing, it’s hard to really nail down a definition of what it is, since there really isn’t an it yet. (It reminds me of the early days of the Internet, and the early days of the World Wide Web…how could anyone put a definition on either of those in the early days?)

I’m not going to even attempt to define cloud computing. I’m simply treating it as the general trend toward virtualization of computing resources. (Others have done a good job of clarifying the space. See Peter Laird’s post on the cloud market, or Reuven Cohen’s description.)

As of now, the cloud space can be divided into several broad market areas, primarily:

• Software as a Service (Saas)
• Platform as a Services (PaaS)
• Infrastructure as a Service (IaaS)

There is a lot of current activity in all of these areas among vendors and adopters. However, cloud computing will not happen overnight - it will be a gradual transition and, current hype aside, that transition is ongoing in all of these areas.

In the meantime, there are still a lot of existing software applications that will not move to the ‘cloud’ any time soon. Looking at the trends, this raises a lot of issues about how we are going to integrate all of these into a cohesive functional unit. As a result, I chaired a session at CloudCamp to promote some discussion on this topic, and we had 25 or 30 people in the room.

The session didn’t go as smoothly as I hoped, since there was one dissenting voice, who insisted the solution to all these problems was local storage on the client side. That aside, we did cover some good topics I’m summarizing here, and adding some of my other thoughts.

Integration is a very broad term, that encompasses multiple levels, from low level data interfaces to workflow and higher level business process automation. Despite the various levels and interests in the room, there was general consensus that integration is a real issue with cloud computing.

Some participants pointed that we haven’t fully solved the integration problem locally. Despite the vision of SOA, and the existence of messaging, there’s actually a lot of integration done today with Duct Tape and Paper Clips (or, in hard currency, Perl and PL/SQL.) The larger enterprises have the resources and skills to build their own integration capability, but smaller business (the earliest adopters of SaaS), simply don’t have the infrastructure in place. 65% of the enterprises surveyed by Forrested in late 2007 indicated integration issues as their reason for not considering SaaS.

The reality is that integration matrix is still too complex, with it’s permutation of protocols, access methods, and data formats.

Ray Wang of Forrester also notes this integration barrier is ‘more fallacy than fact’, although I don’t agree with that latter statement - cloud raises some significant new integration challenges, and they shouldn’t be underestimated.

If we have difficulty with local integration, the cloud wave only makes integration more difficult. There are a number of reasons why:

• New integration scenarios
• Access to the cloud may be limited
• Dynamic resources
• Performance

Before the cloud model, we had to tie local systems together. With the shift to a cloud model, we now have to connect local applications to the cloud, and we also have to connect cloud applications to each other, which adds new permutations to the matrix.

Its unlikely that everything will move to a cloud model all at once, so even the simplest scenarios require some form of local / remote integration. It’s also likely that we will have applications that never leave the building, due to regulatory constraints like HIPPA, GLBA, and general security and NPPI issues. All of this means integration must cross a firewall somwehere.

Cloud to Cloud also raises issues. Do we rely on the (competing) vendors to provide integration with each other ? Where is the integration hosted ? Does the integration live in the cloud as well? And if so, how does it connect to those local applications ?

Access to cloud resources, either SaaS, PaaS, or pure infrastructure, is more limited than local applications.

With local applications, we usually had complete access to the application, even when there were no good integration points in the original design. With custom applications, adding integration hooks was possible. Even with commercial applications, it was always possible to slip in database triggers to raise events and provide hooks for integration access.

Once applications move to the cloud, custom applications must be designed to support integration because we no longer have that low level of access. For SaaS, we are dependent on the vendor to provide the integration hooks and API’s. For example, the SalesForce.com Web Services API doesn’t support transactions against multiple records, which means integration code has to handle that logic. For PaaS, the platform might support integration for aplications on the platform. Platform to Platform is still an open question.

Some would argue that a limited set of APIs will improve the situation, since backdoor access is what led integration into the ‘Duct Tape’ mess in the first place. They have a valid point. But those API’s must be able to handle the integration required, and the popularity of BeautifulSoup tells me screen scraping as a workaround is alive and well.

The true cloud model abstracts away most of our notions of physical hardware, and everything becomes a service in some form. This is one of the benefits of cloud, but it also means integration models change. In a world where applications and infrastructure move and change dynamically, traditional notions of tightly coupled integration are no longer valid. Add to this the issues of application versioning (no longer under our control in a SaaS environment), or PaaS platform changes (also no longer under our control) and tight coupling becomes a dead end.

It clear from the SaaS vendors that the Web is the way to go when it comes to client access. It seems that lower level interfaces will follow the same REST route.

Cloud or not, we still can’t get away from physical limitations. Although we may see better application scalability and performance, the network distances between elements in the cloud are no longer under our control. Bandwidth isn’t the limiting factor in most integration scenarios, round trip latency is. On the LAN, we can optimize. In the cloud, we lose that ability, and have to live with longer latency in combination with SLA’s from multiple vendors.

These performance constraints change the integration model. Integration can no longer depend on high performance local access - anyone that does complex analysis on, say, SalesForce leads, or anyone trying to pull SaaS data into a warehouse will tell you performance is already an issue.

There are a lot of implications as a result of the shift to a cloud model, and I originally hoped we could get into the deeper issues during CloudCamp, but time wasn’t on our side.

Cloud vendors understand that integration is an issue, particularly at the application and platform level. We are increasingly seeing both integration as a service, and integration as part of the platform.

The question is what form will that integration capability take ? Will it be open and standards based like Open Social? Or will it be vendor and platform specific ?

Theres a lot at stake here, because there a strong possibility that integration is becoming part of the vendor lock-in strategy.

Peter Laird wrote up a great analysis on Terms of Service for SaaS providers and it got me wondering: What about the emerging Platform as a Service providers?

They’ve got an especially difficult challenge because they’re running other people’s code. How do you ensure availability when it’s not your code? Google’s App Engine conspicuously avoids the problem by not providing any SLA.

Generally speaking, there seems be three categories of providers:

Obviously, the closer you get to bare metal, the harder it is for you to provide service guarantees.

I’ve already noted that AWS provides for 99.9% in their SLA, but that’s only for the service itself. Nowhere in that document does it say anything about your image’s uptime. Seems that you are on your own entirely.

This is a tricky problem even for the app builders. They all support some kind of scripting environment and you can only sandbox so much. Not to pick on CogHead, but I found this post on their forum. So, as you can see, these problems are going to show up everywhere.

I’d love to hear from anyone that has more details on SLA from platform providers.

I saw today via Techmeme that Twitter was excited and proud that they were able to achieve 97.3% uptime during the Apple WWDC keynote yesterday.

If I were them, I’d be a more humble and a little more circumspect. Reading thought he comments, you’d think they just landed a man on the moon.

First, let me say that under many circumstances achieving 97.3% availability is grounds for termination. Most Enterprise SLAs specify 99.9% or more with service credits applied for failure. Amazon’s SLA provides for 99.9% with 10% credited back if they fall below that and 25% credit below 99%.

Salesforce.com had some serious trouble with availability a while back and people were legitimately wondering if they would survive the crisis. Today, they make a provision for these SLA failure expenses, and so far, have been lucky enough (i.e. smart enough) not to have to make any payments.

Just to put that in perspective, 99.9% uptime translates to about 44 minutes downtime per month (99%, about 440). So, at 97.3% for the (roughly) four hours of peak time usage during the keynote the were unavailable for about 6.5 mins. or nearly 15% of the downtime budget for the month.

This isn’t’ something I would be proud of.

Second, your users are not your QA or test engineering department. They claim:

…we learned a lot during this stress test and that will translate to better performance down the line.

Finally, turning off features to support peak loads is treating the symptoms, not the underlying problem.

Is it any wonder their site is as unreliable as it is? With this kind of attitude, I don’t think things are going to materially change anytime soon.

CIO magizine reports on a recent survey by Forrester indicates that integration remains the number one inhibitor for Enterprises adopting SaaS.

This comes as no surprise to me.

We’ve been talking to all the large hosted application providers and they all say the same thing: My subscribers have data inside their organization and they need to get it out. The larger the company, the more likely they need to integrate their SaaS application with behind the firewall data and other applications. When integrating that data becomes more difficult (or less secure, or available) the extra integration complexity is traded off against the simplicity of a SaaS alternative. Once the complexity gets too large, it can quickly overcome the benefits of SaaS.

Integrating SaaS with internal apps remains problematic since access generally requires software that runs inside the organization to manipulate and transform the data and to initiate integration with the SaaS apps for higher security.

There are a number of hosted integration alternatives emerging, but few adequately address the behind the firewall issue. The ones do, suffer from either being too complex for the SaaS marketplace, or too limited in function to handle the task.

We here at SnapLogic have been working on the problem for a while now. We think we’ve got the right approach to this problem. We’ve got open source software that you can download and run wherever the problem lives. Our distributed approach allows you to apply the necessary access and/or transformation functions wherever it’s needed, and deploy in on top of your existing infrastructure, weather that in your data center, in the DMZ, or in a co-location facility. And since it works just like the web so you already know how it works to keep it running fast and secure.

Of all the possible deployment scenarios for SnapLogic, integration SaaS applications with enterprise apps is by far the most common.

I read that Google will be providing all their Android software under an Apache license. Ed Burnette at ZDnet says:

With Android, carriers like Sprint and Cingular are free to use either open or closed source for any custom applications they install on their new Android phones. And of course, third party programs (like the ones you write) can be open or closed.

Then concludes:

It wouldn’t surprise me to see completely free/open distributions of Android which eschew any proprietary add-ons in favor of open source alternatives.

I seriously doubt this.

Given AT&T, Verizon’s and other carriers history on locking their hardware I’m pretty sure that all we’ll see (in anything at all) is their version of Android that’s bound to the handset with proprietary drivers, etc. that makes it the only distro that works.

I know why they went with Apache, because no carrier would go anywhere near a Copyleft license and at the end of the day without carriers Android isn’t going anywhere.

I read a great article by Neil Raden at Intelligent Enterprise about what BI 2.0 is going to look like. It goes into a lot of detail about BI 1.0 and it’s shortcomings (complexity, cost, maintenance, shelfware, etc.), then goes on to describe BI 2.0.

Specifically, he says that it….

has to provide simple, personal analytical tools on an as-needed basis with a minimal footprint and cost. Rather than relying solely on a rigid metaphor like data warehousing, BI needs the ability to access data anywhere it can be found and to performing integration on the fly, if necessary.

To me, I think we’re closer than anyone thinks. The ability to access data anywhere and perform integration on the fly is exactly what SnapLogic does. And on the front end, there is an emerging class of browser-based clients that have the analytical horsepower of Excel.

In fact, we’re partnering with Kirix, a company that has developed a ‘Data Browser’ to provide exactly these kinds of BI 2.0 solutions. If you provide your data as SnapLogic RESTful data services, all you have to do is point Kirix Strata to them and you’ve got a powerful BI 2.0 solution that can analyze data without the cost, complexity of a BI 1.0 approach.

We’re going to be talking about this together this week in a webinar we’re doing together. If you’ve got some time, please register and attend.

Gartner recently released it’s list of Top Ten Disruptive Technologies for 2008-2012 and it’s interesting to see what made the list:

It’s also interesting to see how it compares to the list of Top Ten Strategic Technologies for 2008 that was published back in October 2007.

Not sure what makes some technologies Disruptive and others Strategic, but the four that are both Strategic and Disruptive are:

It would be hard to argue any of these don’t belong. It also reflects the reality of where most, if not all of the innovation is taking place right now.

That’s a catchy title? Don’t you think?

Lately we’ve been talking a lot about high level capabilities of SnapLogic and some of the interesting things that can be done with it. Googleing HTML meta data, access by desktop applications, LinkedIn Mashups, all the rest. But we’ve gotten a bunch of basic questions like: All I want to do is expose my database queries as a service? How do I do that? How can I parametrize it so that I don’t have to create a bunch of separate Resources all doing basically the same thing?

Boring? Perhaps. But sometimes those are the kinds of problems that people need to solve every day.

So we made another screencast demo. This one shows how you can use SnapLogic to create a set of RESTful data services where each URI accesses a database and responds with the result of a parametrized query.

Check it out here.

I spent the weekend at BarCamp Boston, and had a great time hacking and socializing with the rest of the attendees.

There were 256 people registered on the Wiki, official attendance was ‘226 unique humans’ over Saturday and Sunday. For many people, this was their first BarCamp Boston event. I estimate 70% of the Saturday attendees returned Sunday, and we had a flush of new attendees on Sunday. Over the two days, 91 really good pizzas were consumed, in addition to a continuous supply of coffee, soda, and snacks.

The venue was Matignion High School. I liked it - minimalist yet functional, and easy to get to. It’s hard to describe the experience of being in a high school cafeteria with 200 other technology lovers, with 50 simultaneous conversations taking place. It was like, well, being in a high school cafeteria

The session board filled up quickly on both Saturday and Sunday. On Saturday, I cauge Matt McKeon’s talk on Many Eyes, Joe Cascio’s talk about Distributed Twitter, and parts of the Google App Engine and Amazon Web Services talks.

In the afternoon, I gave a talk on Data Services for RIA’s, and led a roundtable discussion on public data. Public data is a really popular topic, and I’ll be spending more time exposing this type of data with SnapLogic in the future.

Session scheduling was tight, but there was plenty of time for casual talks in the morning and afternon breaks, and I got to meet a lot of really interesting people.

Sunday, I caught John Eckmans talk about the Twitter API, the Django BOF, and the session on the next BostonBarCamp. I also joined in on a good afternoon session about Open Source licensing and business, as well as a Code Secrets talk. I spent much more time on Sunday in spontaneous conversations, which is why I really like the Unconference style.

I was also good to meet some hardware hackers, since my favourite programming language has always been solder. There were some Arduino advocates around doing their thing, and I packed my TuxDroid, since it’s been collecting dust since OSCon last year, and needed a workout. I forgot how much fund TuxDroid was, and I can think of a dozen projects with him that I don’t have time for. Tux also got to meet Shimon Rura, organizer of BarCamp Boston.

BarCamp Boston is now on my list of preferred events, and I’ll try to make the next one. Tags for the event were BarCampBoston3 and #bcb3 ( for hashtags)

I came across the Amazon Web Services bandwidth graphic again today over at AWS blog.

I remember back in January when I first saw these stats I wondered why would anyone compare these two values? They have almost no relation to one another and don’t reveal much (if anything) about true comparative metrics like users, visitors, revenue.

Its pretty straightforward to outsource bandwidth to Akamai or another CDN, and almost everyone does and I’m sure that Amazon’s IT guys are constantly looking for ways to keep that number low. For AWS, S3 is storage so I expect bandwidth to be huge.

It reminds me of the early days of the net where sites would quote site traffic, not in page views, or visitors, but ‘hits’. They sure do sound impressive, but almost impossible to use for comparison.

I think AWS are great and anyone that isn’t spending time figuring out how to take advantage of them has their head in the sand. It’s just that these kinds of comparisons are no substitute for real information.

Nice tidbit in today’s Deal Journal about clawbacks kicking in for Blackstone. The sad thing is that if returns really do drop, and these clawback ever kick in, they could get a do over by starting another fund. I’ve been wondered about this for a while.

Interesting thread over at Java Posse about the irony of how the GPL has somehow been co-opted by SpringSource to achieve their business objectives.

This should come as no surprise to anyone that’s familiar with the FSF and the GPL. They’ve never been against capitalism (in spite of some people’s view of Richard Stallman). Eben Moglen frequently speaks about how to use the GPL to a avoid becoming your competitor’s lunch.

I came across this model for Cloud Computing at Dell.

Twelve layers? Give me a break. This just strikes me as silly. It’s a hodge podge of business model, management, facilities, abstractions.

Am I to believe that you can overlay a Business Model on top of any combination of the lower levels? Transport on top of Data Management? Huh? I don’t get it.

Jim, couldn’t you have at least stopped at eleven?

Seriously, the ISO Networking Model stack only had seven layers and still 5 & 6 are always collapsed into 7.

IMHO, Leaky Abstractions would be the least of my worries here.

Needs some work.

Some chatter around the recent dust-up triggered by mySQL’s change of heart regarding close-sourcing portions of their Enterprise Edition.

Savio jumps right in and questions if the Community is hurting the OSS business model. Matthew Aslett of the 451 Group follows up today noting that there seems to be….:

…. growing animosity of some sectors of the open source software user community towards commercial software vendors and activities. Where once the commercial success of an open source vendor was to be celebrated, increasingly it seems it is treated by some as a reason for caution and doubt.

Like Savio, I was disappointed to hear of mySQL’s decision, but I think this problem was initially of their own making, and changing their mind will only make matters worse. To me the social contract that exists between the community and an open source business should be crystal clear laying out what is expected from everyone. Debian and Fumambol make their social contract explicit and document them here and here.

I think that’s a great idea and is something every open source business ought to do. When you start changing things, as mySQL did, is when you start to get crosswise with your community.

Perhaps explicit Community Social Contracts are something who’s time has come.

I’m looking forward to BarCampBoston 3, which is coming up this weekend. There are 150 people registered so far, and over half are new attendees for this event (like me.)

The RedMonk Unconference at Community One absolutely rocked, and so far the energy level for Boston is shaping up to be at that level or better, which would be awesome.

I proposed a talk on Data Services for rich applications, and there seem to be quite a few attendess interested in web applications and Python. But for me, the real attraction of unconferences is the face to face interaction. You never know whats going to really transpire, and thats part of the fun.

It looks like one thing that will happen at BarCamp Boston is a programming un-competition. It’s tempting to enter…I’ll wait and see what the words are….

It’s not too late to sign up..

I got my first chance to try out SnapLogic on an Asus EeePC running Linux this afternoon.

SnapLogic Designer and Explorer started up in Firefox with no problems. Screen real estate is a little tight for real SnapLogic development work (this was a 7″ model), but nonetheless it was surprisingly usable for such a small screen.

Theres something really satisfying about keeping things simple enough that it’s possible to run on lightweight machines with no software installation needed.

A woman’s Mac was stolen and when the thieves were using it, she used Back to My Mac to take their pictures.

Book’m Danno

Something’s going on over at Moscone, and it ain’t healthy…..

I read an interesting article titled Economics and the Entrepreneur. It discusses in some detail the works of Joseph Schumpeter, John Maynard Keynes and John Kenneth Galbraith.

It goes into some of their economic theories, which are quite well known, but it was interesting to learn how their theories were influenced by their own personal experiences.

McCraw, a distinguished professor emeritus of business history at Harvard and author of the Pulitzer Prize-winning Prophets of Regulation (1984), ably draws out the contrast between the two men. Keynes knew nothing but stability during his life, enjoying a lifelong attachment to Cambridge University, where he inherited a faith in smooth and automatic economic progress. This faith, and the stability of which it was born, help explain Keynes’s intellectual response to the Depression. Schumpeter, however, knew anything but stability–the Austro-Hungarian Empire of his youth disappeared in World War I, and Austria was later annexed by Hitler’s Germany.

You can almost imagine that Shumpeter actually lived Creative Destruction.

But what I found most interesting were their disparate views of the entrepreneur.

Schumpeter was schooled in the tradition of Austrian economics, which emphasizes the crucial role of the entrepreneur who innovates and takes what may seem to be irrational risk–a perspective regarded by the Keynesians as quaintly Romantic.

Or how about this….

Galbraith had written in 1967 that the entrepreneur was a “diminishing figure”: “Apart from access to capital, his principal qualifications were imagination, capacity for decision[making] and courage in risking money, including, not infrequently, his own. None of these qualifications is especially important for organizing intelligence or effective in competing with it.” In the entrepreneur’s place was supposed to be the “technostructure,” which would stabilize demand and eliminate uncertainty from the market.

Boy, how times have changed.

This email is being sent out to folks from IONA (and probably Microsoft and Actional as well) and it left me speechless.

How on earth, in this day and age, can any of these companies look their customers in the eye and tell them that their software license fees are being used to further develop products and innovate?

Let’s assume that they can get 100 people to attend. Assuming they spend $2K on the venue, another $5K for the speaker, $10K for travel etc. for a bunch of folks, plus $10K cruise and booze (all really low estimates), comes in around $25-30K or $250-$300 per person!

I know that’s about right because that’s right around what proprietary software companies willingly pay to get qualified leads for their multi-six-figure software packages.

Its not the economics that surprise me so much, because trade shows and other awareness events can cost about the same, but the fact that somehow entertaining prospects and getting them drunk is going to make a sale more likely just seems anachronistic.

Perhaps they need to be drunk to consider their offerings?

I thought we had all moved past this. But I guess not.

They say they’re going to talk about Best Practices and Worst Practices. My advice is that they talk about the Worst Practice of spending a fortune on things the customer doesn’t care about.

Out of touch and out of their minds.

IONA seems to be desperately hanging onto their proprietary model, while at the same time trying to foster community around their open source Fuse products. To me, they’re in a tough spot: a tiny proprietary software business ($16M revenue last quarter) trying to compete with multi-billion dollar enterprise software companies, but without the commitment to an open source offering that’s going to attract the community to keep it competitive (or relevant).