(This is the first in a three-part series)
As a guy who’s spent the better part of his technology career in and around the challenge of integration, I’ve gotten my fair share of bumps and bruises from years of working with large and small organizations on this issue. Over the years there have been many approaches to solving the business and technical obstacles to integration, but the drivers have always been the same, i.e. moving information across the demand chain and supply chain of an organization, and getting a full 360-degree view of customers to promote sales and customer satisfaction.
To compare the various approaches to this challenge, it’s important to consider how our system infrastructures have and haven’t changed over the past couple decades. I’m old enough to have lived through the days of getting mainframe systems to behave well in a distributed network environment, and then the Web 1.0 days of getting distributed systems to scale, and making them work better in a web environment. Today we have a new set of requirements and while they’re different, they are only additive to those that came before, fifteen years later, we’re still battling the same architecture problems.
The way we attempted to solve the integration challenge in the past was hard. It required a ton of coding and special skills, and of course, product upgrades were guaranteed to break these arduously built integrations. Hand coding the integration points sometimes sounded good at first. Developers loved it because it was a little more intellectually interesting than working inside someone else’s tool, but it created massive downstream issues. Lack of documentation, the need for specialized skill sets, and an exponential growth in the number of point to point integrations made for anything but an agile IT environment.
One of America’s largest retailers recently told me that 75% of their integrations are done in a commercial product environment and require 5 full-time employees to manage day to day, while 25% are hand-coded integrations that require 50 full-time employees. Now that’s a lot of daily care and feeding.
SOA, Web Services, ETL, and EAI were the promises of the mid-90s, meant to eliminate the need for hand coding. While there have been some successes, it’s been a hard fought journey. I’ve heard many enterprise service bus (ESB) initiatives described as expensive, convoluted “big dig” projects (if you’ve been to Boston in the past three decades, you know what I’m talking about). Yes it got done, but at an enormous cost, and well out of scope of the original objective.
In my next post, I’ll share my thoughts on how integration challenges and solutions have evolved since the 90s, and offer up some additional considerations that are even more important today.