The motivations for SnapLogic
Hello, Mike here, signing on.
Now that we have released our Beta version of SnapLogic, it’s a good time for me to reflect on the motivations behind the founding of SnapLogic, and what is driving us forward.
This is longer than a typical blog posting. But the motivations behind an Open Source project are important, and the attempt to answer these questions drove many of the concepts behind SnapLogic, and resulted an a set of principles that will keep reappearing as we move forward.
My decision to start the SnapLogic project was a convergence of several themes that all seemed to come together at the right time in early 2006:
- Why is data access getting harder ?
- What is happening to the enterprise software business model ?
- Why is all software becoming so complex ?
The first theme was my increasing dissent with the direction that data access and manipulation was taking. Despite years of computer industry progress, the seemingly straightforward problem of creating a data abstraction layer was turning into one API after the other, from simple files and database specific programmer API’s, to ODBC, Corba, JDBC, DAO, SOAP, WS-*, SDO, and a list of many others I’ve omitted. Rather than converging toward a better abstraction for data, the only apparent progress we had made was in adding complexity, creating TLA’s and ETLA’s. In the data access domain, API lock-in policies from vendors seemed to be a stronger driving force than actually solving the problem. On a daily basis, I was solving problems that all seemed to have their roots in the complexity of data access. Around 1970, Unix gave us a data abstraction where every file was a stream of byes, and pipelines could process data sequentially. But as an industry we have failed to generalize a concept like that to all data. There had to be a better solution to this data access problem.
The second theme was my increasing awareness that something was wrong with the traditional ‘enterprise software’ business model. The pressure to increase license sales growth was forcing enterprise software companies on a treadmill, where new deals depend on new product releases, which in turn require new features to justify the higher license fees. This more complex software is harder to sell and takes longer to develop. Yet, on the customer front, the more complex software costs more, and requires longer implementation cycles. And to top it all off, just when the software is installed, integrated, and humming along, it’s time to start planning the upgrade to the new vendor release. Enterprise software is a tough business, and companies face a lot of challenges. But it sure seemed like
“death-wielding market forces such as open source” were forcing changes in the industry.
The third theme was the tendency toward increasing complexity in software systems, rather than simplification. I have always believed one of the the important aspects of engineering is packaging up solutions to complex problems in a usable way. Clay Shirky had an interesting view on this in his thoughts on Situated Software:
Situated software isn’t a technological strategy so much as an attitude about closeness of fit between software and its group of users, and a refusal to embrace scale, generality or completeness as unqualified virtues.
I had experienced the power of the situated software concept, when building out the internal infrastructure for a global customer support organization. Rather than try to solve every imaginable problem, we concentrated on simple applications that made team members more productive – wikis, forums, and fast, full text search of bugs and customer history are incredible tools when applied to a distributed customer support team. But for me, Clay’s real insight was not just Situated Software per se – it was proof of the potential that can be exposed by looking at a problem from a different perspective. Challenge the accepted norms, look at a problem in terms of what is really needed to solve it, focus on the solution in terms of what the user needs. Of course, that approach is at odds with the feature treadmill of enterprise software.
Based on these themes, I started thinking about a solution for the Gordian Knot of data access. My usual approach to complex problem solving is to put something on the back burner, and let it simmer while I read or think about something else. One of those other things was a visit to a Web 2.0 meeting hosted by the San Francisco Bay Area SIGCHI.
I really went to the meeting to get an update on the latest web developments. At the time, for me Web 2.0 seemed to have little in common with data integration. I thought it was all about Ajax, rich interfaces, and user interaction.
I left there with the realization that the web was really all about data integration, and a newly discovered awareness of how fast the web actually was evolving. And, most importantly, the recognition that the web didn’t seem to be limited by the same constraints that were holding back software ‘inside the firewall.’ There was more to this than just Web 2.0 principles. Something deeper was enabling Web 2.0 and the overall growth of the web.
I had never thought about software architecture of the web. It was just there, and it worked, so I never had to think about it. I knew about browsers, HTTP, proxies, caches, and other pieces of the puzzle. But I had never truly studied the architecture behind all of this. So, I went back to the basic references, and started reading about REST. It clicked, although not on the first pass. (OK, not on the second pass either. It required chewing and digestion.)
So, what happens when you challenge the established concepts behind data integration, and combine the REST architecture of the web, the power of Community and Open Source, and the principles of Web 2.0 ? SnapLogic is the beginning of that journey.