SnapLogic’s Vice President of Engineering Vaikom Krishnan recently outlined the 10 New Requirements for Modern Data Integration. Here is a presentation that summarizes these requirements.
With the recent publication of the Gartner Magic Quadrant for Enterprise Integration Platform as a Service (iPaaS), the awareness of a new approach for connecting data, applications, APIs and the Internet of Things is on the rise. A good example is the recent feature in NetworkWorld on the category: iPaaS: What this cloud technology is and why it’s important, which includes an overview of SnapLogic customer GameStop.
The right iPaaS solution should be built to connect the 3 A’s of enterprise integration: Anything (data, apps, APIs, things), Anytime (real-time, batch, streaming), Anywhere (cloud, on prem, hybrid). I’m often asked what are some of the most common iPaaS use cases today.
Here are four:
Over the past few weeks I have outlined what we believe are the 6 primary requirements for integration platform as a service (iPaaS):
- Fully-Functional Cloud-based Service
- Single Platform for Big Data, Application and API Integration
- Elastic Scale
- Built on Modern Standards
- Broad Cloud and On-Premises Connectivity
- Self-Service for Citizen Integrators
The final post in the series focuses on what is probably the most disruptive change to traditional approaches to data and application integration – the demand for self-service integration from the business and the rise of the so-called Citizen Integrator.
Integration tools have historically been the domain of the specialists. The black belts. The plumbers. They have also required distinct skill sets and domain knowledge – from enterprise application integration (EAI) and enterprise service buses (ESBs) and services oriented architecture (SOA) to extract, transformation and load (ETL) and enterprise data warehousing (EDW) and business intelligence (BI). Different teams. Different tools. Different worlds.
With apologies to S.E. Hinton, that was then, this is now…
A few years ago, Gartner predicted that Citizen Developers will build at least 25% of new business applications by 2014. They’ve also predicted that in a few years, the CMO will spend more on IT than the CIO and the majority of new integration flows will be developed outside the control of IT departments. Of course, it’s not a new phenomenon that people want access to trusted information – including when, where and how they want it – but the The Nexus of Forces (also known has SMAC – social, mobile, analytics, cloud) has shaken up traditional roles and responsibilities when it comes to the business and IT relationship. In the impatient enterprise, waiting days, weeks or months to get access to this information is not an option.
With cloud deployments comes cloud expectations, and speed and time to value are the primary drivers for integration platform as a service (iPaaS). Enter the Citizen Integrator.
Citizen Integrators generally have a solid background in data, but they are not classically trained on the legacy tools of the trade. And if they do have experience with ETL/ELT or EAI tools, newer approaches to solving both old and new integration challenges may require some re-thinking (as explained in this iRobot video). If the Citizen Integrator has an applications background, they are typically well-versed in SaaS and have a good grasp of specific end-points and business processes. If they have an analytics background, the Citizen Integrator may have used business intelligence tools and may now even have the coveted “Data Scientist” title. On average, however, Citizen Integrators are in the line business. The have solid Excel skills and a role that requires more and more access to data from multiple systems. Digital marketers are a good example of a team that may look to tools like SnapLogic, AWS Redshift and Tableau for self-service integration of data from multiple channels, low-cost data warehousing and advanced data visualization. And they want the entire infrastructure up and running in days, not months or years.
If this is where you see integration going at your company – whether the focus is connecting data, applications or APIs, here are a few features that we believe a cloud integration service is going to require in order to meet the needs of your Citizen Integrators:
We’ve recently talked about the 5 different signs that you need a better cloud integration strategy, including struggles with the integrator’s dilemma, swivel chair integration and considerations about going back to on-prem due to diminishing SaaS returns. As a solution to finding an integration platform to satisfy the 5 signs we’ve talked about, we’ve also come up with some key features and functionality of the SnapLogic Elastic Integration Platform that you should know about, including its ability to handle both structured and unstructured data, going beyond point-to-point integration, functionality in hybrid integration use cases and the ability to scale out elastically. Read on for how these different features can help you jumpstart a new and improved cloud integration strategy:
- It’s built to handle structured and unstructured data – our support for documents allows our Snap endpoints to directly consume hierarchical data in native format and send it on to downstream Snaps in a pipeline. There is also no requirement to flatten data into records or turn a JSON document in a string or BLOB type.
- It’s built to handle streaming and batch app and data integration requirements – as Loraine Lawson has said in reference to our article Why Buses Don’t Fly in the Cloud: Thoughts on ESBs, putting a JSON format on a traditional, XML-based ESB is like making a silk purse out of a sow’s ear.
- It goes beyond point-to-point integration – in this chalk talk we talk about the challenges of point-to-point cloud integration and outline the benefits of an elastic integration platform (more on that below). SnapLogic operates on a hub and spoke integration architecture, enabling customers to incrementally integrate more complex apps both in the cloud and on-premises at a much lower cost than traditional integration and without re-work or manual coding.
- It’s built for hybrid integration use cases – as SaaS adoption increases and gains greater acceptance in IT organizations, enterprises require modern integration processes that offer multiple solutions for different scenarios. These scenarios are: cloud to cloud, cloud to ground, ground to ground and hybrid (cloud to cloud and ground).
- It’s a multi-tenant, software-defined cloud service – the primary concept of software-defined networking (SDN) is “the decoupling of the system that makes decisions about where traffic is sent (the control plane) from the underlying systems that forward traffic to the selected destination (the data plane). The SnapLogic Elastic Integration Platform is architected on these SDN concept; the “control plane” controls where and how data is process based on user configuration and preferences, while the “data plane” (aka the Snaplex) does that actual processing of data as per instructions received from the control place. You can learn more about our software-defined integration here.
- It’s built to scale out elastically – one of the requirements of a modern iPaaS is elastic scale capabilities, and SnapLogic has built-in smarts to automatically scale the Snaplex out and in to handlevariable traffic loads.
- The UX is designed for citizen integrators – our Summer 2014 release refines and improves the HTML5, cloud-based user interface for both advanced users and “citizen integrators” who require maximum ease of use and self-service. As our UX team says about designing the SnapLogic Elastic Integration Platform, “it wasn’t enough to simply round the corners of an icon – we needed to rethink and rebuild the technology.”
Over the last few months we’ve written a lot about the architecture of SnapLogic’s new elastic integration platform as a service (iPaaS). In 2010, the company recognized that in order to be able to meet the changing application and data integration requirements of the Social, Mobile, Analytics (Big Data), Cloud and the Internet of Things (SMACT) era, a new platform would be required. A few of the organizing principles that went in to the building of the SnapLogic Elastic Integration Platform included:
- It had to be able to handle structured and unstructured data;
- It had to be built for data and API streaming, but also have the ability to handle batch data integration requirements;
- It had to go beyond point-to-point integration and deliver cloud-to-cloud and cloud-to-ground orchestration;
- It had to be built for hybrid integration use cases, with a software-defined architecture;
- It had to be delivered as a multi-tenant cloud service;
- It had to be able to scale out to take advantage of cloud and big data infrastructures;
- And finally, it had to have a user experience designed for “citizen integrators” without sacrificing power, performance and the ability to handle complex, multi-point data and application integration use cases.
Here are the top 5 posts we’re written in the past few months (based on blog traffic) that will help you better understand the architecture of the SnapLogic Elastic Integration Platform, what we value, and why we believe heritage matters when it comes to cloud integration. (Not surprisingly, 3 of the 5 were authored by our Chief Scientist, Greg Benson.)
- Technical Advantages of JSON-centric iPaaS
- Managing Errors in an iPaaS
- Reliability in the SnapLogic iPaaS
- Talking Cloud Integration @ iRobot
- What to Look for in a Modern Integration Platform
We’re glad you find this information useful. You can also download this technical whitepaper for a deeper dive on the SnapLogic Elastic Integration Platform.
Be sure to subscribe to our blog to receive email updates for new posts, and get ready for some exciting news about the SnapLogic Summer 2014 release that we’ll be announcing tomorrow. It’s going to be our most significant release so far!
One of the primary requirements of an integration platform as a service (iPaaS) and why integration heritage matters in the cloud is that the technology is built on modern web standards – specifically JSON and REST.
“It’s easy to put rows and columns into a document, but vice-versa doesn’t work.”
– Craig Stewart, Director of SnapLogic Product Management in this recorded demonstration
SnapLogic deals with documents, rather than just the rows and columns of the traditional ETL type tools. In his post, Technical Advantages of JSON-centric iPaaS, SnapLogic’s Chief Scientist Greg Benson reviews 5 data processing and end users benefits of using documents as our native data type:
- Documents are a better match to modern web services.
- Documents result in more succinct Pipelines.
- A document model allows Pipelines to be loosely coupled. (Watch this customer video to hear about the benefits of “schema-less integration”)
- A document model allows for greater Pipeline reuse.
- Documents are a superset of records.
When it comes to SnapLogic’s foundational support for modern web standards, he writes:
“Our support for documents allows our Snap endpoints to directly consume hierarchical data in native format and send it on to downstream Snaps in a Pipeline. This means that there is no requirement to flatten data into records or to turn a JSON document in a string or BLOB type.”
When it comes to the benefits of a JSON-centric approach supporting structured and unstructured data seamlessly, he concludes, “This native support for documents is one of the many architectural innovations we have developed to help businesses connect both web services and traditional data stores.”
Here is a JSON and Table view of a Twitter Query Snap in the SnapLogic Elastic Integration Platform Designer:
Representational State Transfer (REST)
As we wrote in the Why Buses Don’t Fly Whitepaper, “Mobile and enterprise APIs are primarily exposed over the REST protocol with the data encoded via JSON. From an integration platform perspective, REST and JSON together are increasingly replacing SOAP and XML, making ESBs less relevant in today’s enterprise SMAC architecture.”
SnapLogic’s architecture is entirely REST-based. We have REST Snaps (as displayed in the screenshot to the right) and REST is how the control plane communicates with the data plane (see the post on Software Defined Integration and the high-level architecture diagram below). As we outlined back in 2011, in the early days of building our new Elastic Integration Platform:
“REST lets you publish your data and have others – regardless of where they might be – work with it. Just looking at the URI gives you an indication of how to proceed. Yet despite all of these advantages, basing SnapLogic on REST gave us the same security and massive scalability as the overall Web itself.”
In the next modern integration requirements post, we’ll cover connectivity in more detail.
In this series of posts I’m reviewing 6 essential integration platform as a service (iPaaS) ingredients, as outlined in this post: What to Look for in a Modern Integration Platform. I covered why it’s important that the iPaaS be a fully-functional cloud-based service (based on a software-defined architecture) here. In this post I’ll cover the requirement for a single platform for data, apps and API integration.
We’ve written a lot about the so-called Integrator’s Dilemma faced by enterprise IT organization whose lightweight SaaS apps are being weighed down by their heavy-weight data and application integration tools that were built well before era of big data, social, mobile and cloud computing (SMAC). One CIO I spoke with recently referred to his legacy technologies for connecting SaaS and on-premises applications with each other (and all of his companyies’ disparate data sources) as “boat-anchor integration.” The good news is that innovation is back in the integration market and as sure as death and taxes, disparate silos of integration will converge in the cloud under the more agile iPaaS umbrella. At SnapLogic, we’re talking to customers and prospects about the benefits of an integration platform that is software-defined and built from the ground up to handle batch and streaming requirements. Yes, these can be for very different use cases:
- Batch data integration is typically for getting data in and out of a SaaS app (initial load or scheduled replication) for analytical requirements. The industry has historically called this capability: extraction, transformation and load (ETL), which extends to data cleansing and quality as well as master data management (MDM). The relevance of traditional ETL approaches are now being questioned as Hadoop deployments grow and legacy tools in this category were built to handle rows and columns and structured data integration requirements. A few months ago we ran a popular webinar with Dave Linthicum called Beyond Batch: Is ETL Dead? At the recent Hadoop Summit, the relationship between the data warehouse, SQL and Hadoop was one of the hottest topics. Check out the tweet stream or headlines like this one: Vendors fret as Hadoop encroaches on lucrative data warehouse business.
- Real-time streaming integration is typically for more operational than analytical use cases. Common SaaS application integration uses cases are connecting front-office apps like Salesforce and Workday with back-office financial systems like SAP, Oracle EBS and NetSuite. Typically event-based, SaaS application integration continues to require features like Guaranteed Delivery, but the need to adapt quickly to frequent changes, API updates and the demand for end-user self-service has made legacy approaches inadequate. XML and SOAP-based integration tools in this category were only designed to handle small data – high-frequency, low-latency messages. We’ve written about why the heavy-weight enterprise service bus (ESB) will become less and less relevant for today’s agile iPaaS requirements in this whitepaper – Thoughts on ESBs: Why Buses Don’t Fly in the Cloud. You can also read about the technical advantages of a JSON-centric iPaaS here.
So as we continue to see the convergence of historically distinct integration categories (data, application, process), increasingly we believe a single platform will be the right approach and as such it is one of the primary requirements for a modern iPaaS solution. In my next post I’ll talk about Elastic Scale. In the meantime, you can check out a demonstration of the SnapLogic Elastic Integration Platform here.