This week Jason Bloomberg, industry expert and author of the book, The Agile Architecture Revolution, delivered a great presentation called: It’s the 21st Century, Why Isn’t Your Data Integration Loosely Coupled? He reviews some of the challenges with traditional middleware connectors and tight coupling, where any change in the data format or interface requirements for either end of any interaction requires an update of the connector. The result is a very brittle integration environment with too many single points of failure. (See the iPaaS requirement for fluidity in hybrid deployments.)
Jason goes on to summarize the benefits of loose coupling and review the limitations of Web Services, XML Schemas and rigid, strongly typed data formats. The presentation reviews REST, JSON and what he calls the “schema-less data trap.” He concludes with the following data integration do’s and don’ts:
- Plan ahead for horizontal scalability and fault tolerance (cloud friendliness)
- Favor document-centric data formats
- Use rigid, centralized middleware
- Rely heavily on fixed schemas
Specifically on the topic of the enterprise service bus (ESB), Jason has this to say:
- Essentially the ESB is traditional middleware with Web Services added
- The ESB is typically an older, single-point-of failure, hub-based deployment
- The ESB is not “cloud friendly” and not up to the task – not designed to be horizontally scalable and state information is maintained centrally
He has this to say about SnapLogic “design-time introspection” and our Snaps, which he calls, “next-generation connectors”:
- They can gather the metadata automatically so that integration configuration can be performed dynamically
- SnapLogic lets you automate the configuration of the integration so you have greater flexibility as you deal with changing interactions.
You can check out the entire presentation, which also includes a SnapLogic Elastic Integration Platform demonstration by Craig Stewart, here and I’ve embedded the slides below. Be sure to also check out Jason’s review of SnapLogic Snaps: Re-Inventing Intelligent Connectors and our whitepaper: Why Buses Don’t Fly in the Cloud.
This week SnapLogic Solutions Architect Jason Slater was interviewed about our new MQTT Snaps and Spring 2015 release on the Integration Developer News site. He notes that, “when it comes to Internet of Things (IoT) data, we see 100% alignment with our approach to big data and cloud application integration.”
Here are a few other comments from Jason about how SnapLogic connects data, applications and now things faster:
- “Customers we talk to about their IoT strategy universally accept that traditional data warehousing approaches are not going to be feasible for this kind of big data storage and analytics… you simply can’t get ‘Things’ into a data warehouse.”
- Commenting on SnapLogic’s big data integration capabilities, he noted: “Data scientists [and even business analysts] looking for big data insights don’t need to write Java code and complex queries….From the perspective of end-user productivity and time to value, SnapReduce makes designing complex big data integration easy by dynamically generating MapReduce code behind the scenes, leveraging the Cascading framework. This allows business analysts and data scientists to avoid writing Java code and complex queries and instead focus on what’s most important: gaining insight from their big data.”
- “In addition to MQTT, we’re working toward supporting a number of emerging standards to ensure IoT data is able to be managed effectively as part of an overall big data integration and data lake strategy.”
You can read the entire Integration Developer News (IDN) write up on our Spring 2015 release here: SnapLogic Embraces IoT in Latest iPaaS; Adds Value with Hadoop-Ready Integrations.
Be sure to also register for the IDN web conference on iPaaS in June.
In 2014, Ovum’s Saurabh Sharma, who recently published the Integration Platform as a Service (iPaaS) Decision Matrix, wrote about the lack of Internet of Things (IoT) standards and how this is hindering adoption. This week Loraine Lawson at ITBusinessEdge wrote about the intersection of big data and IoT, pointing to new Forrester and Dimensional Research reports that highlight the challenges of managing unstructured data in the enterprise.
The Spring 2015 release of the SnapLogic Elastic Integration Platform extends our cloud application and big data integration capabilities to IoT with support for Message Queuing Telemetry Transport (MQTT). This means SnapLogic customers can easily build dataflow pipelines that connect to an MQTT broker for sensors, mobile and connected devices and stream data to analytical and other applications in real time. One of the primary use cases for the new MQTT Snaps is rapidly integrating IoT data with other big data sources and enterprise applications for predictive, advanced analytics and data visualization.
Given the nature of our JSON-centric iPaaS, it’s important to note that MQTT is just the first of a set of standards and protocols we’re looking at supporting as more and more customers seek to harness and manage IoT data as part of an overall data lake strategy. Later this year, the SnapLogic iPaaS will support additional IoT protocols such as AMQP, CoAP, OMA, Lightweight M2M and ETSI Smart M2M.
In this demonstration, you can see the new MQTT Snaps in action. The demonstration highlights the platform’s ability to:
- Ingest real-time device messages in real time
- Validate, transform and route messages
- Integrate message data with application or API data
- Persist messages on Hadoop, Amazon Redshift, Google BigQuery and others
- Provide bidirectional communication with devices
- Provide seamless cross-protocol support as we continue to introduce support for new protocols and standards
At SnapLogic, we’re focused on ensuring our customers can go beyond legacy ETL and ESB technologies and connect faster – whether you’re integrating big data, cloud applications or the Internet of Things. We look forward to your feedback on the new MQTT Snaps and what hearing you’d like to see next when it comes to the intersection of big data and IoT.
“The problem with traditional connectors is that they are tightly coupled – any change in the data format or interface requirements for either end of any interaction would require an update of the connector, at the risk of a failed interaction.”
– Jason Bloomberg, President, Intellyx
Join us next Tuesday, May 19th for an interactive webinar with digital transformation and SOA thought leader, Jason Bloomberg. In this webinar we’ll hear from Jason about how connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI back in the 1990s and how the rise of SOA and Web Services was in part intended to resolve the limitations of such traditional connectors, but often fall short. Additional topics covered will include:
- A discussion of the age-old problem of implementing loosely coupled data integration
- An architectural approach to solving this difficult problem
- A demonstration of SnapLogic’s approach to solving the data integration challenge in a scalable and cloud-friendly manner that aligns with modern application architectures
Before joining the webinar next week, you can also review last week’s Spring 2015 release and learn a little more about Jason Bloomberg here:
Jason Bloomberg is the leading industry analyst and expert on achieving agile digital transformation by architecting business agility in the enterprise. He writes for Forbes, Wired, and his biweekly newsletter, the Cortex. As president of Intellyx, he advises business executives on their digital transformation initiatives, trains architecture teams on Agile Architecture, and helps technology vendors and service providers communicate their agility stories. His latest book is The Agile Architecture Revolution.
A few of our past blog posts also address some of the topics we’ll be diving into next week. Check them out:
Register for the webinar here – we look forward to next week’s interactive discussion.
According to a recent Gartner report, “Systems of innovation that leverage mobile and Web applications will demand coherent data integration and application integration techniques to gain just-in-time insights and activate responses across continuously evolving events. Systems of record that interact with a growing range of endpoints (including machine-generated data and IoT-related data) will involve linking of business flows with data access and analysis of event streams.”
Today we announced the general availability of the SnapLogic Spring 2015 release, which expands our focus on big data integration with new support for the Internet of Things (IoT). Along with many new and updated Snaps, the new release introduces a set of Snaps for Message Queuing Telemetry Transport (MQTT), which is a lightweight machine-to-machine connectivity protocol. The new Snaps will allow SnapLogic customers to create pipelines that connect to an MQTT broker for sensors, mobile and connected devices and stream data to analytical and other applications in real time. Later this year, we’re looking at supporting additional IoT protocols such as AMQP, CoAP, OMA Lightweight M2M, and ETSI Smart M2M.
“The Spring 2015 release is yet another milestone for our company as we continue to focus on platform innovation and ensuring our enterprise customers can connect data, applications and things faster.”
– Gaurav Dhillon, founder and CEO, SnapLogic
The Spring 2015 release introduces updates and innovations throughout the SnapLogic Elastic Integration Platform, which has recently been recognized by Ovum as an enterprise integration platform as as service (iPaaS) leader. Spring 2015 updates include:
- Pipeline Execution Statistics: When users execute a pipeline, a duration chart now provides an overview of where the pipeline is spending its time as it is processing the stream of JSON documents. The Pipeline Execution Statistics dialog displays how much CPU time is being consumed and how much memory is being allocated for each Snap in real time, allowing for optimized pipeline design and improved system performance.
- Low-Latency Ultra Pipelines: Administrators can now set up multiple receiving Ultra Pipelines nodes to ensure there is no single point of failure and achieve higher availability and reliability. Ultra Pipelines now have improved error handling and performance monitoring in the SnapLogic dashboard.
- Public Monitoring API: In addition to an API for managing users and groups, a new public monitoring API has been introduced allowing customers to proactively query pipeline status and take advantage of their existing monitoring tools to track the health of SnapLogic integrations.
- Multi-Instance Kerberos Authentication: The SnapLogic Hadooplex runs natively as a YARN application on the Hadoop cluster, whether it is running behind the firewall or in the cloud. The Spring 2015 release introduces support for multiple instances authenticated with different Kerberos users on a single Hadoop cluster.
- Hadoop for Humans: With an easy-to-use cloud-based designer, customers can create SnapReduce pipelines that now support router and copy functions as well as advanced expressions for more complex processing and analytics use cases.
- New and Updated Snaps: The Spring 2015 release introduces new Snaps for Anaplan, Google BigQuery, SAP Hana and Splunk. The Salesforce Snap now integrates with CipherCloud’s security and compliance platform and the release also includes many other updated Snaps for applications and technologies including: AWS Redshift, Eloqua, Excel, Expensify, HP Vertica, HDFS, JDBC, LDAP, Microsoft Active Directory, Microsoft SQL Server, MySQL, NetSuite, OpenAir, Oracle, PostgreSQL, PGP, REST, SAP BAPI, ServiceNow, SOAP, and Tableau.
All SnapLogic Elastic Integration Platform customers have been updated to the Spring 2015 release. To learn more about what’s new, visit: www.snaplogic.com/spring2015.
Last week Ovum introduced their iPaaS Decision Matrix. The report noted:
“The rise of software-as-a-service (SaaS) and the increasing heterogeneity of enterprise applications portfolios and data sources have necessitated a shift to suitable alternatives to traditional integration approaches…Enterprises should consider iPaaS as a means to ease the complexity of hybrid integration as they continue to focus on managing the complex interplay of business needs and persistent budget constraints, and still achieve faster time to integration.”
Welcome to the new era of integration innovation. Is it time to call it Integration 2.0 or are we now at 3.0? I’m not sure, but there is definitely a re-thinking going on in the enterprise as IT organizations seek greater agility, speed and flexibility. As legacy vendors in the market deal with private equity buyouts, waiting for a faster horse is clearly not an option.
Gartner recently outlined 5 reasons to begin converging application and data integration.
Last week Loraine Lawson summarized 5 reasons why integration is changing from David Linthicum’s whitepaper on The Death of Traditional Data Integration. They are:
- Cloud computing
- Mass data storage
- Complex data, such as unstructured data
- A service-based approach to data (which is key to mobile apps and wider use of enterprise analytics)
- Streaming data, mobile data and the related data security challenges
Today, Valerie Silverthorne wrote about Nine questions for hybrid cloud integration. They are:
- What are you not going to move to the cloud?
- Where will most of the future development take place?
- What is the use case?
- What kinds of applications are involved?
- Who will manage the integration?
- How quickly can this happen?
- What’s the next application?
- Can I make do with what I have?
- Should you trust the vendor?
The bottom line? We’re seeing a remarkably fast transition of late, with enterprise IT shifting away from legacy data management and integration technologies to iPaaS. The reason for this is that the confluence of several transformational technologies including social, mobile, analytics/big data, cloud computing and the Internet of Things (SMACT) is finally coming into place.
Big data is evolving as a practice and we are quickly approaching a point at which data will be treated as a single source, which will require a different type of architecture. According to John Myers of Enterprise Management Associates (EMA), this architecture will need to be one that is focused beyond a single platform, where operational and analytical workloads work together. This architecture is called a Hybrid Data Ecosystem.
Join us on Wednesday, April 29th for a live webinar with John, Managing Research Director for EMA’s Business Intelligence practice. This webinar will review the drivers associated with big data implementations, evolving technical requirements for big data environments, and how a robust information management layer is important to big data projects.
During the webinar, we’ll also review how recent EMA research describes the following:
- Use cases that drive big data and the importance of Internet of Things and streaming applications in big data
- The impact of cloud implementation avenues for big data projects
- How the EMA Hybrid Data Ecosystem Information Management Layer coordinates integration between disparate platforms and data types
Register now and join John Myers and the SnapLogic team for this exciting webinar to learn about what constitutes the Hybrid Data Ecosystem – and why it’s a necessity for modern data integration.