This week on TechRepublic, Matt Asay wrote about the rapid rate of adoption of cloud analytics in the enterprise: BI wants to live in the cloud, but IT may not let it. His article shares some great stats and industry analyst insights about cloud analytics adoption (and resistance):
Gartner data showed 45% of surveyed enterprises looking to cloud BI solutions in 2014, up from 30% the previous three years.
Gartner analyst Joao Tapadinhas said, enterprises “will move the analytics app closer to the data,” because, he continued, “As more data sources move to the cloud, it makes more sense to also adopt cloud BI solutions because that’s where the data is. It’s easier to connect to cloud data using a cloud solution.”
SnapLogic recently extended our support for powering cloud analytics with a new set of Snaps for Google BigQuery. As explained on their website, “Google BigQuery enables super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure.” SnapLogic for Google BigQuery integration is an intelligent connector, what we call Snaps, that enables customers to transfer up to petabytes of data into and out of Google’s Cloud Platform. With SnapLogic, data can be moved at any latency (batch, real-time, via triggers, and Ultra Pipelines) to meet the requirements of a diverse set of business intelligence users. With the SnapLogic Elastic Integration Platform, citizen integrators can easily extract data from a wide variety of data sources and formats and load it into Google BigQuery. Whether you’re connecting to on-premise applications, social media, mobile and big data sources, SnapLogic allows you to connect faster.
According to our TechValidate survey in 2014, “speed and time to value are the primary business drivers for integration platform as a service (iPaaS).” That’s why our focus at SnapLogic is to ensure our customers “Connect Faster.” This means a unified and modern iPaaS that connects data, applications, APIs and Things faster. In recent conversations with SnapLogic customers, I’ve heard the following benefits of a faster cloud and big data integration platform:
“Using SnapLogic + AWS Redshift, we reduced our entire delivery system from manufacturing to customer – from 27 to 14 days.”
“We went from 60 integration use cases to 6 pipelines. Schema-less integration allowed us to run integrations with systems going through constant schema changes, without the integrations breaking.”
“We can do more in two hours with SnapLogic than we could in two days with traditional solutions.”
“Take an opportunity to understand SnapLogic’s core platform design. It will enable you to leverage the platform more fully. The fact that data coming onto the platform becomes a JSON documentcan dramatically change how you manage your integration project for the better.”
“In 1 year with an ESB, we only managed to get 5 processes running. In one SnapLogic training session, we were able to build 5 processes that were operational in 4 hours.”
“The chief hurdles are in people’s minds: it is not an ETL platform and not an ESB. People familiar with both need to rethink their approach.”
Here’s a summary of how SnapLogic Connect’s Faster:
Connect Data Faster
Just as Hadoop, Spark, NoSQL and the data lake are revolutionizing your enterprise data infrastructure, legacy extraction, transformation and loading (ETL) tools are being re-imagined for modern business analytics. Able to run as a native YARN application, SnapLogic respects data gravity and delivers powerful big data integration with SnapReduce, the Hadooplex and 300+ Snaps.
Get more value and a faster return on cloud applications like Salesforce, ServiceNow and Workday with low-latency Ultra Pipelines and a self-service design, administration and monitoring interface built for citizen integrators. SnapLogic goes beyond point-to-point integration tools with broad connectivity, orchestration, and a streaming data architecture that allow enterprise IT organizations to connect cloud applications faster.
With MQTT Snaps and support for other Internet of Things (IoT) protocols, SnapLogic customers can rapidly ingest messages directly into a Hadoop cluster. SnapReduce harnesses Hadoop’s scalable processing and generates complex MapReduce code behind the scenes so business analysts and data scientists can focus on what’s most important: gaining insight from their big data.
The bottom line in the Gartner survey is that “the skills gaps continue to be a major adoption inhibitor for 57 percent of respondents, while figuring out how to get value from Hadoop was cited by 49 percent of respondents.” When it comes to big data integration, at SnapLogic we believe there’s a need for ”Hadoop for Humans” – technologies that will accelerate adoption and value, without having to hire an army of programmers to wire things up with code. I recently wrote about the different expectations of the data warehousing traditionalists and the Hadoop-centric crowd at big data conferences in this post: The Data Lake: Half Empty or Half Full? A slide presented at Gartner’s recent Business Intelligence Summit explored the persona distinctions this way: “Suits vs. Hoodies.”
So while the promise of Hadoop (and other big data technologies) is clear, is the reality different in your organization? We wanted to go beyond Gartner’s 284 Research Circle member respondents and get a broader view of the enterprise reality. We’ve partnered with leading technology solution providers in the industry to develop a Hadoop Maturity Model. The goal, according to my friend Bruno Aziza who is a self-proclaimed Data Nerd, is to get “a true picture of the “State of the Hadoop Nation.”
Here’s how it works: Take 5 mins right now to complete the survey here. Upon survey completion, you will:
Receive a score along the Hadoop Maturity Adoption curve
Be emailed a complementary copy of the survey’s aggregated results (so you can benchmark your company’s position by industry, geography, etc)
Be entered in a drawing for a $100 Amazon gift card
Your responses will remain confidential and will be used only to construct the aggregated results. Click here to take the survey now. In a few weeks, you will also get a copy of the survey results, which will allow you to assess your Hadoop maturity relative to other organizations in your industry.
We’ll be sharing some of the survey results at the Hadoop Summit in San Jose in June. The SnapLogic Team will be there and look forward to seeing you there. More details here.
Jason goes on to summarize the benefits of loose coupling and review the limitations of Web Services, XML Schemas and rigid, strongly typed data formats. The presentation reviews REST, JSON and what he calls the “schema-less data trap.” He concludes with the following data integration do’s and don’ts:
Plan ahead for horizontal scalability and fault tolerance (cloud friendliness)
This week SnapLogic Solutions Architect Jason Slater was interviewed about our new MQTT Snaps and Spring 2015 release on the Integration Developer News site. He notes that, “when it comes to Internet of Things (IoT) data, we see 100% alignment with our approach to big data and cloud application integration.”
“Customers we talk to about their IoT strategy universally accept that traditional data warehousing approaches are not going to be feasible for this kind of big data storage and analytics… you simply can’t get ‘Things’ into a data warehouse.”
Commenting on SnapLogic’s big data integration capabilities, he noted: “Data scientists [and even business analysts] looking for big data insights don’t need to write Java code and complex queries….From the perspective of end-user productivity and time to value, SnapReduce makes designing complex big data integration easy by dynamically generating MapReduce code behind the scenes, leveraging the Cascading framework. This allows business analysts and data scientists to avoid writing Java code and complex queries and instead focus on what’s most important: gaining insight from their big data.”
“In addition to MQTT, we’re working toward supporting a number of emerging standards to ensure IoT data is able to be managed effectively as part of an overall big data integration and data lake strategy.”
The Spring 2015 release of the SnapLogic Elastic Integration Platform extends our cloud application and big data integration capabilities to IoT with support for Message Queuing Telemetry Transport (MQTT). This means SnapLogic customers can easily build dataflow pipelines that connect to an MQTT broker for sensors, mobile and connected devices and stream data to analytical and other applications in real time. One of the primary use cases for the new MQTT Snaps is rapidly integrating IoT data with other big data sources and enterprise applications for predictive, advanced analytics and data visualization.
Given the nature of our JSON-centric iPaaS, it’s important to note that MQTT is just the first of a set of standards and protocols we’re looking at supporting as more and more customers seek to harness and manage IoT data as part of an overall data lake strategy. Later this year, the SnapLogic iPaaS will support additional IoT protocols such as AMQP, CoAP, OMA, Lightweight M2M and ETSI Smart M2M.
In this demonstration, you can see the new MQTT Snaps in action. The demonstration highlights the platform’s ability to:
Ingest real-time device messages in real time
Validate, transform and route messages
Integrate message data with application or API data
Persist messages on Hadoop, Amazon Redshift, Google BigQuery and others
Provide bidirectional communication with devices
Provide seamless cross-protocol support as we continue to introduce support for new protocols and standards
At SnapLogic, we’re focused on ensuring our customers can go beyond legacy ETL and ESB technologies and connect faster – whether you’re integrating big data, cloud applications or the Internet of Things. We look forward to your feedback on the new MQTT Snaps and what hearing you’d like to see next when it comes to the intersection of big data and IoT.
“The problem with traditional connectors is that they are tightly coupled – any change in the data format or interface requirements for either end of any interaction would require an update of the connector, at the risk of a failed interaction.”
– Jason Bloomberg, President, Intellyx
Join us next Tuesday, May 19th for an interactive webinar with digital transformation and SOA thought leader, Jason Bloomberg. In this webinar we’ll hear from Jason about how connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI back in the 1990s and how the rise of SOA and Web Services was in part intended to resolve the limitations of such traditional connectors, but often fall short. Additional topics covered will include:
A discussion of the age-old problem of implementing loosely coupled data integration
An architectural approach to solving this difficult problem
A demonstration of SnapLogic’s approach to solving the data integration challenge in a scalable and cloud-friendly manner that aligns with modern application architectures
Before joining the webinar next week, you can also review last week’s Spring 2015 release and learn a little more about Jason Bloomberg here:
Jason Bloomberg is the leading industry analyst and expert on achieving agile digital transformation by architecting business agility in the enterprise. He writes for Forbes, Wired, and his biweekly newsletter, the Cortex. As president of Intellyx, he advises business executives on their digital transformation initiatives, trains architecture teams on Agile Architecture, and helps technology vendors and service providers communicate their agility stories. His latest book is The Agile Architecture Revolution.
A few of our past blog posts also address some of the topics we’ll be diving into next week. Check them out: