This week on TechRepublic, Matt Asay wrote about the rapid rate of adoption of cloud analytics in the enterprise: BI wants to live in the cloud, but IT may not let it. His article shares some great stats and industry analyst insights about cloud analytics adoption (and resistance):

  • According to Howard Dresner’s Wisdom of Crowds Cloud Business Intelligence Market Study, the percentage of enterprises currently using public cloud BI increased by more than 17% from 2013 to 2014, and by more than 53% since 2012.
  • Gartner data showed 45% of surveyed enterprises looking to cloud BI solutions in 2014, up from 30% the previous three years.
  • Gartner analyst Joao Tapadinhas said, enterprises “will move the analytics app closer to the data,” because, he continued, “As more data sources move to the cloud, it makes more sense to also adopt cloud BI solutions because that’s where the data is. It’s easier to connect to cloud data using a cloud solution.”

SnapLogic recently extended our support for powering cloud analytics with a new set of Snaps for Google BigQuery. As explained on their website, “Google BigQuery enables super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure.” SnapLogic for Google BigQuery integration is an intelligent connector, what we call Snaps, that enables customers to transfer up to petabytes of data into and out of Google’s Cloud Platform. With SnapLogic, data can be moved at any latency (batch, real-time, via triggers, and Ultra Pipelines) to meet the requirements of a diverse set of business intelligence users. With the SnapLogic Elastic Integration Platform, citizen integrators can easily extract data from a wide variety of data sources and formats and load it into Google BigQuery. Whether you’re connecting to on-premise applications, social media, mobile and big data sources, SnapLogic allows you to connect faster.

Here’s a demonstration of SnapLogic for Google BigQuery. For more information on this Snap and all of our 300+ cloud and big data integration Snaps, please contact us or check out our YouTube Channel to see SnapLogic in action.

Connect-Faster-Logo-2015According to our TechValidate survey in 2014, “speed and time to value are the primary business drivers for integration platform as a service (iPaaS).” That’s why our focus at SnapLogic is to ensure our customers “Connect Faster.” This means a unified and modern iPaaS that connects data, applications, APIs and Things faster. In recent conversations with SnapLogic customers, I’ve heard the following benefits of a faster cloud and big data integration platform:

  • “Using SnapLogic + AWS Redshift, we reduced our entire delivery system from manufacturing to customer – from 27 to 14 days.”
  • “We went from 60 integration use cases to 6 pipelines. Schema-less integration allowed us to run integrations with systems going through constant schema changes, without the integrations breaking.”
  • “We can do more in two hours with SnapLogic than we could in two days with traditional solutions.”
  • “Take an opportunity to understand SnapLogic’s core platform design. It will enable you to leverage the platform more fully. The fact that data coming onto the platform becomes a JSON document can dramatically change how you manage your integration project for the better.”
  • “In 1 year with an ESB, we only managed to get 5 processes running. In one SnapLogic training session, we were able to build 5 processes that were operational in 4 hours.”
  • “The chief hurdles are in people’s minds: it is not an ETL platform and not an ESB. People familiar with both need to rethink their approach.”

Here’s a summary of how SnapLogic Connect’s Faster:

Connect Data Faster

Just as Hadoop, Spark, NoSQL and the data lake are revolutionizing your enterprise data infrastructure, legacy extraction, transformation and loading (ETL) tools are being re-imagined for modern business analytics. Able to run as a native YARN application, SnapLogic respects data gravity and delivers powerful big data integration with SnapReduce, the Hadooplex and 300+ Snaps.


SnapLopgic-Connect-Cloud-AppsConnect Cloud Applications Faster

Get more value and a faster return on cloud applications like Salesforce, ServiceNow and Workday with low-latency Ultra Pipelines and a self-service design, administration and monitoring interface built for citizen integrators. SnapLogic goes beyond point-to-point integration tools with broad connectivity, orchestration, and a streaming data architecture that allow enterprise IT organizations to connect cloud applications faster.

SnapLogic_Connect-ThingsConnect Things Faster

With MQTT Snaps and support for other Internet of Things (IoT) protocols, SnapLogic customers can rapidly ingest messages directly into a Hadoop cluster. SnapReduce harnesses Hadoop’s scalable processing and generates complex MapReduce code behind the scenes so business analysts and data scientists can focus on what’s most important: gaining insight from their big data.

Next Steps:

The headline writers had a field day recently as Gartner poured cold water on Hadoop’s party with pessimistic adoption survey. It’s not to say Hadoop isn’t happening, but according to Gartner, the pace of production deployments is not keeping up to the hype. Check some of the headlines:

In our big data integration infographic earlier this year, we found similar results: Enterprise IT Uncertainty Around Big Data Initiatives in 2015.

The bottom line in the Gartner survey is that “the skills gaps continue to be a major adoption inhibitor for 57 percent of respondents, while figuring out how to get value from Hadoop was cited by 49 percent of respondents.” When it comes to big data integration, at SnapLogic we believe there’s a need for ”Hadoop for Humans” – technologies that will accelerate adoption and value, without having to hire an army of programmers to wire things up with code. I recently wrote about the different expectations of the data warehousing traditionalists and the Hadoop-centric crowd at big data conferences in this post: The Data Lake: Half Empty or Half Full? A slide presented at Gartner’s recent Business Intelligence Summit explored the persona distinctions this way: “Suits vs. Hoodies.”

So while the promise of Hadoop (and other big data technologies) is clear, is the reality different in your organization? We wanted to go beyond Gartner’s 284 Research Circle member respondents and get a broader view of the enterprise reality. We’ve partnered with leading technology solution providers in the industry to develop a Hadoop Maturity Model. The goal, according to my friend Bruno Aziza who is a self-proclaimed Data Nerd, is to get “a true picture of the “State of the Hadoop Nation.”

large_Hadoop_summit_2015_surveyHere’s how it works: Take 5 mins right now to complete the survey here. Upon survey completion, you will:

  • Receive a score along the Hadoop Maturity Adoption curve
  • Be emailed a complementary copy of the survey’s aggregated results (so you can benchmark your company’s position by industry, geography, etc)
  • Be entered in a drawing for a $100 Amazon gift card

Your responses will remain confidential and will be used only to construct the aggregated results. Click here to take the survey now. In a few weeks, you will also get a copy of the survey results, which will allow you to assess your Hadoop maturity relative to other organizations in your industry.

We’ll be sharing some of the survey results at the Hadoop Summit in San Jose in June. The SnapLogic Team will be there and look forward to seeing you there. More details here.

intellyx_snaplogicThis week Jason Bloomberg, industry expert and author of the book, The Agile Architecture Revolution, delivered a great presentation called: It’s the 21st Century , Why Isn’t Your Data Integration Loosely Coupled? He reviews some of the challenges with traditional middleware connectors and tight coupling, where any change in the data format or interface requirements for either end of any interaction requires an update of the connector. The result is a very brittle integration environment with too many single points of failure. (See the iPaaS requirement for fluidity in hybrid deployments.)

Jason goes on to summarize the benefits of loose coupling and review the limitations of Web Services, XML Schemas and rigid, strongly typed data formats. The presentation reviews REST, JSON and what he calls the “schema-less data trap.” He concludes with the following data integration do’s and don’ts:


  • Plan ahead for horizontal scalability and fault tolerance (cloud friendliness)
  • Favor document-centric data formats


  • Use rigid, centralized middleware
  • Rely heavily on fixed schemas

Specifically on the topic of the enterprise service bus (ESB), Jason has this to say:

  • no-esbEssentially the ESB is traditional middleware with Web Services added
  • The ESB is typically an older, single-point-of failure, hub-based deployment
  • The ESB is not “cloud friendly” and not up to the task – not designed to be horizontally scalable and state information is maintained centrally

He has this to say about SnapLogic “design-time introspection” and our Snaps, which he calls, “next-generation connectors”:

  • They can gather the metadata automatically so that integration configuration can be performed dynamically
  • SnapLogic lets you automate the configuration of the integration so you have greater flexibility as you deal with changing interactions.

You can check out the entire presentation, which also includes a SnapLogic Elastic Integration Platform demonstration by Craig Stewart, here and I’ve embedded the slides below. Be sure to also check out Jason’s review of SnapLogic Snaps: Re-Inventing Intelligent Connectors and our whitepaper: Why Buses Don’t Fly in the Cloud.

This week SnapLogic Solutions Architect Jason Slater was interviewed about our new MQTT Snaps and Spring 2015 release on the Integration Developer News site.  He notes that, “when it comes to Internet of Things (IoT) data, we see 100% alignment with our approach to big data and cloud application integration.”

jason_slater_snaplogicHere are a few other comments from Jason about how SnapLogic connects data, applications and now things faster:

  • “Customers we talk to about their IoT strategy universally accept that traditional data warehousing approaches are not going to be feasible for this kind of big data storage and analytics… you simply can’t get ‘Things’ into a data warehouse.”
  • Commenting on SnapLogic’s big data integration capabilities, he noted: “Data scientists [and even business analysts] looking for big data insights don’t need to write Java code and complex queries….From the perspective of end-user productivity and time to value, SnapReduce makes designing complex big data integration easy by dynamically generating MapReduce code behind the scenes, leveraging the Cascading framework.  This allows business analysts and data scientists to avoid writing Java code and complex queries and instead focus on what’s most important:  gaining insight from their big data.”
  • “In addition to MQTT, we’re working toward supporting a number of emerging standards to ensure IoT data is able to be managed effectively as part of an overall big data integration and data lake strategy.”

You can read the entire Integration Developer News (IDN) write up on our Spring 2015 release here: SnapLogic Embraces IoT in Latest iPaaS; Adds Value with Hadoop-Ready Integrations.

Be sure to also register for the IDN web conference on iPaaS in June.

In 2014, Ovum’s Saurabh Sharma, who recently published the Integration Platform as a Service (iPaaS) Decision Matrix, wrote about the lack of Internet of Things (IoT) standards and how this is hindering adoption. This week Loraine Lawson at ITBusinessEdge wrote about the intersection of big data and IoT, pointing to new Forrester and Dimensional Research reports that highlight the challenges of managing unstructured data in the enterprise.

snaplogic_iotThe Spring 2015 release of the SnapLogic Elastic Integration Platform extends our cloud application and big data integration capabilities to IoT with support for Message Queuing Telemetry Transport (MQTT). This means SnapLogic customers can easily build dataflow pipelines that connect to an MQTT broker for sensors, mobile and connected devices and stream data to analytical and other applications in real time. One of the primary use cases for the new MQTT Snaps is rapidly integrating IoT data with other big data sources and enterprise applications for predictive, advanced analytics and data visualization.

Given the nature of our JSON-centric iPaaS, it’s important to note that MQTT is just the first of a set of standards and protocols we’re looking at supporting as more and more customers seek to harness and manage IoT data as part of an overall data lake strategy. Later this year, the SnapLogic iPaaS will support additional IoT protocols such as AMQP, CoAP, OMA, Lightweight M2M and ETSI Smart M2M.

In this demonstration, you can see the new MQTT Snaps in action. The demonstration highlights the platform’s ability to:

  • Ingest real-time device messages in real time
  • Validate, transform and route messages
  • Integrate message data with application or API data
  • Persist messages on Hadoop, Amazon Redshift, Google BigQuery and others
  • Provide bidirectional communication with devices
  • Provide seamless cross-protocol support as we continue to introduce support for new protocols and standards

At SnapLogic, we’re focused on ensuring our customers can go beyond legacy ETL and ESB technologies and connect faster – whether you’re integrating big data, cloud applications or the Internet of Things. We look forward to your feedback on the new MQTT Snaps and what hearing you’d like to see next when it comes to the intersection of big data and IoT.

LKDN_IntellyxWebinar_180x110“The problem with traditional connectors is that they are tightly coupled – any change in the data format or interface requirements for either end of any interaction would require an update of the connector, at the risk of a failed interaction.”

– Jason Bloomberg, President, Intellyx

Join us next Tuesday, May 19th for an interactive webinar with digital transformation and SOA thought leader, Jason Bloomberg. In this webinar we’ll hear from Jason about how connectors have been a traditional enterprise application integration (EAI) tool since the dawn of EAI back in the 1990s and how the rise of SOA and Web Services was in part intended to resolve the limitations of such traditional connectors, but often fall short. Additional topics covered will include:

  • A discussion of the age-old problem of implementing loosely coupled data integration
  • An architectural approach to solving this difficult problem
  • A demonstration of SnapLogic’s approach to solving the data integration challenge in a scalable and cloud-friendly manner that aligns with modern application architectures

Before joining the webinar next week, you can also review last week’s Spring 2015 release and learn a little more about Jason Bloomberg here:

Jason Bloomberg is the leading industry analyst and expert on achieving agile digital transformation by architecting business agility in the enterprise. He writes for Forbes, Wired, and his biweekly newsletter, the Cortex. As president of Intellyx, he advises business executives on their digital transformation initiatives, trains architecture teams on Agile Architecture, and helps technology vendors and service providers communicate their agility stories. His latest book is The Agile Architecture Revolution.

A few of our past blog posts also address some of the topics we’ll be diving into next week. Check them out:

Register for the webinar here – we look forward to next week’s interactive discussion.