Will the Cloud Save Big Data?

This article was originally published on ITProPortal.

Employees up and down the value chain are eager to dive into big data, hunting for golden nuggets of intelligence to help them make smarter decisions, grow customer relationships and improve business efficiency. To do this, they’ve been faced with a dizzying array of technologies – from open source projects to commercial software products – as they try to wrestle big data to the ground.

Today, a lot of the headlines and momentum focus around some combination of Hadoop, Spark and Redshift – all of which can be springboards for big data work. It’s important to step back, though, and look at where we are in big data’s evolution.

In many ways, big data is in the midst of transition. Hadoop is hitting its pre-teen years, having launched in April 2006 as an official Apache project – and then taking the software world by storm as a framework for distributed storage and processing of data, based on commodity hardware. Apache Spark is now hitting its strides as a “lightning fast” streaming engine for large-scale data processing. And various cloud data warehousing and analytics platforms are emerging, from big names (Amazon Redshift, Microsoft Azure HDInsight and Google BigQuery) to upstart players like Snowflake, Qubole and Confluent.

The challenge is that most big data progress over the past decade has been limited to big companies with big engineering and data science teams. The systems are often complex, immature, hard to manage and change frequently – which might be fine if you’re in Silicon Valley, but doesn’t play well in the rest of the world. What if you’re a consumer goods company like Clorox, or a midsize bank in the Midwest, or a large telco in Australia? Can this be done without deploying 100 Java engineers who know the technology inside and out?

At the end of the day, most companies just want better data and faster answers – they don’t want the technology headaches that come along with it. Fortunately, the “mega trend” of big data is now colliding with another mega trend: cloud computing. While Hadoop and other big data platforms have been maturing slowly, the cloud ecosystem has been maturing more quickly – and the cloud can now help fix a lot of what has hindered big data’s progress.

The problems customers have encountered with on-premises Hadoop are often the same problems that were faced with on-premises legacy systems: there simply aren’t enough of the right people to get everything done. Companies want cutting-edge capabilities, but they don’t want to deal with bugs and broken integrations and rapidly changing versions. Plus, consumption models are changing – we want to consume data, storage and compute on demand. We don’t want to overbuy. We want access to infrastructure when and how we want it, with just as much as we need but more.

Big Data’s Tipping Point is in the Cloud

In short, the tipping point for big data is about to happen – and it will happen via the cloud. The first wave of “big data via the cloud” was simple: companies like Cloudera put their software on Amazon. But what’s “truly cloud” is not having to manage Hadoop or Spark – moving the complexity back into a hosted infrastructure, so someone else manages it for you. To that end, Amazon, Microsoft and Google now deliver “managed Hadoop” and “managed Spark” – you just worry about the data you have, the questions you have and the answers you want. No need to spin up a cluster, research new products or worry about version management. Just load your data and start processing.

There are three significant and not always obvious benefits to managing big data via the cloud: 1) Predictability – the infrastructure and management burden shifts to cloud providers, and you simply consume services that you can scale up or down as needed; 2) Economics – unlike on-premises Hadoop, where compute and storage were intermingled, the cloud separates compute and storage so you can provision accordingly and benefit from commodity economics; and 3) Innovation – new software, infrastructure and best practices will be deployed continuously by cloud providers, so you can take full advantage without all the upfront time and cost.

Of course, there’s still plenty of hard work to do, but it’s more focused on the data and the business, and not the infrastructure. The great news for mainstream customers (well beyond Silicon Valley) is that another mega-trend is kicking in to revolutionize data integration and data consumption – and that’s the move to self-service. Thanks to new tools and platforms, “self-service integration” is making it fast and easy to create automated data pipelines with no coding, and “self-service analytics” is making it easy for analysts and business users to manipulate data without IT intervention.

All told, these trends are driving a democratization of data that’s very exciting – and will drive significant impact across horizontal functions and vertical industries. Data is thus becoming a more fluid, dynamic and accessible resource for all organizations. IT no longer holds the keys to the kingdom – and developers no longer control the workflow. Just in the nick of time, too, as the volume and velocity of data from digital and social media, mobile tools and edge devices threaten to overwhelm us all. Once the full promise of the Internet of Things, Artificial Intelligence and Machine Learning begins to take hold, the data overflow will be truly inundating.

The only remaining question: What do you want to do with your data?

Ravi Dharnikota is the Chief Enterprise Architect at SnapLogic. 

Talking “SMACT” With CIOs

SMACTWe recently reviewed the many fall tech events happening in the Bay Area and elsewhere and just got back from a few CIO-specific conferences last week. There have been a wide range of topics covered during recent events, including the next wave of cloud computing, cloud analytics, integration platform as a service (iPaaS), big data and big data integration.

Last week in Miami Beach we were at the Technology Business Management (TBM) Conference, which brought together Global 2000 CIOs, CTOs and CFOs to hear from peers and learn about new solutions on the market that can help modernize and transform enterprise IT organizations. While at the TBM Conference we had the opportunity to speak with CIOs and other IT leaders about how SnapLogic customers have reduced the complexity of their integrations by up to 85% and connected data, applications and APIs at least 4x faster. We also discussed common use cases for our platform such as cloud and on-premises application integration, digital marketing, big data analytics, self-service for “citizen integrators,” and the establishment of an agile enterprise integration layer or fabric.

old vs. newAt the Midmarket CIO Forum in Tucson, also last week, we met with CIOs from a variety of industries and reviewed the new data, application and API integration challenges that are facing today’s IT leaders. We discussed the “Integrator’s Dilemma” and how older, more traditional ETL tools and approaches to integration aren’t built for the new data challenges. Lastly, we talked about how to avoid getting “SMACT” summed up by the following:

  • Don’t settle for SO SO (same old, same old)
  • The first step to solving the Integrator’s Dilemma is recognizing it exists!
  • When it comes to Social, Mobile, Analytics, Cloud computing and the Internet of Things  wait to integrate!

You can find our presentation slides here. Also be sure to check out some of the social buzz below from TBM Conference and the Midmarket CIO Forum, as well as our infographic here on why CIOs are getting “SMACT”:

[Infographic] Why Are CIOs Getting SMACT?

According to industry analysts, within 5 years the CMO will spend more on IT than the CIO. Check out our new infographic, which explains why CIOs are getting “SMACT” due to the adoption of Social, Mobile, Analytics and Big Data, Cloud Computing and the Internet of Things, and provides some compelling statistics in each of these categories. You can also learn more here about the SnapLogic Elastic Integration Platform, delivering real-time, event-driven application integration and batch-oriented and streaming big data integration for analytics in a single platform.

[Infographic] Why Are CIOs Getting SMACT?

Here are the sources for all of the SMACT Infographic stats:

In addition to downloading a PDF of the above infographic here, check out some additional SnapLogic resources below all about the new world of social, mobile, analytics and big data, cloud computing and the Internet of Things:

Technology Conference Season Has Arrived!

It’s back to school for the kids and back to the conference center for technology companies and their customers, employees, and partners. With a focus primarily on cloud applications and platforms, big data and analytics, here’s a list of September events in the US that we’re tracking at SnapLogic. What are we missing? Where else should we be? This list of Big Data events is a helpful resource and there’s a good overview of cloud computing events here. Safe travels!

SnapLogic Recognized as a Cloud and Big Data Leader

As we continue to roll out our Elastic Integration Platform to more and more enterprise customers, I’m excited to say that SnapLogic has been recognized this week by two prominent publications. Here’s a summary: sandhill_logoThe Sand Hill Cloud 50 represents a unique set of players of various sizes and hues that span SaaS, Paas and IaaS, security, storage and services spaces and that stand out from the crowd with a unique differentiation and value proposition.” According to Sand Hill,  the 50 “represents a unique set of cloud players that stand out from the crowd.” Here’s what they said about SnapLogic:

“Integration platform as a service that connects cloud applications, APIs and disparate data sources with the rest of the enterprise. Named a Visionary in the Gartner Magic Quadrant in the integration platform as a service category. Customers include Netflix, CapitalOne, iRobot and Acxiom. The company enjoys 100 percent growth in YoY bookings.”

DBTA_100Database Trends and Applications magazine introduced the second annual DBTA 100 list of companies that matter most in data. According to the DBTA, “with organizations increasingly seeking to become data-driven entities—companies that actually use the data they are amassing for competitive advantage—DBTA set out to recognize innovative providers of hardware, software, and services….The 100 companies that matter in data comprises both seasoned veterans and disruptive new vendors.” As part of the DBTA 100, we had the opportunity to publish a “View From the Top” company overview. Here’s my summary of SnapLogic and what we’re setting out to do: Enterprise IT organizations today are facing a dilemma—their legacy integration technologies were built before the era of big data, social, mobile and cloud (SMAC) computing and simply can’t keep up. With respect to Clayton Christensen and his book The Innovators Dilemma, we call this The Integrator’s Dilemma. In 2013 SnapLogic introduced the industry’s first Elastic Integration platform as a service (iPaaS). Fast, multi-point and modern, it’s built from the ground up to handle today’s data, application and API connectivity challenges.

  • Data Integration as a Service:  Don’t let old ETL slow down your new analytics. SnapLogic goes beyond rows and columns-centric tools while providing pre-built Snaps for Tableau, Amazon Redshift and big data sources.
  • Cloud Application Integration: Got Salesforce? Workday? ServiceNow? What about SAP and Oracle? Whether it’s cloud-to-cloud or cloud-to-ground connectivity, if you’ve got SaaS, we’ve got Snaps.
  • APIs and the Internet of Things: Keep up to date and be ready for what’s next with Elastic Integration. With over 160 pre-built connectors, called Snaps, and a software-defined multi-tenant architecture that respects data gravity, the SnapLogic Elastic Integration Platform is ideally suited for today’s hybrid IT environments. It’s the Integrator’s Solution for the SMAC era.

I should also note that SnapLogic is included in this week’s BVP Cloudscape: Top 300 Private Cloud Companies. It’s a pretty impressive list of cloud and big data companies. We’re also listed in the 2014 OnDemand 100 Top Private Companies.

To learn more about SnapLogic, visit SnapLogic.com/resources or Contact Us.

Cloud Speed the Primary Business Driver for Integration Platform as a Service (iPaaS)

TechValidate SnapLogic survey resultsThis week we published the results of a survey we ran in March with TechValidate, which asked about the barriers to software-as-a-service (SaaS) adoption and the business and technical drivers for cloud-based integration services. We’ll be reviewing the details of the research in a webinar on April 25th, which will also provide a detailed overview of the SnapLogic Integration Cloud.

Here are some of the key findings from the research:

  • 56% of survey respondents are running four or more SaaS applications.
  • 43% prioritized application and data integration challenges as a barrier to SaaS application adoption in their companies.
  • 59% of survey respondents listed speed or time to value as the primary business driver for a cloud integration service.
  • 52% said a modern and scalable architecture was the primary technical requirement of a iPaaS.

When asked about the challenges of legacy integration tools for cloud integration (see our my colleagues post on Why Integration Heritage Matters and his summary of Why the Enterprise Service Bus Doesn’t Fly in the Cloud), 43% took issue with the requirement for costly hardware purchases and software installation and configuration. 37% found on-premise integration tools to be too expensive due to the perpetual licensing model and 35% noted that change management is painful where end point changes mean integration re-work.

Integration Platform as a Service

As I noted in the press release, the results of this TechValidate survey are in line with the conversations we’re having with our customers, partners and prospects. As SaaS application, analytics and API adoption grows in the enterprise, the ability to connect with other systems is the essential ingredient to long-term customer success. Integration should be a cloud accelerator not a bottleneck, which is why increasingly companies of all sizes are looking for modern, elastic integration alternatives to power their cloud services initiatives.

I hope you can join us for the webinar next week. You can also download the complete survey results here.

Fast Just Got Faster: Snap Patterns Accelerate Integrations, Starting with Amazon Redshift

The SnapLogic team is very excited to announce our new SnapLogic Integration Cloud Free Trial for Amazon Redshift at the AWS Summit in San Francisco. Built on the the most recent innovation from SnapLogic called Snap Patterns, which are are pre-built packaged integrations that customers can configure through a step-by-step wizard, our goal is to help you accelerate your time to value with AWS Redshift by as much as 10x. There’s no coding necessary!

The free trial comes with Snap Patterns for some commonly recurring data integration requirements for Amazon Redshift that are common challenges for cloud data warehousing customers. For starters, the Redshift Loader Patterns take away the complexity of the initial load from Amazon RDS for MySQL, Oracle, SQL Server and PostgreSQL into Amazon Redshift. Users of the SnapLogic Snap Patterns can then use MicroStrategy, Qlikview, Tableau, Tidemark or other leading business intelligence (BI) tools to run analytics and visualize their data in Redshift.

SnapLogic Free Trial for Amazon RedshiftSnap Patterns for Amazon Redshift allow organizations to:

  • Accelerate cloud data warehouse adoption with prebuilt patterns that can be configured by an automatically-generated series of steps.

  • Rapidly connect Amazon Redshift to a variety of relational database services including Amazon RDS for MySQL, PostgreSQL, Oracle and SQL Server.

  • Quickly load data into an Amazon S3 bucket and kick off the Amazon Redshift import process in a single step.

  • Easily replicate source tables into their Amazon Redshift clusters and detect daily changes to keep data synchronized.

  • Take advantage of core REST and SOAP Snaps for broader connectivity.

  • Visually design a variety of data operations using a set of core Snaps such as Binary, Flow, Script, Transform and XML.

  • Do sophisticated extract, transform and load (ETL) operations such as slowly changing dimensions type 2 (SCD2) and database lookups without any coding.

  • Start with a free trial of an easy to use wizard and upgrade to the full SnapLogic integration platform as a service (iPaaS) in order to connect to data from Salesforce.com, Teradata, Netezza, SAP, Oracle ERP and other systems without losing any of their existing work.

If you’re ready to get started with the trial, check out the following resources:

Additional resources are available here:

We look forward to hearing your feedback as this trial powers your cloud analytics initiatives. And if you happen to be at the AWS Summit in San Francisco today, please swing by our booth (#621) for more information!