Integrate through the big data insights gap

By Bill Creekbaum

Whether you’re an analyst, data scientist, CxO, or just a “plain ol’ business user,” having access to more data represents an opportunity to make better business decisions, identify new and innovative opportunities, respond to hard-to-identify threats … the opportunities abound.

More data – from IoT, machine logs, streaming social media, cloud-native applications, and more – is coming at you with diverse structures and in massive volumes at high velocity. Traditional analytic and integration platforms were never designed to handle these types of workloads.

The above data is often associated with big data and tends to be accessible by a very limited audience with a great deal of technical skill and experience (e.g., data scientists), limiting the business utility of having more data. This creates a big data insights gap and prevents a much broader business user and analyst population from big data benefits. Our industry’s goal should be to help business users and analysts operationalize insights from big data. In fact, Forbes has declared that 2017 is the year that big data goes mainstream.

There are two critical elements needed to close this big data insights gap:

  • A scalable data platform: Handles big data that is compatible with “traditional” analytic platforms
  • An integration platform: Acquires large volumes of high-velocity diverse data without IT dependency

To address the first element, Amazon has released Amazon Redshift Spectrum as part of their growing family of AWS big data services. Optimized for massive data storage (e.g., petabytes and exabytes) that leverages S3 and delivered with the scalable performance of Amazon Redshift, AWS is making the above scenarios possible from an operational, accessibility, and economic perspective:

  • Operational: Amazon Redshift Spectrum allows for interaction with data volumes and diversity not possible with traditional OLAP technology.
  • Accessibility: SQL interface allows business users and analysts to use traditional analytic tools and skills to leverage these extreme data sets.
  • Economic: Amazon Redshift Spectrum shifts the majority of big data costs to S3 service which is far more economical than storing the entire data set in Redshift.

Clearly, Amazon has delivered a platform that can democratize the delivery of extremely large volumes of diverse business data to business users and analysts, allowing them to use the tools they currently employ, such as Tableau, PowerBI, QuickSight, Looker, and other SQL-enabled applications.

However, unless the large volumes of high velocity and diverse data can be captured, loaded to S3, and made available via Redshift Spectrum, none of the above benefits will be realized and the big data insights gap will remain.

The key challenges of acquiring and integrating large volumes of high velocity and diverse data:

  • On-prem in a Cloud-Native World: Many integration platforms were designed long ago to operate on-premises and to load data to an OLAP environment in batches. While some have been updated to operate in the cloud, many will fail with streaming workloads and collapse under the high volume of diverse data required today.
  • Integration is an “IT Task”: Typical integration platforms are intended to be used by IT organizations or systems integrators. Not only does this severely limit who can perform the integration work, it will also likely force the integration into a lengthy project queue, causing a lengthy delay in answering critical business questions.

To address the second element in closing the big data insights gap, business users and analysts themselves must be able to capture the “big data” so that business questions can be answered in a timely manner. If it takes a long and complex IT project to capture the data, the business opportunity may be lost.

To close the big data insights gap for business users and analysts, the integration platform must:

  • Handle large volumes of high velocity and diverse data
  • Focus on integration flow development (not complex code development)
  • Comply with IT standards and infrastructure

With the above approach to integration, the practical benefit is that those asking the business questions and seeking insights from having more data are able to leverage the powerful capabilities of Amazon Redshift Spectrum and will be able to respond business opportunities while it still matters.

Amazon’s Redshift Spectrum and the SnapLogic Enterprise Integration Cloud represent a powerful combination to close the big data insights gap for business users and analysts. In upcoming blog posts, we’ll look at actual use cases and learn how to turn these concepts into reality.

Interested in how SnapLogic empowers cloud warehouse users with up to a 10x improvement in the speed and ease of data integration for Redshift deployments, check out the white paper, “Igniting discovery: How built-for-the-cloud data integration kicks Amazon Redshift into high gear.”

Bill Creekbaum is Senior Director, Product Management at SnapLogic. Follow him on Twitter @wcreekba.

SnapLogic Sits Down with theCUBE at AWS re:Invent to Talk Self-Service Cloud Analytics

SnapLogic was front-and-center at AWS re:Invent last week in Las Vegas, with our team busier than ever meeting with customers and prospects, showcasing our solutions at the booth, and networking into the evening with event-goers interested in all things Cloud, AWS integration and SnapLogic.

Ravi Dharnikota, SnapLogic’s Head of Enterprise Architecture and Big Data Practice, took time out to stop by and visit with John Furrier, co-founder of the live video interview show theCUBE.  Ravi was joined by Matt Glickman, VP of Products with our partner Snowflake Computing, for a wide-ranging discussion on the changing customer requirements for effective data integration, SaaS integration, warehousing and analytics in the cloud.  

The roundtable all agreed — organizations need fast and easy access to all data, no matter the source, format or location — and legacy solutions built for a bygone era simply aren’t cutting it.  Enter SnapLogic and Snowflake, each with a modern solution designed from the ground-up to be cloud-first, self-service, fully scalable and capable of handling all data. Customers using these solutions together — like Kraft Group, owners of the New England Patriots and Gillette Stadium — enjoy dramatic acceleration in time-to-value at a fraction of the cost by eliminating manual configuration, coding and tuning while bringing together diverse data and taking full advantage of the flexibility and scalability of the cloud.

To make it even easier for customers, SnapLogic and Snowflake recently announced tighter technology integration and joint go-to-market programs to help organizations harness all data for new insights, smarter decisions and better business outcomes.

To watch the full video interview on theCUBE, click here.

Don’t Let Cloud Be Another Silo: Accelerate Your AWS data integration

Gone are the days when enterprises had all of their apps and data sources on-premises. Today is the era of big data, cloud and hybrid deployments. More and more enterprises are rapidly adopting different SaaS applications and hosting their solutions in public clouds including Amazon Web Services and Microsoft Azure. But soon enterprises realize that their SaaS applications and on-premises data sources are not integrated with their public cloud footprint and the integration itself becomes an expensive and time consuming undertaking.

Continue reading “Don’t Let Cloud Be Another Silo: Accelerate Your AWS data integration”

SnapLogic Roadshow – Is The Data Warehouse Dead?

Short answer to that provocative question: no – it’s just changing. Our data integration road show hit four U.S. cities over the past two weeks. The keynote presentation was delivered by James Markarian, SnapLogic’s CTO. He shared his perspective on modern data management, the role of the data warehouse in a hybrid cloud environment, and important considerations for an enterprise data lake strategy.

James Markarian presents at the SnapLogic roadshow July 2016.
James Markarian presents at the SnapLogic roadshow July 2016.

 

We were also joined by our partner, Amazon Web Services, who highlighted their cloud data management platform, with an emphasis on Amazon Redshift. Together we highlighted some of the joint SnapLogic/AWS success stories around hybrid cloud data integration, including GameStop, Box, CapitalOne, and eero.

The bulk of James’ keynote focused on the changes to the data landscape that are affecting the role and structure of the data warehouse.  He described “data warehousing 1.0” from the 1980’s which introduced a convenient, single place to warehouse all data, but which was expensive and reliant on scripting to integrate sources. He contrasted that with “data warehousing 2.0” of the 1990’s which saw the rise of ETL processes and data marts, but which was still rigid and typically on-premises. Since that era, however, data warehousing has remained generally static. Now, with the dramatic increase in unstructured/polystructured data, plus cloudification of data sources, data warehousing 2.0 has fallen a bit short. Enter the data lake. James cautioned the audience not to think of a data lake as an amorphous, no-rules dumping ground for unstructured data. Instead, he identified multiple “zones” within the data lake, each of which has certain requirements, rules and uses.

Finally, James illustrated how lakeshore data marts and cloud-based data warehouses – connected using SnapLogic – address some of the risk and high labor costs that can be associated with Hadoop-centric data lakes.

Attendees – some of whom were already far along in their data lake adoption journey and some still in a “data warehouse 2.0” environment – certainly left with food for thought. Watch this space for the next time SnapLogic comes to a city near you.

In the mean time, don’t forget to subscribe to our data management podcast series called SnapTalk.

Yelp Selects SnapLogic For Redshift, Salesforce and Workday Integration

I recently had the pleasure of chatting with SnapLogic customer Yelp. Given the nature of their business, Yelp had a lot of customer data that they needed to process and act upon quickly in order to optimize their revenue streams. They decided to adopt the Amazon Redshift data warehouse service to give them the analytics speed and flexibility they needed. So the next question was: how to get the data into Redshift efficiently.

Once they discovered that SnapLogic had the out-of-the-box connectors they needed — not only for Redshift but for data sources Salesforce and Workday — it came down to build versus buy. They could build the integrations using in-house resources, but there was an opportunity cost and speed penalty that came with a DIY approach. In the end, they chose SnapLogic and estimate that they cut development time in half.

And they’re not done – they are connecting Workday with Redshift next. Yelp told me, “Looking ahead, we’re planning to deploy the Workday Snap to connect our human resources data to Redshift. SnapLogic has proven to be a tremendous asset.” Sounds like a 5-star review. Read more here.

The View From The Cloud With Amazon Redshift

SnapLogic continues to celebrate our partnership with Amazon Web Services with a variety of in-person and online events. The SnapLogic Elastic Integration Platform helps customers integrate Amazon database services with any other data source in the cloud, on-premises or in hybrid mode, for faster, more effective decision-making.

On July 22 we are very pleased to co-present a webinar with Amazon titled “Weathering the Data Storm: How SnapLogic and AWS Delivered Analytics in the Cloud for Earth Networks.” Our mutual customer, Earth Networks, will share their story of moving to a self-service cloud for data integration, preparation and analytics.

We’re also a proud sponsor of several regional AWS Summits. Last week we participated in the Chicago event, and we’ll be at the New York City event on July 9. These have been great opportunities to talk with members of the growing AWS community and share how companies are using SnapLogic with cloud-based database services.

At these events and on our website we have been showcasing our Amazon Redshift integration. This integration helps customers quickly transfer up to petabytes of data into and out of Redshift. Data can be moved at any latency, giving customers the flexibility to handle a variety of use cases. We’re currently offering a free trial for Redshift, where users can log in and begin loading data right away.

Please check out the trial and let us know what you think.

SNAP IN…to Data, Apps and APIs

A few months ago we published the results of a TechValidate survey that found that Cloud Speed and Time to Value are the number one drivers for adopting a modern integration platform as a service (iPaaS). Our team has put together a new series of graphics to highlight the benefits of being able to SNAP IN…to the past, present and future of your data, applications and APIs. I’ve posted a few of the graphics below created by Roger Chan, our graphic design leader. You can download the results from the TechValidate research here.

SNAP IN…to Data, Apps, and APIs

Snap In to Your Data, Apps, APIs

 

 

 

 

 

SNAP IN…to AWS Redshift and RDS

Redshift Integration

 

 

 

 

 

SNAP IN…to Workday

workday integration

 

 

 

 

 

SNAP IN…to Big Data

Snap In to Big Data

 

 

 

 


SNAP IN…to Salesforce

Snap IN Salesforce1

 

 

 

 

 

Snap Integration from SnapLogic.

And because it’s Friday, here’s one from one of my favorite movies….

snaplogic integration

 

 

 

 

 

To learn more about SnapLogic elastic integration for big data, SaaS and on-premises applications and APIs visit SnapLogic.com. Have a great Labor Day weekend!