Moving your data warehouse to the cloud: Look before you jump

By Ravi Dharnikota

Where’s your data warehouse? Is it still on-premises? If so, you’re not alone. Way back in 2011, in its IT predictions for 2012 and beyond, Gartner said, “At year-end 2016, more than 50 percent of Global 1000 companies will have stored customer-sensitive data in the public cloud.”

While it’s hard to find an exact statistic on how many enterprise data warehouses have migrated, cloud warehousing is increasingly popular as companies struggle with growing data volumes, service-level expectations, and the need to integrate structured warehouse data with unstructured data in a data lake.

Cloud data warehousing provides many benefits but getting there isn’t easy. Migrating an existing data warehouse to the cloud is a complex process of moving schema, data, and ETL. The complexity increases when restructuring of database schema or rebuilding of data pipelines is needed.

This post is the first in a “look before you leap” three-part series on how to jump-start your migration of an existing data warehouse to the cloud. As part of that, I’ll also cover how cloud-based data integration solutions can significantly speed your time to value.

Beyond basic: The benefits of cloud data warehousing

Cloud data warehousing is a Data Warehouse as a Service (DWaaS) approach that simplifies time-consuming and costly management, administration, and tuning activities that are typical of on-premises data warehouses. But beyond the obvious – data warehouses being stored in the cloud - there’s more. Processing is also cloud-based, and all major solution providers charge separately for storage and compute resources, both of which are highly scalable.

All of which leads us to a more detailed list of key advantages:

  • Scale up (and down): The volume of data in a warehouse typically grows at a steady pace as time passes and history is collected. Sudden upticks in data volume occur with events such as mergers and acquisitions, and when new subjects are added. The inherent scalability of a cloud data warehouse allows you to adapt to growth, adding resources incrementally (via automated or manual processes) as data and workload increase. The elasticity of cloud resources allows the data warehouse to quickly expand and contract data and processing capacity as needed, with no impact to infrastructure availability, stability, performance, and security.
  • Scale out: Adding more concurrent users requires the cloud data warehouse to scale out. You will be able to add more resources – either more nodes to an existing cluster or an entirely new cluster, depending on the situation – as the number of concurrent users rises, allowing more users to access the same data without query performance degradation.
  • Managed infrastructure: Eliminating the overhead of data center management and operations for the data warehouse frees up resources to focus where value is produced: using the data warehouse to deliver information and insight.
  • Cost savings: On-premises data centers themselves are extremely expensive to build and operate, requiring staff, servers, and hardware, networking, floor space, power, and cooling. (This comparison site provides hard dollar data on many data center elements.) When your data warehouse lives in the cloud, the operating expense in each of these areas is eliminated or substantially reduced.
  • Simplicity: Cloud data warehouse resources can be accessed through a browser and activated with a payment card. Fast self-service removes IT middlemen and democratizes access to enterprise data.

In my next post, I’ll do a quick review of additional benefits and then dive into data migration. If you’d like to read all the details about the benefits, techniques, and challenges of migrating your data warehouse to cloud, download the Eckerson Group white paper, “Jump-Start Your Cloud Data Warehouse: Meeting the Challenges of Migrating to the Cloud.

Ravi Dharnikota is Chief Enterprise Architect at SnapLogic. Follow him on Twitter @rdharn1

Integrate through the big data insights gap

By Bill Creekbaum

Whether you’re an analyst, data scientist, CxO, or just a “plain ol’ business user,” having access to more data represents an opportunity to make better business decisions, identify new and innovative opportunities, respond to hard-to-identify threats … the opportunities abound.

More data – from IoT, machine logs, streaming social media, cloud-native applications, and more – is coming at you with diverse structures and in massive volumes at high velocity. Traditional analytic and integration platforms were never designed to handle these types of workloads.

The above data is often associated with big data and tends to be accessible by a very limited audience with a great deal of technical skill and experience (e.g., data scientists), limiting the business utility of having more data. This creates a big data insights gap and prevents a much broader business user and analyst population from big data benefits. Our industry’s goal should be to help business users and analysts operationalize insights from big data. In fact, Forbes has declared that 2017 is the year that big data goes mainstream.

There are two critical elements needed to close this big data insights gap:

  • A scalable data platform: Handles big data that is compatible with “traditional” analytic platforms
  • An integration platform: Acquires large volumes of high-velocity diverse data without IT dependency

To address the first element, Amazon has released Amazon Redshift Spectrum as part of their growing family of AWS big data services. Optimized for massive data storage (e.g., petabytes and exabytes) that leverages S3 and delivered with the scalable performance of Amazon Redshift, AWS is making the above scenarios possible from an operational, accessibility, and economic perspective:

  • Operational: Amazon Redshift Spectrum allows for interaction with data volumes and diversity not possible with traditional OLAP technology.
  • Accessibility: SQL interface allows business users and analysts to use traditional analytic tools and skills to leverage these extreme data sets.
  • Economic: Amazon Redshift Spectrum shifts the majority of big data costs to S3 service which is far more economical than storing the entire data set in Redshift.

Clearly, Amazon has delivered a platform that can democratize the delivery of extremely large volumes of diverse business data to business users and analysts, allowing them to use the tools they currently employ, such as Tableau, PowerBI, QuickSight, Looker, and other SQL-enabled applications.

However, unless the large volumes of high velocity and diverse data can be captured, loaded to S3, and made available via Redshift Spectrum, none of the above benefits will be realized and the big data insights gap will remain.

The key challenges of acquiring and integrating large volumes of high velocity and diverse data:

  • On-prem in a Cloud-Native World: Many integration platforms were designed long ago to operate on-premises and to load data to an OLAP environment in batches. While some have been updated to operate in the cloud, many will fail with streaming workloads and collapse under the high volume of diverse data required today.
  • Integration is an “IT Task”: Typical integration platforms are intended to be used by IT organizations or systems integrators. Not only does this severely limit who can perform the integration work, it will also likely force the integration into a lengthy project queue, causing a lengthy delay in answering critical business questions.

To address the second element in closing the big data insights gap, business users and analysts themselves must be able to capture the “big data” so that business questions can be answered in a timely manner. If it takes a long and complex IT project to capture the data, the business opportunity may be lost.

To close the big data insights gap for business users and analysts, the integration platform must:

  • Handle large volumes of high velocity and diverse data
  • Focus on integration flow development (not complex code development)
  • Comply with IT standards and infrastructure

With the above approach to integration, the practical benefit is that those asking the business questions and seeking insights from having more data are able to leverage the powerful capabilities of Amazon Redshift Spectrum and will be able to respond business opportunities while it still matters.

Amazon’s Redshift Spectrum and the SnapLogic Enterprise Integration Cloud represent a powerful combination to close the big data insights gap for business users and analysts. In upcoming blog posts, we’ll look at actual use cases and learn how to turn these concepts into reality.

Interested in how SnapLogic empowers cloud warehouse users with up to a 10x improvement in the speed and ease of data integration for Redshift deployments, check out the white paper, “Igniting discovery: How built-for-the-cloud data integration kicks Amazon Redshift into high gear.”

Bill Creekbaum is Senior Director, Product Management at SnapLogic. Follow him on Twitter @wcreekba.

Disconnected data is a drag on innovation

By Scott Behles

What do you consider to be a business’ most valuable asset? Is it the cash it holds? Product inventory? Property perhaps? In the pre-internet age, these traditional assets may have supported businesses and could be easily accounted for on an organization’s balance sheet, but the lifeblood of the 21st-century organization is, without question, data.

Whether it’s customer data, financial data, or increasingly machine data, the insights that can be gleaned from an organization’s data repository are invaluable in developing new products and services, deciding the future roadmap for a business, and gaining competitive advantage.

But are businesses taking full advantage of the data at their fingertips? Particularly in larger enterprises with multiple departments, global offices, and disparate IT systems, data often remains relegated to the department that is considered its primary owner. The finance department handles the accounting data while customer data stays with the marketing and sales teams, for instance.

It’s an antiquated way of handling things, and one that means company leaders and other business decision makers rarely see the full picture of what’s going on across the organization, leading to stifled innovation, unforeseen market threats, and missed opportunities.

Convincing business leaders that this is a serious problem can be a tough sell though. Unless you can assign a dollar figure to how significantly disconnected data is negatively impacting a business, you’ll likely not get much of a reaction.

Thankfully, our new Disconnected Data research has done just that.

We surveyed 500 businesses users and IT decision makers in large businesses across the US and UK and found that the wasted time and resources, duplication of work, and missed opportunities caused by disconnected data is collectively costing businesses $140 billion annually.

That stat alone might raise eyebrows, but when we dug a little deeper we uncovered that this issue in large businesses is likely having a far greater impact.

First, more than one-fifth were unaware of what data other departments actually held and one in six didn’t even know how many data sources actually existed. Against this backdrop, it’s even more surprising to learn that, on average, workers were spending more time searching for, acquiring, entering, or moving data than actually analyzing and making decisions on the data. Workers spending most of their time collecting some but not all data, and at the expense of possibly not incorporating it into their decision-making, paints less than a rosy picture for large businesses’ data-driven strategies.

To their credit, most of our respondents are aware of this problem. More than half (57%) admitted that their organization is struggling with data silos and nearly the same percentage said that data silos are a barrier to meeting their organization’s business objectives.

The business objectives most affected? Seizing new opportunities and driving innovation. A shocking 72% felt that siloed data was causing their business to miss out on opportunities, and a third stated that it was holding back innovation in product and services.

For us here at SnapLogic, that last stat is the real stinger. We firmly believe that innovation should be priority #1 for any business that wants to succeed and thrive in today’s fast-moving digital era. Without innovation, products and services won’t evolve which means customers won’t benefit from the latest developments and will start to look elsewhere. If a business can’t innovate, then its days are numbered. If disconnected data is standing in the way of that innovation, it’s a problem that must be solved. And quickly.

To read our complete study on “The High Cost of Disconnected Data,” to get all the details.

Scott Behles is Head of Corporate Communications at SnapLogic. Follow him on Twitter @sbehles

HR transformation: Four takeaways from HR Tech World San Francisco

Diletta D'OnofrioBy Diletta D’Onofrio

Some might argue data is an organization’s most valuable asset. Others will unequivocally say it’s your people. But few will debate this: those companies who manage to bring together the latest in technology with the best of the employee will outperform the competition and be the leaders in the marketplace.

This intersection of people and tech was on full display last week at HR Tech World in San Francisco. I had the privilege of attending the two-day event, where I mingled with Human Resource Officers (HROs), learned from keynote speakers and thought leaders, and tested the latest tech solutions at the vendor booths. From SAP SuccessFactors to Workday, IBM to Cornerstone, Deloitte to ADP – they were all there, making this conference a must-attend for those like me who are interested in exploring the latest ideas, best practices, and technologies to transform HR in the digital era.

Here are a few of my takeaways from my two days at HR Tech World:

  • “Breaking HR”:  So much has changed in the last five years, say nothing of the last 30 years. And yet, as Cisco Chief People Officer Francine Katsoudas explained in her presentation, many global, Fortune 100 organizations are still running HR processes and technologies that were developed for a 1970’s workplace. Katsoudas argued that to capitalize on the promise of the digital age companies must think and work differently, which means incremental HR improvements may need to be pushed aside in favor of a complete overhaul. In other words, we must have the courage to “break HR” if we want to truly lead the Future of Work.
  • Put people at the center: An organization’s digital transformation initiative cannot succeed without effective workforce transformation. Internal processes need to follow the employee experience and not the other way around. Tech-native millennials have new expectations around what makes for a great workplace. Saddling them with archaic processes or legacy systems is a recipe for failure, a sure-fire way to doom any transformation effort. As HR thought leader, author, and LeapGen CEO Jason Averbook noted on stage, “HR technology drives culture,” so invest in the best tech.
  • Tech is getting smarter, faster, easier: It seems like just yesterday that vendors at events such as this one were touting their cloud capabilities as a tech differentiator, but now the cloud is table stakes. Today, the new tech battleground is around artificial intelligence (AI) and machine learning (ML) – you couldn’t walk five feet without hearing new vendor strategies and technology capabilities to help automate and accelerate routine tasks that were once the domain of only humans. “Rethink processes with machine learning,” demanded Yvette Cameron, SVP and Global Head of Strategy at SAP SuccessFactors, in her presentation. She asked, “are your systems and processes ‘continuous, connected, intelligent, and live?'”
  • Data integration – The keys to unlocking HR transformation: To be honest, I got lost a few times in the Exhibition Hall, what with the dozens of vendor booths touting their solutions for recruiting and onboarding, payroll and compensation, learning and development, performance management, workforce planning and analytics, and more. While many vendors offer “HR suites,” with several applications that are said to seamlessly work together, I spoke with several HROs who admitted they’ve got dozens of HR applications in place across their enterprise, and often from multiple vendors. Maybe ADP for payroll, Cornerstone for performance management, Workday for talent management, and so on. Getting them all to work together remains a struggle for most companies, putting a strain on budgets, resources, productivity, and time-to-value. Time and again, I heard – “Integration, integration, integration!” – the companies I met, across all industries and of all sizes, repeatedly said it was a top priority to find better, faster ways to integrate apps and data sources across complete, end-to-end business processes, such as hire-to-retire, for example.

It was a great two days. Lots learned, and lots to do as we continue to help our customers in HR integrate their applications and data sources to accelerate HR transformation. I look forward to next year’s HR Tech World event.

Diletta D’Onofrio leads the Digital Transformation Group at SnapLogic. Follow her on Twitter @ddonofrio13.

Less frosting, more cake: Data integration transforms customer experience

By Nada daVeiga

I’ll start with the frosting. As far as I can tell, it’s been the Year of the Customer for several years now. During this time, every company has gotten the “customer experience” (CX) religion – improve it or die. Thousands of software applications have emerged during what’s now called the Age of the Customer, focused on improving CX by providing the right individual with the right interaction or information, at the right time.

The Age of the Customer has spawned an entirely new software category, marketing technology (martech), chronicled tirelessly by industry analyst Scott Brinker, who goes by @chiefmartec on Twitter. His oft-shared, visual history of the martech product landscape looks like this:

 

 

 

 

 

 

 

 

 

Of the 2017 marketing technology landscape, Brinker notes:

  • There are now 5,381 solutions on the graphic, 39 percent more than last year
  • There are now 4,891 unique companies on the graphic, up 40 percent from last year
  • Only 4.7% of the solutions from 2016 were removed (and another 3.5 percent changed in some fundamental way – their name, their focus, or their ownership)[1]

Where’s the cake?

My point is that there is a lot of frosting here – thousands of applications designed to address the sexiest elements of customer experience. But what’s missing is cake. Data is the cake onto which martech frosting should be added. Integrated enterprise data is the foundation for effective CX strategies to be built on because otherwise, you’re just playing an expensive guessing game.

That’s where enterprise integration comes in. With the expansion of digital channels and new customer initiatives, the variety and volume of customer signals are more diverse than ever. Beyond classical CRM systems around sales and service, understanding the customer lifecycle means bringing together data from, in addition to martech apps, sources including social media, websites, field service, quote management apps, and Internet-enabled things like mobile devices to sensors.

Bake the cake – integrate your enterprise data

More than ever, your company needs to focus on the cake of data, and the enterprise integration required to create it. The good news is, today’s enterprise integration cloud solutions make it easier than ever to build a rich data foundation for comprehensive, effective initiatives in the Age of the Customer.

To learn how to design your integration strategy to enable success with your customer initiatives, download the white paper, “Integration in the age of the customer: The five keys to connecting and elevating customer experience.” In it, you’ll find actionable insights on how to optimize your organization’s data integration strategy for the digital customer, including:

  • Why you need to ensure your organization’s integration strategy is customer-focused
  • How to plan around the entire customer lifecycle
  • Which five integration strategies help speed customer analytics and experience initiatives
  • How to put the odds of customer success in your favor

Download the white paper today!

Nada daVeiga is VP Worldwide Pre-Sales, Customer Success, and Professional Services at SnapLogic. Follow her on Twitter @nrdaveiga.

[1]Marketing Technology Landscape Supergraphic (2017): Martech 5000,” Scott Brinker, May 10, 2017.

 

 

 

Gaurav Dhillon on Nathan Latka’s “The Top” Podcast

Popular podcast host Nathan Latka has a built a large following getting top CEOs, founders, and entrepreneurs to share strategies and tactics that set them up for business success. A data industry veteran and self-described “company-builder,” SnapLogic founder and CEO Gaurav Dhillon was recently invited by Nathan to appear as a featured guest on “The Top.”

Nathan is known for his rapid-fire, straight-to-the-point questioning, and Gaurav was more than up to the challenge. In this episode, the two looked back at Gaurav’s founding of Informatica in the ’90s; how he took that company public and helped it grow to become a billion-plus dollar business; why he stepped away from Informatica and decided to start SnapLogic; how data integration fuels digital business and why customers are demanding modern solutions like SnapLogic’s that are easy to use and built for the cloud; and how he’s building a fast-growing, innovative business that also has it’s feet on the ground.

The two also kept it fun, with Gaurav fielding Nathan’s “Famous Five” show-closing questions, including favorite book, most admired CEO, advice to your 20-year-old self, and more.

You can listen to the full podcast above or via the following links: