Will the Cloud Save Big Data?

This article was originally published on ITProPortal.

Employees up and down the value chain are eager to dive into big data, hunting for golden nuggets of intelligence to help them make smarter decisions, grow customer relationships and improve business efficiency. To do this, they’ve been faced with a dizzying array of technologies – from open source projects to commercial software products – as they try to wrestle big data to the ground.

Today, a lot of the headlines and momentum focus around some combination of Hadoop, Spark and Redshift – all of which can be springboards for big data work. It’s important to step back, though, and look at where we are in big data’s evolution.

In many ways, big data is in the midst of transition. Hadoop is hitting its pre-teen years, having launched in April 2006 as an official Apache project – and then taking the software world by storm as a framework for distributed storage and processing of data, based on commodity hardware. Apache Spark is now hitting its strides as a “lightning fast” streaming engine for large-scale data processing. And various cloud data warehousing and analytics platforms are emerging, from big names (Amazon Redshift, Microsoft Azure HDInsight and Google BigQuery) to upstart players like Snowflake, Qubole and Confluent.

The challenge is that most big data progress over the past decade has been limited to big companies with big engineering and data science teams. The systems are often complex, immature, hard to manage and change frequently – which might be fine if you’re in Silicon Valley, but doesn’t play well in the rest of the world. What if you’re a consumer goods company like Clorox, or a midsize bank in the Midwest, or a large telco in Australia? Can this be done without deploying 100 Java engineers who know the technology inside and out?

At the end of the day, most companies just want better data and faster answers – they don’t want the technology headaches that come along with it. Fortunately, the “mega trend” of big data is now colliding with another mega trend: cloud computing. While Hadoop and other big data platforms have been maturing slowly, the cloud ecosystem has been maturing more quickly – and the cloud can now help fix a lot of what has hindered big data’s progress.

The problems customers have encountered with on-premises Hadoop are often the same problems that were faced with on-premises legacy systems: there simply aren’t enough of the right people to get everything done. Companies want cutting-edge capabilities, but they don’t want to deal with bugs and broken integrations and rapidly changing versions. Plus, consumption models are changing – we want to consume data, storage and compute on demand. We don’t want to overbuy. We want access to infrastructure when and how we want it, with just as much as we need but more.

Big Data’s Tipping Point is in the Cloud

In short, the tipping point for big data is about to happen – and it will happen via the cloud. The first wave of “big data via the cloud” was simple: companies like Cloudera put their software on Amazon. But what’s “truly cloud” is not having to manage Hadoop or Spark – moving the complexity back into a hosted infrastructure, so someone else manages it for you. To that end, Amazon, Microsoft and Google now deliver “managed Hadoop” and “managed Spark” – you just worry about the data you have, the questions you have and the answers you want. No need to spin up a cluster, research new products or worry about version management. Just load your data and start processing.

There are three significant and not always obvious benefits to managing big data via the cloud: 1) Predictability – the infrastructure and management burden shifts to cloud providers, and you simply consume services that you can scale up or down as needed; 2) Economics – unlike on-premises Hadoop, where compute and storage were intermingled, the cloud separates compute and storage so you can provision accordingly and benefit from commodity economics; and 3) Innovation – new software, infrastructure and best practices will be deployed continuously by cloud providers, so you can take full advantage without all the upfront time and cost.

Of course, there’s still plenty of hard work to do, but it’s more focused on the data and the business, and not the infrastructure. The great news for mainstream customers (well beyond Silicon Valley) is that another mega-trend is kicking in to revolutionize data integration and data consumption – and that’s the move to self-service. Thanks to new tools and platforms, “self-service integration” is making it fast and easy to create automated data pipelines with no coding, and “self-service analytics” is making it easy for analysts and business users to manipulate data without IT intervention.

All told, these trends are driving a democratization of data that’s very exciting – and will drive significant impact across horizontal functions and vertical industries. Data is thus becoming a more fluid, dynamic and accessible resource for all organizations. IT no longer holds the keys to the kingdom – and developers no longer control the workflow. Just in the nick of time, too, as the volume and velocity of data from digital and social media, mobile tools and edge devices threaten to overwhelm us all. Once the full promise of the Internet of Things, Artificial Intelligence and Machine Learning begins to take hold, the data overflow will be truly inundating.

The only remaining question: What do you want to do with your data?

Ravi Dharnikota is the Chief Enterprise Architect at SnapLogic. 

Cloud fluency: Does your data integration solution speak the truth?

There’s been a lot of cloud-washing in the enterprise data integration space — vendors are heavily promoting their cloud solutions, yet for many, only a skinny part of their monolithic apps has been “cloudified.”

In an era of “alternative facts,” it’s important to make technology decisions based on truths. Here is an important one: A big data integration solution built on genuine, made-for-the-cloud platform as a service (PaaS) technology offers important benefits including:

  1. Self-service integration by “citizen integrators,” without reliance on IT
  2. For IT organizations, the ability to easily connect multiple data sets, to achieve a bespoke enterprise tech environment

These are in addition to the traditional benefits of cloud solutions: no on-premise installation; continuous, no-fuss upgrades; and the latest software innovation, delivered automatically.

Why “built for the cloud” matters

You can’t get these benefits with “cloudified” software that was originally invented in 1992. Of course, I’m referring to Informatica; while the company promotes its cloud capabilities, the software largely retains a monolithic architecture that resides on-premises, and does most of its work there, too.

In contrast, SnapLogic is purpose-built for the cloud, meaning there are no legacy components that prevent the data and application integration service from running at cloud speed. Data streams between applications, databases, files, social and big data sources via the Snaplex, a self-upgrading, elastic execution grid.

In more everyday terms, SnapLogic has 100% cloud fluency. Our technology was made for the cloud, born in the cloud, and it lives in the cloud.

The consumerization of data integration

Further to point 1 above, “citizen integrators,” industry futurists like R. “Ray” Wang have been talking about the consumerization of IT for more than half a decade. And that is exactly what SnapLogic has mastered. Our great breakthrough, our big innovation, is that we have consumerized the dungeon-like, dark problem of data integration.

Integration used to be a big, boring problem relegated to the back office. We’ve brought it from the dungeon to the front office and into the light. It is amazing to see how people use our product. They go from one user to hundreds of users as they get access to data in a secure, organized and appropriately access-controlled manner. But you don’t have a cast of thousands of IT people enabling all this; users merely help themselves. This is the right model for the modern enterprise.

“An ERP of one”

As for the second major benefit of a true cloud solution — a bespoke enterprise tech environment, at a fraction of the time and cost of traditional means — here’s a customer quote from a young CEO of a hot company that’s a household name.

“Look, we’ve got an ‘ERP of one’ by using SnapLogic — a totally customized enterprise information environment. We can buy the best-of-the-best SaaS offerings, and then with SnapLogic, integrate them into a bespoke ERP system that would cost a bajillion dollars to build ourselves. We can custom mix and match the capabilities that uniquely fit us. We got the bespoke suit at off-the-rack prices by using SnapLogic to customize our enterprise environment.”

To my mind, that’s the big payoff, and an excellent way to think about SnapLogic’s value. We are able to give our customer an “ERP of one” faster and cheaper than they could have ever imagined. This is where the world is going, because of the vanishingly low prices of compute power and storage, and cloud computing.

Today you literally can, without a huge outlay, build your own enterprise technology world. But you need the glue to realize the vision, to bring it all together. That glue is SnapLogic.

Find out more about how and why SnapLogic puts best-of-breed enterprise integration within every organization’s grasp. Register for this upcoming webinar featuring a conversation with myself, industry analyst and data integration expert David Linthicum, and Gaurav Dhillon, SnapLogic’s CEO and also an Informatica alumnus: “We left Informatica. Now you can, too.”

SNP_Thumb_Informatica

Gaurav Dhillon is CEO at SnapLogic. Follow him on Twitter @gdhillon.

Enterprise IoT: Watching Cat Videos Without Getting Caught (or, How I Learned to Stop Looking Over My Shoulder and Trust the CEO Proximity Alert

We have a slight problem at SnapLogic. While we spend a vanishingly small percent of the day watching adorable cat videos on the Internet, it seems our CEO always shows up behind our desks while doing so. If only we knew when our CEO was nearby and could get an alert when he was.

Continue reading “Enterprise IoT: Watching Cat Videos Without Getting Caught (or, How I Learned to Stop Looking Over My Shoulder and Trust the CEO Proximity Alert”

SnapLogic Live: Enabling the Citizen Integrator

In the midst of some big announcements recently – our new Partner Connect program, kicking off a July data warehouse roadshow and yesterday’s Top Places to work award – we’d like to take time during this week’s SnapLogic Live to review our definition of the term ”citizen integrator,” and how to enable someone in that position to facilitate the various integration demands within their company.

Continue reading “SnapLogic Live: Enabling the Citizen Integrator”

What You Need to Know About Modern Data Integration

dave_linthicumLast week we hosted a webinar with industry analyst, thought leader and author David Linthicum that focused how the enterprise problem domains are changing and how data integration technology must change with it. The presentation boiled this down to 5 critical and lesser known data integration requirements, how to understand them, and how to pick the right approaches and technology to solve the problems.

According to David, the 5 most critical things to understand about modern data integration are:

  1. Workloads and data are likely to be distributed across traditional systems, private clouds, and public clouds.
  2. Data is growing quickly, with big data and data lakes common within most enterprises.
  3. Data must be delivered in real-time, on-demand, in support of most modern applications and business processes.
  4. Security is now systemic, it can no longer be an after thought.
  5. DevOps is the new standard for building and deploying applications and data store.

Continue reading “What You Need to Know About Modern Data Integration”

SnapLogic on the Radar with MWD Advisors

MWD AdvisorsSnapLogic was recently reviewed by advisory firm MWD Advisors for our efforts to reinvent integration platform technology by creating one unified platform that can address many different kinds of application and data integration use cases.

A few highlights from the report:

  • An in-depth look at our multi-tenant, AWS-hosted platform which includes the SnapLogic Designer, Manager and Dashboard
  • The Snaplex execution environment; namely, the Cloudplex, Groundplex and Hadooplex
  • 3 competitive differentiators – deployment flexibility, a unified approach across multiple integration scenario types and both scalability and adaptability

Read the full review here, or take a look below. You can also check out other SnapLogic reviews on our website.

MWD Advisors is a specialist advisory firm providing practical industry insights to business leaders and technology professionals working to drive change with the help of digital technology.

The 3 A’s of Enterprise Integration

This post originally appeared on Data Informed.

binary-big-dateAs organizations look to increase their agility, IT and lines of business need to connect faster. Companies need to adopt cloud applications more quickly and they need to be able to access and analyze all their data, whether from a legacy data warehouse, a new SaaS application, or an unstructured data source such as social media. In short, a unified integration platform has become a critical requirement for most enterprises.

According to Gartner, “unnecessarily segregated application and data integration efforts lead to counterproductive practices and escalating deployment costs.”

Don’t let your organization get caught in that trap. Whether you are evaluating what you already have or shopping for something completely new, you should measure any platform by how well it address the “three A’s” of integration: Anything, Anytime, Anywhere. Continue reading “The 3 A’s of Enterprise Integration”