I guess you could say that Big Data is now such a mainstream topic that there’s a band named after it. Coming off of the successful Hortonworks IPO this week, The Wall St. Journal published this article: The Joys and Hype of Software Called Hadoop – Big Data Is Hot in Silicon Valley, and Hadoop Underpins Craze. To get some context, a bit of history, and a few insights into what’s coming next, I sat down with SnapLogic’s co-founder and CEO Gaurav Dhillon to discuss all things data, what’s different and what’s the role of data integration in the era of social, mobile, analytics, cloud and the Internet of Things (SMACT).

Here’s the podcast:

And here’s the transcript of a few key parts of the discussion:

Gaurav DhillonWhat’s different now? Why are some of these becoming mainstream topics? What’s changed?
Statisticians have been around, maybe not as long as mathematicians, but shortly thereafter they arrived. They have been using traditional technologies. Just because somebody has a sharper tool doesn’t make them a better craftsman necessarily. I agree, there’s some element of, “Oh, well sprinkle data on it and it will be magic.” Right? This magic sprinkling. I would first of all concede that statisticians have been using forms of these technologies, which we now call data science, in a way that certain companies have been doing for a long time….

What’s different, is that the volume of information that you have, and the tools that you have, take this out of a high priesthood, into more of a common man perspective. Traditionally, we democratized reporting. We democratized how people get information. There’s a potential, if this industry lives up to the promise and is responsible about all the resources available to it, there’s a potential to democratize the benefit of what would be a very highly-funded, very niche, almost a national government level of effort, for many people. There’s an opportunity to democratize that using open-source technologies, falling prices, better products, and smarter graduates from college.

Analytics is getting more than its share of attention in the market, but the feeds for these analytical systems still don’t seem to be getting the same attention. A lot of companies are stuck using what they used to use, or trying to use what they used to use to solve some of the newer problems. What’s changing on the plumbing side of things that is getting you excited?
You’re right, plumbing, like Rodney Dangerfield, doesn’t get you respect in this business. It’s always about the “gee whiz”, the graphics. “Oh, look at this thing we found.” So on and so on. What’s changing is that the plumbing is letting you have twice as many data scientists. There’s a scarcity of a job type in the world, it is data scientists. The fact that you can have twice as many chefs, like giving the sous chef business to somebody else, is hugely important to everybody. I don’t care how rich you are, because there’s always somebody as rich as you, or richer than you, competing with you. They fact that you can almost double the energy you can put into this very important area is a huge deal, and it’s causing an increase in importance in plumbing. New words like data wrangling are springing up, to really show you how the sous-chef end of preparing this wonderful result is becoming more important. How do you profile this data? How do you a priori make sure it arrives? How do you engage with the data? How do you combine and transform? These sorts of things are, I think, raising in importance because the straightforward payback of having twice the benefit from it is very, very clear.

I’ve heard you use the term “so-so integration“. Same old, same old. Why wouldn’t I just use the tools that I’ve got? I’ve spent a lot of money on these. Why wouldn’t I use those to solve some of these newer data challenges that you talk about?
The question is not, can you or can’t you? Some of our customers are some of the best-funded, biggest corporations in the world. You could. The question is, should you? In this business, it’s always been, can you or should you? What I’ve found in 2 decades of doing this is that, you really shouldn’t. The R&D investments that we have made, and a whole 50 million dollars of capital has been put into building a platform that is enormously capable, we can advance the results of that investment across all our customers. On top of that, there’s certain attributes to the platform that give you the opportunity to tackle how you move to the Cloud, as well as the data element. The change in data gravity, the change in thinking from same old, same old data warehousing, to the modern architectures of the new Hadoop world that we’re seeing. It’s not can you, the question is should you?

What’s the impact on Cloud computing on the data world? What impact is that having and will that have in the next 3 to 5 years?
I think the immediate apparent impact of Cloud computing is that you are able to light up a large number of seats of people using these products with that huge amount of skill sets and plumbing required…

In the marketing area, every marketing department has a dozen or more Cloud/SaaS applications, often without the CIO or anybody else in the business knowing about them. They don’t think of them as applications – they think of them as websites. That’s just for a small company. Multiply that times a Fortune 500 company, and there are probably hundreds and hundreds of things coming in to the company, and that asteroid belt that these companies are sailing through, is causing more data to be produced, more endpoints to be created, more engagement, and need to cross-tabulate or to be able to combine that information. This is causing a growth in the need for plumbing in the way that hasn’t been there. In the 90’s when you had 4 or 5 systems, if you had SAP and Siebel and PeopleSoft, maybe 1 other you would own. The fact that you have this asteroid belt of websites that run a business today is causing the problem of integration and plumbing to rise to the fore like never before.

2015 Technology predictions – you can watch that part of the discussion here. Here are a few other 2015 technology predictions.

What’s different about SnapLogic, what should people know about SnapLogic that they might not already know?
We built SnapLogic to change how people engage with the modern enterprise… We built this company from the ground up, based on the experiences that we had in the 90’s, to provide something that is very simple to use, scales across the largest set of problems somebody can throw at us, and is extremely well connected – has Snaps for a variety of endpoints and data points. It is a single platform to help you move to the Cloud and use big data, use Hadoop, use data science to solve the analytics set of equations that you face as your business goes through big change.

The data structures are changing, the endpoints are many, many more. We think of them as massively multi-point data-types. In addition, you have a very broad population of users that you didn’t have in the 90’s where integration was more of a back office operation in the dungeon. The engine room of the ship, not quite at the bridge. The passengers didn’t even know it existed, unless it stopped working.

For enterprise IT organizations, speed and time to value are two of the primary business drivers for a cloud-based integration service. This is according to a survey we recently conducted with TechValidate, querying more than 100 U.S. companies with revenues greater than $500 million. The survey addresses the business and technical drivers and barriers for enterprise cloud application adoption in 2015 and beyond. Among the other facts and figures, which can be found in our latest infographic, we also found that participants look for the following technical requirements when evaluating a cloud integration platform as a service (iPaaS) solution:

  • Modern, scalable architecture
  • Modern approach that seamlessly handles cloud constructs
  • Ease of use for less technical users (“citizen integrators”)
  • Single platform to power apps and analytics
  • Support for hybrid deployments
  • Resilient to change

Check out the infographic below and take a look at the full writeup of survey results here.

Techvalidate_infographic

Winter_Snaps_Blog_GraphicBuilding on what’s new with our recent Winter 2015 release, this Saturday we will be delivering a new SnapLogic Snap release, which includes the following new Snaps:

  • Anaplan (Action and Write)
  • Email Archive (provides the functionality to move email messages from one folder to another folder)
  • Join (the old Join Snap will continue to run, but all future enhancements will be made to the new Snap)
  • Salesforce Wave Analytics

The December Snap release also includes fixes and enhancements to the following Snaps:

  • Active Directory Rename
  • CSV Parser and CSV Formatter
  • Email Reader
  • Fixed Width Parser
  • ForEach
  • Google Analytics
  • Jira Snap Pack
  • MongoDB Find
  • Netsuite Create
  • Oracle Snap Pack
  • Salesforce Read, Salesforce SOQL, and Salesforce Writers
  • SAP HANA Snap Pack
  • SAP Snap Pack
  • SQL Server Snap Pack
  • Tableau Snap Pack
  • Union
  • XML Parser and XML Formatter
  • Zuora Read

All SnapLogic customers have received details on this weekend’s Snap release, which does not require service downtime. More specific details on the release are available on the SnapLogic Documentation site. Be sure to read this post for more information about SnapLogic’s over 300 Snaps and Contact Us if you have questions or would like to learn more about the SnapLogic Elastic Integration Platform

451-logoThis week 451 Research published the latest review of our Winter 2015 releaseSnapLogic series D funding and new release readies the iPaaS vendor for a 2015 drive. Here are a few highlights:

“SnapLogic is revving the engines for a rapid start going into 2015. It seeks its fair share of what it believes to be a $500 billion market opportunity for large enterprise hybrid IT and big data integration.”

“The big-data market may be ripe now for integration vendors like SnapLogic to safely to enter. SnapLogic is also gearing up for participation in the API management market, which represents considerable opportunity to attract new developers. All TIBCO customers represent fair game for SnapLogic.”

I’ve embedded the 451 SnapLogic Winter 2015 review below. Be sure to Contact Us and check out our Resource Center for more information about the SnapLogic Elastic Integration Platform.

SnapLogic-BlueHill-GooglePlusEarlier this month we announced the Winter 2015 release of the SnapLogic Elastic Integration Platform. (You can learn more about what’s new here.) This week Blue Hill Research published a Market Alert on the release: SnapLogic Winter 2015: Enhancing Enterprise Security and Usability for Big Data and Cloud Integration. Written by Hyoun Park and James Haight, the report zeroes in on SnapLogic’s new big data integration capabilities, Snaps and Snap Patterns for Salesforce Wave and SAP Hana and enterprise-focused integration platform as a service (iPaaS) deployment features and notes:

“With its Winter 2015 release, SnapLogic is positioning itself as a centralized integration locus to coordinate the migration from traditional client-server IT to the new world of personalized and highly scalable social, networked, mobile, Big Data, and cloud technologies.”

Here are a few other highlights from the report:

“Although Hadoop is still a tool for data scientists and analysts, the Winter 2015 upgrade now positions SnapLogic as an interface to prioritize MapReduce jobs and Big Data integration tasks.”

“By providing additional fail-safes and an enterprise-grade integration testing environment, SnapLogic is playing a role in supporting enterprise-wide technology transformations with development processes and deployment options that reflect the demands of maturing expectations.”

“As SAP HANA Platform continues to grow as a vendor-agnostic platform, and as SnapLogic continues to expand its collection of 270+ integrations, Blue Hill expects SnapLogic customers to be well-positioned to develop SAP HANA integrations across cloud applications.”

I’ve embedded the report below and you can download it directly here. You can also learn more about our Winter 2015 release here.

It’s technology predictions season! In this video, SnapLogic co-founder and CEO Gaurav Dhillon shares his predictions for cloud computing and big data in 2015, including:

  1. Cloudification: More people in business will be using applications in the cloud, causing ”cloudification” to happen in a big way in 2015.
  2. Self-Service: There will be a greater expectation of self-service and consumer-friendly technologies; customers won’t put up with complicated ways of doing things and vendors will be expected to do the work.
  3. Big Data: The big data projects are going to have to get to a more straightforward ROI or die.

ResourcePageButton (1)Today we announced our Winter 2015 release, with a focus on securing cloud application and big data integration for hybrid deployments. The release introduces enhanced security and lifecycle management capabilities for enterprise integration platform as a service (iPaaS) deployments, new and updated cloud and on-premises Snaps and a continued focus on productivity and self-service features for “citizen integrators.” As our CEO Gaurav Dhillon said, “We’re seeing a re-imagining of integration in the enterprise as CIOs look for greater business alignment and agility as well as a faster payback on their cloud application and analytics investments. Our Elastic Integration Platform is now processing over 10 billion JSON documents per month and with a focus on security, performance and developer productivity, the Winter 2015 release enables our customers to further expand their iPaaS deployments from real-time data synchronization and orchestration to big data integration initiatives.

Be sure to check out the webinar for detailed demonstrations of what’s new, including:

  1. iPaaS for the Enterprise: Enhanced encryption, lifecycle management, improved developer productivity
  2. Big Data Integration: SnapReduce 2.0 and Hadooplex now available as add-on options
  3. New and Updated Snaps: Including Salesforce Analytics Cloud, Sumo Logic and SAP HANA

You can also take a look at a few of the resources to get the full overview of the Winter 2015 release and new features that are now available for SnapLogic customers: