“Hide the Tech” to Take Big Data Mainstream

Originally posted on InsideBigData.

By Ravi Dharnikota

Big data is a big deal – and a big opportunity. The challenge is that most big data progress has been limited to big companies with big engineering and data science teams. The systems can be complex, immature and hard to manage. That might be OK if you’re a well-trained developer in Silicon Valley, but it doesn’t play well if you’re a jack-of-all-trades IT leader at a bank in Atlanta or an on-the-move business leader in Amsterdam. How can you tap into big data if you don’t have an army of engineers who stay steeped in the latest tech?

Fortunately, help is on the way thanks to several trends that move tech from the foreground to the background.

First and foremost, big data is moving into the cloud. Amazon, Microsoft and Google now deliver Hadoop and Spark “as a service,” eliminating the need to spin up a cluster, research new software products or worry about version management. Companies are increasingly moving their big data workloads into the cloud, hiding complexity from their users while relying on the world’s best data center professionals to manage the tech. They want access to infrastructure when and how they want it, with just as much as they need but more.

Next up is the emergence of “serverless” computing. This builds on the cloud trend, while removing even more tech dependencies. Just load your data and start processing. Tell your cloud provider what you want to do, how much data you want to crunch and where you want to run it – and they will spin up the infrastructure when you need it, and spin it down when you don’t. That’s incredible for retailers and CPG companies, for example, who may have seasonal businesses and therefore fluctuating data needs throughout the year. They can do critical data analytics as needed without having to pay for robust infrastructure during the off-peak periods.

The third big trend is the move to self-service, fueled by new tools and platforms that democratize both data integration and data consumption. “Self-service integration” makes it fast and easy to connect systems, create data pipelines and automate processes without the need for intensive coding. Likewise, “self-service analytics” makes it easy for analysts and business users to manipulate data without IT intervention. On both fronts, self-service lowers the bar on tech skill requirements, opening the data door for millions of new users.

Last but certainly not least is machine learning and artificial intelligence, which are being used to embed more and more intelligence into applications. The systems simply learn from the data that flows through – and deliver predictive insights or recommend actions accordingly. For example, we can now offer “self-driving” integration that learns from millions of metadata elements and billions of data flows to give users expert step-by-step directions in building their data pipelines. That shortens the learning curve for business users and analysts while freeing tech teams for higher-value innovation and governance.

All four of these trends are powered by very advanced technologies, but what’s remarkable is how much they actually “hide” the tech to put more focus on the data and the business – exactly where the focus should be. It’s very similar to what we’re seeing with the incredibly intuitive consumer software that just works out of the box without a lot of training or manual intervention. Like those consumers, companies want to get more things done faster with convenience and fast access. They want cutting-edge capabilities, but they don’t want to deal with bugs and broken integrations and rapidly changing versions.

At the end of the day, most companies just want better data and faster answers – without the technology headaches. “I should just be able to write the query and get an answer back.” Thanks to the cloud, serverless computing, self-service platforms and self-learning technologies, we’re getting closer to that goal. While big data isn’t yet at the Google search or Alexa skill level of simplicity, we should all aspire to take it there.

Ravi Dharnikota is Chief Enterprise Architect at SnapLogic. Follow him on Twitter @rdharn1

Mossberg out. Enterprise technology still in

By Gaurav Dhillon

A few weeks ago, the legendary tech journalist, Walt Mossberg, penned his last column. Although tech journalism today is vastly different than it was in 1991, when his first column appeared in the Wall Street Journal, or even five or 10 years ago, voices like Walt’s still matter. They matter because history matters – despite what I see as today’s widely held, yet unspoken belief that nothing much important existed prior to the invention of the iPhone.

Unpacking that further, history matters because the people who learn from it, and take their cues from it, are those who will drive the future.

Enterprise tech history is still unfolding

I like to think of myself as one of those people, certainly one who believes that all history is meaningful, including tech history. As tech journalism’s eminence grise, Walt not only chronicled the industry’s history, he also helped to define it. He was at the helm of a loose cadre of tech journalists and industry pundits, from Robert X. Cringely to Esther Dyson, who could make or break a company with just a few paragraphs.

Walt is now retiring. So what can we learn from him? The premise of his farewell column in Recode is that tech is disappearing, in a good way.”[Personal] tech was once always in your way. Soon, it will be almost invisible,” he wrote, and further, “The big software revolutions, like cloud computing, search engines, and social networks are also still growing and improving, but have become largely established.”

I’ll disagree with Walt on the second point. The cloud computing revolution, which is changing the way enterprises think and operate, is just beginning. We are at a juncture populated by unimaginably large quantities of data, coupled with an equally unquenchable thirst by enterprises to learn from it. The world has gone mad for artificial intelligence (AI) and analytics, every permutation of which is fueled by one thing: data.

The way we use data will become invisible

In his column, Walt observed that personal tech is now almost invisible. We use and benefit from it in an almost passive way. The way data scientists and business users consume data is anything but. Data is still moved around and manually integrated, on-premises and in the cloud, with processes that haven’t changed much since the 1970s. Think about it – the 1970s! It’s no secret that extract, transfer, and load (ETL) processes remain the bane of data consumers’ existence, largely because many enterprises are still using 25-year-old solutions to manage ETL and integrate data.

Cloud Computing

The good news is, data integration is becoming much easier to do, and is well on its way to becoming invisible. Enterprise integration cloud technology promises to replace slow and cumbersome scripting and manual data movement with fast, open, seamless data pipelines, optimized with AI techniques.

Remember how, as Internet use exploded in the late 1990s, the tech industry was abuzz with companies offering all manner of optimization technologies, like load balancing, data mirroring, and throughput optimization? These days you never hear about these companies anymore; we take high-performance internet service for granted, like the old-fashioned dial tone.

I am confident that we are embarking on a similar era for enterprise data integration, one in which modern, cloud-first technologies will make complex data integration processes increasingly invisible, seamlessly baked into the way data is stored and accessed.

Making history with data integration

I had the pleasure of meeting Walt some years ago at his office, a miniature museum with many of the personal tech industry’s greatest inventions on display. There, his love of tech was apparent and abundant. Apple IIe? Nokia Communicator 9000? Palm Treo and original iPod? Of course. If Walt were to be at his keyboard, in his office, for another couple of years, I’m pretty sure his collection would be joined by a technology with no physical form factor, but of even greater import: the enterprise cloud.

Hats off to you, Walt. And while you may have given your final sign-off, “Mossberg out,” enterprise tech is most definitely still in.

Follow me on Twitter @gdhillon.

Gaurav Dhillon is CEO of SnapLogic. You can follow him on Twitter @gdhillon.