In his decades working in the data and analytics industry, SnapLogic CTO James Markarian has witnessed few mainstream events that have sparked as much discussion and elicited as many questions – around the value and accuracy of predictive analytics – as our recent election.
In a new blog post on Forbes, James examines where the nation’s top pollsters (who across the board predicted a different election outcome) possibly went wrong, why some predictions succeed and others fail, what businesses who have invested in data analytics can learn from the election, and how new technologies such as integration platform as a service (iPaaS) can help them make sense of all their data to make better predictions.
Be sure to read James’s blog, titled “What The Election Taught Us About Predictive Analytics”, on Forbes here.
As data increasingly becomes the means by which businesses compete, companies are restructuring operations to build systems and processes liberating data access, integration and analysis up and down the value chain. Effective data management has become so important that the position of Chief Data Officer is projected to become a standard senior board level role by 2020, with 92 percent of CIOs stating that a CDO is the best person to determine data strategy.
With this in mind as you evaluate your data strategy for 2017, here are seven predictions to contemplate to build a solid framework for data management and optimization.
- Self-Service Data Integration Will Take Off
Eschewing the IT bottleneck designation and committed to being a strategic partner to the business, IT is transforming its mindset. Rather than be providers of data, IT will enable users to achieve data optimization on a self-service basis. IT will increasingly decentralize app and data integration – via distributed Centers of Excellence based on shared infrastructure, frameworks and best practices – thereby enabling line-of-business heads to gather, integrate and analyze data themselves to discern and quickly act upon insightful trends and patterns of import to their roles and responsibilities. Rather than fish for your data, IT will teach you how to bait the hook. The payoff for IT: satisfying business user demand for fast and easy integrations and accelerated time to value; preserving data integrity, security and governance on a common infrastructure across the enterprise; and freeing up finite IT resources to focus on other strategic initiatives.
- Big Data Moves to the Cloud
As the year takes shape, expect more enterprises to migrate storage and analysis of their big data from traditional on-premise data stores and warehouses to the cloud. For the better part of the last decade, Hadoop’s distributed computing and processing power has made it the standard open source platform for big data infrastructures. But Hadoop is far from perfect. Common user gripes include complexity and instability – not all that surprising given all the software developers regularly contributing their improvements to the platform. Cloud environments are more stable, flexible, elastic and better-suited to handling big data, hence the predicted migration.
- Spark Usage Outside of Hadoop Will Surge
This is the year we will also see more Spark use cases outside of Hadoop environments. While Hadoop limps along, Spark is picking up the pace. Hadoop is still more likely to be used in testing rather than production environments. But users are finding Spark to be more flexible, adaptable and better suited for certain workloads – machine learning and real-time streaming analytics, as examples. Once relegated to Hadoop sidekick, Spark will break free and stand on its own two feet this year. I’m not alone in asking the question: Hadoop needs Spark but does Spark need Hadoop?
- A Big Fish Acquires a Hadoop Distro Vendor?
Hadoop distribution vendors like Cloudera and Hortonworks paved the way with promising technology and game-changing innovation. But this past year saw growing frustration among customers lamenting increased complexity, instability and, ultimately, too many failed projects that never left the labs. As Hadoop distro vendors work through some growing pains (not to mention limited funds), could it be that a bigger, deeper-pocketed established player – say Teradata, Oracle, Microsoft or IBM – might swoop in to buy their sought after technology and marry it with a more mature organization? I’m not counting it out.
- AI and ML Get a Bit More Mainstream
Off the shelf AI (artificial intelligence) and ML (machine learning) platforms are loved for their simplicity, low barrier to entry and low cost. In 2017, off the shelf AI and ML libraries from Microsoft, Google, Amazon and other vendors will be embedded in enterprise solutions, including mobile varieties. Tasks that have until now been manual and time-consuming will become automated and accelerated, extending into the world of data integration.
6. Yes, IoT is Coming, Just Not This Year
Connecting billions and billions of sensor-embedded devices and objects over the internet is inevitable, but don’t yet swallow all the hype. Yes, there is a lot being done to harness IoT for specific aims, but the pace toward the development of a general-purpose IoT platform is closer to a canter than a gallop. IoT solutions are too bespoke and purpose-built to solve broad, commonplace problems – the market still nascent with standards gradually evolving – that a general-purpose, mass-adopted IoT platform to collect, integrate and report on data in real-time will take, well, more time. Like any other transformation movement in the history of enterprise technology, brilliant bits and pieces need to come together as a whole. It’s coming, just not in 2017.
- APIs Are Not All They’re Cracked Up to Be
APIs have long been the glue connecting apps and services, but customers will continue to question their value vs investment in 2017. Few would dispute that APIs are useful in building apps and, in many cases, may be the right choice in this regard. But in situations where the integration of apps and/or data is needed and sought, there are better ways. Case in point is iPaaS (integration platform as a service), which allows you to quickly and easily connect any combination of cloud and on-premise technologies. Expect greater migration this year toward cloud-based enterprise integration platforms – compared to APIs, iPaaS solutions are more agile, better equipped to handle the vagaries of data, more adaptable to changes, easier to maintain and far more productive.
I could go on and on, if for no other reason that predictions are informed “best guesses” about the future. If I’m wrong on two or three of my expectations, my peers will forgive me. In the rapidly changing world of technology, batting .400 is a pretty good statistic.
This week, the SnapLogic team will be supporting one of our partners, Amazon Web Services, in Las Vegas for the annual AWS re:Invent conference. This gathering of the global AWS community will feature hands-on labs and bootcamps and cover topics such as infrastructure maintenance, and improving developer productivity, network security and application performance.
Continue reading “Connect with SnapLogic at AWS re:Invent”
Last week, part of the SnapLogic team was in New York City for the Strata/Hadoop World conference. It’s one of the largest big data events in the U.S. and has grown steadily larger over recent years. The agenda has shifted a bit as well – from largely academic discussions and how-to presentations by open source committers to real-world case studies by non-ISV enterprises.
With that in mind, I’d like to share a story from one of our enterprise customers. In fact, this customer is a 100+ year old financial institution. Perhaps not a company that you would associate with the cutting edge of data management technologies… Due the nature of their industry, I can’t share their name.
Like many established companies, this bank’s data processing and storage systems have been acquired or added over the years based on the most pressing needs and compliance requirements at the time. They ultimately found themselves trying to manage an unwieldy mix of 240+ interfaces and applications. Continue reading “A Hadoop Data Lake For Banking: A SnapLogic Story”
Data visualization is a hot topic, but is only useful when the most up-to-date data is made available. This SnapLogic Live session features Tableau Integration for customers seeking rapid integration to connect Tableau with other cloud and on-premises applications and data sources.
Continue reading “SnapLogic Live: Tableau Integration”
SnapLogic and Amazon Web Services are hosting a series of exclusive live seminars starting this week in Dallas. Next week we’ll be in Chicago and New York, followed by Palo Alto later in the month. The seminar series is focused on the future of data warehouse solutions and analytics in the modern enterprise. A key question that we’ll address is: Is the Data Warehouse Dead? Continue reading “Is the Data Warehouse Dead?”
Take a look at our new video outlining SnapLogic’s various points of integration with Microsoft and the Azure and Cortana ecosystems. This is the latest in a series of announcements and go-to-market activities we’ve done over the last nine months regarding our work with Microsoft, including our initial partnership announcement.
Produced in conjunction with Microsoft, this brief video features SnapLogic’s VP of Product Management, Craig Stewart. In it, Craig discusses how SnapLogic’s hybrid architecture is a natural fit for consuming and moving large data sets between on premises sources and the cloud, making it the ideal integration solution for Microsoft Azure data stores, HDInsight and Cortana Intelligence solutions.
The video also explores SnapLogic’s many Snaps for Microsoft solutions including Azure SQL Data Warehouse, Azure SQL Database, Azure Active Directory, Azure SQL Bulk Load and Blob Storage, plus on-premises Microsoft SQL Server, Dynamics, Active Directory and more.
Stay tuned for more additions to our library of solutions for Microsoft Azure, HDInsight, and Cortana customers.