Data volumes are exponentially increasing and many organizations are starting to realize the complexity of their growing data movement and data management solutions. Data exists in various systems, and getting meaningful value out of it has become a major challenge for many companies. Also, most of the data is usually stored in relational systems like MySQL, PostgreSQL and Oracle, these being the mainstream databases primarily used for OLTP purposes. NoSQL systems like Cassandra, MongoDB and DynamoDB have also emerged with tunable consistency model in order to store some of these mission critical data. Customers then typically move these data to much bigger systems like Teradata and Hadoop (OLAP) that can store large amounts of data, so they can run analytics, reporting or complex queries against it. There is also a recent trend where some of these data are moved to the cloud, especially to Amazon RedShift or Snowflake and also to HDInsights or Azure Data Warehouse.
SnapLogic co-founder and CEO Gaurav Dhillon sat down recently with Scott Kupor, managing partner at Andreessen Horowitz, for a wide-ranging podcast discussion of all-things-data.
The two discussed how the data management landscape has changed in recent years, the rise of advanced analytics, the move from data warehouses to data lakes, and other changes which are enabling organizations to “take back their enterprise.”
The future for big data processing lies in the adoption of commercial Hadoop distributions and their supposed deployments. The macro use case for big data are data lakes, which are massive amounts of structured and unstructured data that do not carry the same restrictions as traditional data warehouses. They store everything, including every type of data, any volume, any scope of data that may be used by enterprise data users, for any reason.
Despite the power and potential of data lakes, many enterprises continue to approach this technology with the same data integration approaches and mechanisms they’ve used in the past, none of which work well. How can we tap into the power of the data lake? Continue reading “The Data Lake Data Integration Challenge”
SnapLogic and Amazon Web Services are hosting a series of exclusive live seminars starting this week in Dallas. Next week we’ll be in Chicago and New York, followed by Palo Alto later in the month. The seminar series is focused on the future of data warehouse solutions and analytics in the modern enterprise. A key question that we’ll address is: Is the Data Warehouse Dead? Continue reading “Is the Data Warehouse Dead?”
Big data is evolving as a practice and we are quickly approaching a point at which data will be treated as a single source, which will require a different type of architecture. According to John Myers of Enterprise Management Associates (EMA), this architecture will need to be one that is focused beyond a single platform, where operational and analytical workloads work together. This architecture is called a Hybrid Data Ecosystem.
Join us on Wednesday, April 29th for a live webinar with John, Managing Research Director for EMA’s Business Intelligence practice. This webinar will review the drivers associated with big data implementations, evolving technical requirements for big data environments, and how a robust information management layer is important to big data projects.
During the webinar, we’ll also review how recent EMA research describes the following:
- Use cases that drive big data and the importance of Internet of Things and streaming applications in big data
- The impact of cloud implementation avenues for big data projects
- How the EMA Hybrid Data Ecosystem Information Management Layer coordinates integration between disparate platforms and data types
Register now and join John Myers and the SnapLogic team for this exciting webinar to learn about what constitutes the Hybrid Data Ecosystem – and why it’s a necessity for modern data integration.
“Our research uncovers best practices that innovative organizations use not only to prepare and integrate big data but also more tightly unify it with analytics and operations across enterprise and cloud computing environments.”
– Mark Smith, CEO & Chief Research Officer, Ventana Research
Our latest webinar, featuring industry expert Mark Smith, focused on integration as a way to make full use of big data coming into the enterprise from a variety of sources and in incompatible formats. That being said, most organizations lack the technology to automate this process and manage this daunting challenge. Rather than relying on existing tools not specifically designed for this purpose, Ventana Research recommends that businesses use technology designed specifically to handle big data integration as adoption can significantly impact the ability to succeed in the world of nonstop data.
During the interactive discussion, Mark covered five best practices for attaining excellence in big data integration, which are:
- Evaluate efficiency of processes: Organizations need to increase agility rather than waste significant amounts of time on data integration-related tasks; integration capabilities need to be flexible enough to deliver cycles of processing to satisfy an array of different needs.
- Examine new approaches: Only one third of organizations are satisfied with their current technology and more than half of organizations say their current infrastructure is not fast enough or flexible enough; almost half said the technology is simply inadequate.
- Evaluate technology needs: Research shows that what matters most in selecting big data integration technology are its usability and reliability; the top three factors driving big data integration are business improvement, analytics and BI initiatives, and improvement in the quality of business processes.
- Investigate dedicated technology: Using dedicated data integration improves integration processes; however, currently only 12 percent of organizations use dedicated technology for this purpose.
- Gain benefits that outweigh costs: Organizations need to gain value from data and pinpoint the areas of business in which investment can help, allowing acquisition and deployment to address an organization’s needs
Mark also talked about how cloud is playing an increasing role as data can be accessed anywhere; according to research, 35% of organizations are integrating cloud-based systems with those on-premises. For full content of the webinar, take a look at the presentation slides or watch the recording here; you can also check out our infographic that addresses enterprise IT drivers, questions and uncertainties around big data integration. Ventana Research has also published an ebook available now on our website on the same topics covered in the webinar. Download it here.
Finally, we live tweeted Wednesday’s webinar with the hashtag #BDI. Check out the full roundup below with some great insights from Mark Smith:
Last week we attended Strata + Hadoop World, put on by O’Reilly Media and SnapLogic partner Cloudera. We enjoyed talking to attendees, customers and Big Data experts about swimming in the data lake vs. sinking in a data warehouse, big data integration and the next generation of ETL tools. We kept our booth lively by inviting attendees to jump into the data lake with us in our awesome photo booth, complete with props! Full album of everyone who participated here. Our Chief Scientist, Greg Benson, also had a speaking opportunity. Stay tuned for a recording of the session; in the meantime, take a look at his presentation slides: Jump into the Data Lake with Hadoop-Scale Data Integration.
Check out a recap of event highlights below and learn more about SnapLogic big data integration here. We’re also hosting an upcoming webinar featuring Ventana Research on how to attain excellence in big data integration.