Iris – Can you build an integration pipeline for me?

The promise of Artificial Intelligence technology is flourishing. From Amazon shopping recommendations, Facebook image recognition, and personal assistants like Siri, Cortana, and Alexa,  AI is becoming part of our everyday lives, whether we know it or not. These apps use information collected from your past requests to make predictions and deliver results that are tailored to your preferences. 

The importance of AI in today’s world is not lost upon us at SnapLogic. We are always trying to keep up with the latest innovations and technologies, so making our software fast, efficient, and automated for our customers has always been our goal. With the Spring release, SnapLogic launched the SnapLogic Integration Assistant. The SnapLogic Integration Assistant is a recommendation engine that uses Artificial Intelligence and machine learning to predict the next step in building a data pipeline architecture. Iris uses advanced algorithms to collect information from millions of metadata elements and billions of data flows to make predictions and deliver results that are tailored to the customer’s needs.

Currently, customers build pipelines by searching and selecting from over 400 Snaps in the SnapLogic catalog and dragging and dropping them into the canvas. Repeating this step for every single Snap, although easy, can make a pipeline building process somewhat tedious and time-consuming. But with the Integration Assistant, operations like these are simplified to give business users the right next steps in the building process, making pipeline building easy and efficient. Besides efficiency and speed, the “self-driving” software shortens the learning curve for line-of-business users to manage their data flows while freeing technology staff for higher-value software development. See how it works in this video.

In the next few steps,  learn how to enable this feature and start building interactive pipelines yourself.

Right now, we have two ways of building pipelines:

  • Choose a Snap from the SnapLogic catalog
  • Use the Integration Assistant for recommending the right Snaps

How to enable the Integration Assistant feature

By default, the Integration Assistant option is turned off,  allowing you to continue building pipelines by selecting Snaps in the SnapLogic Catalog. However, to utilize the Integration Assistant, just head to the Settings icon and check the Integration Assistant option.

Once the Integration Assistant is enabled, you’ll immediately see the benefits of the self-guided user interface. Drag the first snap onto the canvas and the Integration Assistant instantly kicks in and highlights the next suitable Snap. At the same time, it also opens up another panel that lists suggested Snaps on the right-hand side of the canvas. These AI-driven Snap recommendations are based on the historical metadata from your previous workflows.

Next, you can choose to click the highlighted Snap or pick from the recommended list by dragging the suitable Snap into the canvas. This process continues further until you select a snap with a closed output. At this point, the Integration Assistant will stop suggesting Snaps and the pipeline will be ready for execution.

As you can see, the Integration Assistant improves your pipeline building experience by suggesting Snaps that are the best fit for your organization based on the historical metadata flows.

Interested in learning more? Watch a quick demo on our YouTube channel – SnapLogic Spring 2017: Integration Assistant.”

Namita Prabhu is Senior QA Manager at SnapLogic.

Spring 2017 Release: Self-driving integration, field cryptography, MS Dynamics 365 CRM and more

Today the Spring 2017 release is available and featuring artificial intelligence (AI) technology that promises to dramatically reduce the time and cost of cloud, analytics, and digital transformation initiatives. The first AI-powered feature is SnapLogic Integration Assistant, a recommendation engine that delivers expert step-by-step guidance to improve the speed and quality of building a data pipeline — with up to 90% accuracy. Integration Assistant is available to all of our customers starting today at no charge.

SnapLogic’s Spring 2017 release also introduces new and enhanced software that helps our customers jumpstart their CRM, HR, and cloud data warehouse software projects. Finally, the release has added platform-wide features designed to save time and increase productivity.

New or enhanced capabilities in the Spring 2017 release include:

  • SnapLogic Integration Assistant: The SnapLogic Integration Assistant is a recommendation engine that uses machine learning to predict the next step in building a data pipeline for the cloud, analytics, and digital initiatives – with up to 90% accuracy. It is part of SnapLogic’s “Iris” technology – an industry first in applying artificial intelligence for enterprise integration. See how it works in this video.
  • Microsoft Dynamics CRM integration: A new Snap Pack is available to help users create, read, and update records in the cloud and on-premises versions of Microsoft Dynamics. Also, users can delete a record based on account ID and also search with various filter options.
  • Workday integration: The Workday Read Snap now supports page number and page size and shows great performance improvements. The Workday Write Snap now supports bulk operations for many objects and has improved performance from hours and minutes to just seconds.
  • Confluent integration: The updated Confluent Acknowledge Snap greatly reduces potential data losses, eliminates duplicates, and externally acknowledges each message.
  • Enhanced Snaps for Amazon Redshift, Anaplan, Tableau, Apache Hive + Kerberos.
  • Field encryption and decryption: The Transform Snap Pack now includes new Snaps to encrypt and decrypt field values for sensitive data, or entire documents, providing a greater level of data security.
  • Platform enhancements: These updates save time and enhance productivity, including Snaplex restart; enhanced Asset Search with Snap labels; network statistics display to aid troubleshooting and performance optimization; and parameterizable accounts that make it easier to automate or dynamically assign environments in a development lifecycle.

SnapLogic is committed to continuous innovation and the features in the Spring 2017 release for the SnapLogic Enterprise Integration Cloud are examples of how we continue to make integrating data warehouses, applications, IoT, and big data streams faster, easier. and more powerful. To learn more, go to www.snaplogic.com/spring2017.

Applying machine learning tools to data integration

greg-bensonBy Gregory D. Benson

Few tasks are more personally rewarding than working with brilliant graduate students on research problems that have practical applications. This is exactly what I get to do as both a Professor of Computer Science at the University of San Francisco and as Chief Scientist at SnapLogic. Each semester, SnapLogic sponsors student research and development projects for USF CS project classes, and I am given the freedom to work with these students on new technology and exploratory projects that we believe will eventually impact the SnapLogic Enterprise Integration Cloud Platform. Iris and the Integration Assistant, which applies machine learning to the creation of data integration pipelines, represents one of these research projects that pushes the boundaries of self-service data integration.

For the past seven years, these research projects have provided SnapLogic Labs with bright minds and at the same time given USF students exposure to problems found in real-world commercial software. I have been able to leverage my past 19 years of research and teaching at USF in parallel and distributed computing to help formulate research areas that enable students to bridge their academic experience with problems found in large-scale software that runs in the cloud. Project successes include Predictive Field Linking, the first SnapLogic MapReduce implementation called SnapReduce, and the Document Model for data integration. It is a mutually beneficial relationship.

During the research phase of Labs projects, the students have access to the SnapLogic engineering team, and can ask questions and get feedback. This collaboration allows the students to ramp up quickly with our codebase and gets the engineering team familiar with the students. Once we have prototyped and demonstrated the potential for a research project we transition the code to production. But the relationship doesn’t end there – students who did the research work are usually hired on to help with transitioning the prototype to production code.

The SnapLogic Philosophy
Iris technology was born to help an increasing number of business users design and implement data integration tasks that previously required extensive programming skills. Most companies must manage an increasing number of data sources and cloud applications as well as an increasing amount of data volume. And it’s data Integration platforms that help business connect and transform all of this disparate data. The SnapLogic philosophy has always been to truly provide self-service integration through visual programming. Iris and the Integration Assistant further advances this philosophy by learning from the successes and failures of thousands of pipelines and billions of executions on the SnapLogic platform.

The Project
Two years ago, I led a project that refined our metadata architecture and last year I proposed a machine learning project for USF students. At the time, I gave some vague ideas about what we could achieve. The plan was to spend the first part of the project doing data science on the SnapLogic metadata to see what patterns we could find and opportunities for applying machine learning.

One of the USF graduate students working on the project, Thanawut “Jump” Anapiriyakul, discovered that we could learn from past pipeline definitions in our metadata to help recommend likely next Snaps during pipeline creation. Jump experimented with several machine learning algorithms to find the ones that give the best recommendation accuracy. We later combined the pipeline definition with Snap execution history to further improve recommendation accuracy. The end result: Pipeline creation is now much faster with the Integration Assistant.

The exciting thing about the Iris technology is that we have created an internal metadata architecture that supports not only the Integration Assistant but also the data science needed to further leverage historical user activity and pipeline executions to power future applications of machine learning in the SnapLogic Enterprise Cloud. In my view, true self-service in data integration will only be possible through the application of machine learning and artificial intelligence as we are doing at SnapLogic.

As for the students who work on SnapLogic projects, most are usually offered internships and many eventually become full-time software engineers at SnapLogic. It is very rewarding to continue to work with my students after they graduate. After ceremonies this May at USF, Jump will join SnapLogic full-time this summer, working with the team on extending Iris and its capabilities.

I look forward to writing more about Iris and our recent technology advances in the weeks to come. In the meantime, you can check out my past posts on JSON-centric iPaaS and Hybrid Batch and Streaming Architecture for Data Integration.

Gregory D. Benson is a Professor in the Department of Computer Science at the University of San Francisco and Chief Scientist at SnapLogic. Follow him on Twitter @gregorydbenson.