Workday_AxciomThis week at Workday Rising the SnapLogic team demonstrated a set of Snaps and integration patterns that provide a simple yet powerful way to connect Workday Financial Management with information in other enterprise applications such as Salesforce CRM. In addition to SnapLogic’s self-service application and data integration capabilities, the new Snaps and patterns automate critical business processes such as the quote-to-cash cycle and new employee onboarding.

In my conversations at the conference, it was clear that Workday customers are looking always for ways to increase the flexibility of both HR and financial processes, ease the pain of adding or retiring applications, and enable both IT, HR and Finance teams to focus on more strategic business priorities. This was a recurring theme at the SnapLogic booth. You can see a few pictures here.

Some of the key benefits of SnapLogic Elastic Integration for Workday include:

  • SnapLogic Workday Snaps expose the entire Workday API visually, enabling organizations to integrate Workday data without the complexity of a heavyweight enterprise service bus (ESB) or the limitations of traditional connectors
  • SnapLogic makes it simple to connect applications to Workday workflows without coding
  • Easily connect Workday to other cloud or on-premises applications using SnapLogic’s library of 300+ Snaps

Recently SnapLogic customer Acxiom rolled out Project New Day, which saw them go live with Workday HCM and Financial Management as well as Tidemark, Vertex, ADP, Zuora, Cornerstone and other systems. Cherrylyn McCastle, Senior Director of IT at Acxiom put it this way:

“SnapLogic enabled Acxiom to achieve the objective of establishing a seamless, agile, and secure integration framework to facilitate integrations between corporate SaaS applications and a variety of on-premises systems. The new Workday HCM and Finance integrations ensure our front office and operational systems are well connected to the back office.”

Here’s a brief overview of SnapLogic for Workday from Workday Rising and here’s a recorded demonstration. I’d like to thank our friends at Workday for a great conference. The company’s focus on customer success and continued innovation is impressive and SnapLogic was proud to be a Silver Sponsor in Las Vegas.

The Fall 2015 release brings big data integration to iPaaS and allows our customers to achieve maximum benefit from the new hybrid data architecture.”

– Gaurav Dhillon, founder and CEO, SnapLogic

Today SnapLogic announced the Fall 2015 release of our Elastic Integration Platform. The new release is focused on bringing new big data integration capabilities to our unified iPaaS, including including support for Microsoft Cortana Analytics, Spark data processing and a new set of intelligent connectors, called Snaps, for Cassandra.

Partnering with Microsoft to Power Cortana Analytics
SnapLogic’s new partnership with Microsoft will provide customers of both companies a fast on-ramp to advanced analytics in the cloud. Today SnapLogic announced the first product integrations in support of this partnership, including:

  • snaplogic_cortanaPre-built Snaps for Microsoft Azure SQL Data Warehouse, Microsoft Azure SQL Database, and Microsoft Azure Blob Storage. These Snaps provide fast, self-service data connectivity to and from Cortana Analytics and other Azure data services.
  • The ability to deploy SnapLogic’s hybrid execution framework – called an Azureplex – on Microsoft Azure with and within Azure HDInsight for fast, scalable big data analytics.

Joseph Sirosh, Corporate Vice President of Machine Learning and Advanced Analytics at Microsoft had this to say about the partnership in the press release:

“The hallmark of Cortana Analytics Suite is helping organizations transform data into intelligent action. A critical part of that data transformation is connecting data sources, no matter where they exist within the enterprise, with big data stores and analytic tools without the overhead of manual integration or provisioning of compute resources. The Cortana Analytics Suite, with SnapLogic data integration dramatically accelerates time to value, and provides a powerful platform for analytics and machine learning in the cloud.”

You can watch Craig Stewart deliver a SnapLogic introduction and overview of our new Cortana Analytics hybrid cloud data integration capabilities in this recent recording at Microsoft. You can learn more about SnapLogic for Microsoft Cortana Analytics here.

Spark Up Your Pipelines
Building on our mantra, “Hadoop for Humans,” the Fall 2015 release delivers code-free big data platform innovation so that you can focus more time on analytics insights and less time on manual big data integration tasks. The Fall 2015 release adds:

  • snaplogic_sparkThe Sparkplex – Choose to run your pipelines natively on Spark, which is optimal for real-time processing tasks. The Sparkplex consists of a Translator, a Coordinator and Nodes.
  • Spark Snap – Quickly create Spark-based pipelines using SnapLogic’s intuitive, drag-and-drop interface. These high-performance pipelines can draw data from virtually any source and are ideally suited for memory-intensive, iterative processes.

With the combination of the new Sparkplex and Spark Snap, SnapLogic customers can enjoy the performance of Spark without the time and effort involved in creating and maintaining hand-coded integrations between data sources and a Spark cluster. You can watch a demonstration of the Spark Snap here. You can sign up for our Spark Early Access Program here.

Other Platform Enhancements
All SnapLogic customers will be updated to the Fall 2015 in October. In the meantime, we’ve recorded a series of short overview demonstrations of the Fall 2015 release to summarize what’s new from a usability perspective, including:

  • The ability to create pipeline execution tasks directly from the drag-and-drop interface as well as the ability to re-order tabs in the Designer when working on a number of pipelines.
  • For large enterprises using multiple accounts for applications, administrators now have important visibility into which accounts are tied to specific SnapLogic pipelines.

Check out SnapLogic demonstrations here.

snaplogic_fall2015_snapsNew and Enhanced Snaps
We continue to enhance our library of 300+ Snaps for data sources and endpoints on premises, in the cloud or in hybrid cloud environments. New Snaps with the Fall 2015 release include Cassandra and a Streaming Multi-Join Snap that joins two or more streams at a highly optimized performance rate.

Updated and enhanced Snaps include: AWS DynamoDB, Azure Active Directory, Binary File Reader and Writer (AWS Security Token Service support), Eloqua Update, Google BigQuery, JMS, REST Get, Microsoft SQL Server Bulk Load and Oracle RDBMS Bulk Load.

Next steps:

In Part 1 of “A Year in the Life of Big Data” I explored the last 12 months in the big data market through the lens of the Gartner Hype Cycle reports. In the space of one year it went from the peak of hype to “commonplace.” Why is that and what can we expect to see at the upcoming Strata/Hadoop World event next week?

Gartner conducts an annual big data adoption survey that may provide some insights. Gartner’s 2014 big data Investment Survey (Survey Analysis: Big Data Investment Grows But Deployments Remain Scarce in 2014,” Nick Heudecker & Lisa Kart, Gartner, September 9, 2014, subscription required) produced the following:

“Which of the following best describes your organization’s state of big data adoption?”

  • 4%: Don’t know
  • 24% No plans to invest at this time
  • 13%: Knowledge gathering
  • 19%: Developing strategy
  • 27%: Piloting and experimenting
  • 13% Deployed

“What are your organization’s top hurdles or challenges with big data?”

  1. Determining how to get value from big data
  2. Risk and governance issues
  3. Obtaining skills and capabilities needed
  4. Integrating multiple data sources
  5. Integrating big data technology with existing infrastructure
  6. Defining strategy
  7. Funding
  8. Infrastructure/architecture
  9. Leadership or organizational issues
  10. Understanding what “big data” is

Characterizing big data as an over-hyped technology certainly seems appropriate if the number one barrier to adoption cited is trying to determine whether it has value to the organization. Now, let’s flash forward to the newly-released 2015 big data investment survey (Survey Analysis: Practical Challenges Mount as Big Data Moves to Mainstream,” Nick Heudecker and Lisa Kart, Gartner, September 3, 2015, subscription required.)

This year only 14% reported big data deployment and 25% had no plans or didn’t know when they would proceed with big data investment.  

Looking at the big data challenges reported by respondents, we see a reordering of responses from the prior year’s data.

“What are your organization’s top hurdles or challenges with big data?”

  1. Determining value (down from 65% to 55%)
  2. Skills and capabilities (up from 30% to 36%)
  3. Risk and governance (up slightly from 32% to 33%)
  4. Defining strategy (up from 28% to 31%)
  5. Funding (up from 25% to 31%)
  6. Integrating multiple data sources (down from 30% to 26%)
  7. Integrating with existing infrastructure (down from 29% to 25%)
  8. Infrastructure/architecture (from 29% to 22%)
  9. Leadership or organizational issues (from 17% to 18%)
  10. Understanding what big data is (down from 17% to 15%)

Determining value is still the top answer cited, but has dropped in frequency from the prior year’s survey. So, given the trend discussed in part 1 of this blog post — that big data has gone from over-hyped to ubiquitous in the space of one year — and the above survey results, I propose the following:

  • The drop in “determining value” responses indicates that organizations have identified projects and use cases, but are faced with the mundane realities of funding and staffing
  • Skills gaps are even more of an issue and must be addressed by making existing experts more effective
  • Challenges with integrating existing data sources and infrastructure may be slowing down big data projects

strata_Hadoop_WorldSnapLogic can help with the big data integration and with making the scarce big data experts more productive by lessening effort spent on manual data integration activities. But it’s up to vendors like us and events like Strata+Hadoop World to help showcase real-world use cases as well as the technologies that enable them. The upcoming Strata event should provide valuable insights into both for attendees.

So – hope to see you in New York for the conference to see what’s new and exciting in emerging big data technologies. Or, if we believe Gartner, see what’s entirely commonplace in analytics, BI and information infrastructure. Either way, we’re looking forward to the conference. (You can learn more about SnapLogic at Strata+Hadoop here.)

OAuth is an open standard for authorization. OAuth provides client applications a ‘secure delegated access’ to server resources on behalf of a resource owner. It specifies a process for resource owners to authorize third-party access to their server resources without sharing their credentials.


SnapLogic has many Snaps that utilize OAuth, including Box, Concur, Eloqua, LinkedIn, Facebook, and Google Analytics. We also support it in a generic way with our REST Snaps that can be used to connect with providers we have yet to build a Snap for, so it’s useful to understand what OAuth is and how it works.

While it is not necessary to have any prior knowledge of OAuth to continue reading, if you wish to understand the OAuth standard at a deeper level, provides a good starting point.

Let’s dive in with a common use case - you (the user) wish to use SnapLogic (the app) to connect to your Google Drive (the server). In this example, your Google Account is the Owner, the Server is Google’s Identify Platform, and the Client is SnapLogic’s REST Snap.

We will use SnapLogic’s REST Snaps to send and receive data to Google’s Drive API, but it needs to be configured first. As we require accessing content from Google, the Snap needs a way of proving to Google that it has been authorized by the user to interact with their Google Drive, while also allowing the user revoke that access directly from their account (Google provides an “Apps connected to your account” settings page where users can easily review and remove apps).

Our first step is to log into the Google Developers Console and create a new Project:

Create SnapLogic Google Drive Project

Once the Project has been created, we must enable Drive API integration:

Enable Drive API integration

Next, we customize the OAuth consent screen by providing a Product name and, optionally, a logo:

Provide product name and logo to the OAuth consent screen

Finally, we configure a new “OAuth 2.0 client ID” credential to identify our Snap to Google when we ask the user for authorization. We use “” URL as the authorized redirect URI.

Create OAuth 2.0 Client ID for Web Application App

Take note of the generated client ID and secret:

Client ID and Client Secret

We can now create a pipeline, add the REST Get Snap, and configure it to request authorization from the user to list their Google Drive files:

Create new pipeline with REST Get Snap, add new OAuth2 account

When creating the REST OAuth2 Account, we use the client ID and secret provided earlier, and configure the remaining fields with the values specified by the Google OAuth for Web Server Apps documentation:

Configure OAuth2 account

The “Header authenticated” checkbox instructs the REST Snap to include an “Authorization” HTTP Header with every request, whose value is the soon-to-be-acquired access token as a Bearer token. Alternatively, you may choose not to enable this setting and instead include an “access_token” query parameter in each request, whose value is the special expression “$account.access_token“, which was created after a successful authorization.

The “redirect_uri” parameter must be provided in both the auth and token endpoint configs, and the value must match the authorized redirect URI configured for the OAuth 2.0 client ID credential created previously. The “response_type” authentication parameter must have a value of “code” (defined by the OAuth specification), and the “scope” parameter defines the Google Drive capabilities being requested (you may wish to modify the scope to what is appropriate for your use case).

The Google-specific “access_type” and “approval_prompt” parameters are also included in the auth endpoint config. An “access_type” value of “offline” requests Google to return a refresh token after the user’s first successful authorization. This allows the Snap to refresh access to the user’s Google Drive without the user being present. The “approval_prompt” parameter value of “auto“, will instruct Google to provide the refresh token only on the very first occasion the user gave offline consent. A value of “force” will prompt the user to re-consent to offline access to acquire a new refresh token.

Clicking the “Authorize” button will start the OAuth Dance. Depending on whether the User is already logged into their Google Account, or is logged to multiple Google Accounts, they may need to login or choose which Account to use. Either way, as long as the user has not already authorized the app, the user will eventually be prompted to allow the REST Snap to access their Google Drive data:

Snap Authorization consent window

These permissions correspond to the “scopes” that were defined previously. You’ll notice that this is a website and the URL address ( starts with the same value as the one entered for the “OAuth2 Endpoint” field above. The Snap has also appended some of the other fields, plus some extra security parameters have been added by the SnapLogic Platform.

Assuming the User gives consent by clicking the Allow button, the next couple of steps happen behind the scenes on within the SnapLogic Platform and are mostly concerned with checking that neither SnapLogic nor Google are being tricked by the other party.

Google will return an Authorization Code to the SnapLogic Platform, which will immediately send a request to the “OAuth2 Token” URL (also entered above) with the authorization code, client ID, client secret and redirect URI parameters. On a successful receipt of that request, Google will once again redirect back to SnapLogic, but this time will include an access token, an access expiration timestamp, plus a refresh token.

If all goes well, the User will be shown the SnapLogic Designer view with the REST OAuth Account form again visible, except now with values for the access and refresh tokens:

OAuth2 Account with acquired access and refresh tokens

The “Refresh” button is now also visible (due to a refresh token having been acquired), allowing the user to manually acquire a new access token when the existing one expires. The user may also choose to enable the “Auto-refresh token” setting to permit the SnapLogic Platform to automatically refresh any expiring access tokens, enabling a true offline mode.

Automatically refresh access tokens by enabling the Auto-refresh token setting

We can click the “Apply” button to associate the authorized OAuth2 Account with the REST Snap. The user can now begin querying the Google Drive API to list their Google Drive files.

The Google Drive API Reference details the full capabilities of what our integration can interact with. For example, we could list the files whose title contains “Target Customers”. To do this, the ”Service URL” is updated to, and we add a “q” query parameter with the search parameter value “title contains 'Target Customers'“:

REST Get search and list GDrive files

Save and close the settings dialog to validate the pipeline and preview the results:

REST Get preview GDrive API results

et voilà, we have successfully completed an OAuth 2.0 Authorization Dance and used the acquired access token to connect with Google Drive! The full power of the Google Drive API is now accessible within SnapLogic.

I joined SnapLogic because of the power of the platform, industry-leading partnerships and the opportunity to be a part of the next wave of data integration innovation. Whether it’s cloud or big data technology adoption, there’s a re-thinking going on in the analytics and data management industry.


With that in mind, I’m excited to be hosting an RSVP-only Enterprise Architecture dinner at the Liberty Hotel in Boston on Wednesday, September 23rd from 5:30 – 8:30pm. In this networking reception, dinner and presentation attendees will hear from our partner PriceWaterhouseCoopers experts on the changing business and technical dynamics that are driving new approaches to enterprise analytics and data management.

Topics discussed will include:

  • Visionary keynote from John Simmons, big data thought leader at PwC
  • Dealing with data integration as more business applications move to the cloud (Salesforce, Workday, ServiceNow, etc.)
  • The impact of big data on the traditional data warehouse and the vision for a data lake
  • Best practices for running integration workloads in the cloud, on the ground and natively in your Hadoop cluster

The SnapLogic Summit Series is tailored to enterprise IT leaders, information architects and big data experts interested in discussing best practices and hearing how other IT organizations are navigating their big data needs and challenges. The Boston dinner promises to be a great opportunity for enterprise architects to learn and share what’s working and what’s not and to take away some new ideas for dealing today’s hybrid IT landscape.

Space is limited. Please RSVP by September 18th.

This weekend we’re updating our  library of 300+ intelligent connectors with a number of new and updated Snaps. Here is a summary of what’s new and what’s being updated.

New Snaps:

snaplogic_snapsUpdated Snaps:

  • Active Directory
  • AWS DynamoDB
  • AWS Redshift
  • Binary
  • Facebook
  • Flow
  • JDBC
  • JMS
  • LDAP
  • LinkedIn
  • MongoDB
  • Microsoft Exchange
  • Microsoft SQL Server
  • MySQL
  • NetSuite
  • OpenAir
  • Oracle RDBMS
  • PostgreSQL
  • REST
  • Salesforce 
  • SAP Hana
  • SOAP
  • Transform Snap Updates (Join, Sequence, Excel Parser, Sort Snap, Aggregate)
  • Twitter
  • Vertica

Please Contact Support if you have any questions or Contact Us if you like to learn more about SnapLogic’s Elastic Integration Platform.

“Life moves pretty fast. If you don’t stop and look around once in a while, you could miss it.” – Ferris Bueller’s Day Off, 1986.

It feels like there is both more uncertainty and more mainstream adoption in the Big Data market in contrast to a year ago at this time. So, a month before the 2015 Strata/Hadoop World event in New York, it feels like the right time to stop and look around at Big Data then and now.

Industry analysts have an interesting perspective on the issue, speaking regularly to both technology vendors and buyers. So let’s look at milestones from the last year from the perspective of Gartner. [A couple of these reports require a Gartner subscription, but I have referenced publicly-accessible blog posts where possible.]

August 4, 2014: Gartner releases its annual “Hype Cycle” report on Big Data technologies. (“Hype Cycle for Big Data 2014,” Frank Buytendijk, Gartner.) [subscription required]

The hype cycle is a proprietary Gartner maturity model for emerging technologies within a specific category. These regularly-issued reports show where a specific technology is in its lifecycle and how long it will take to reach the next phase of its development. The stages are:

Innovation trigger -> Peak of inflated expectations -> Trough of disillusionment -> Slope of enlightenment -> Plateau of productivity.

The notion here (my interpretation) is that every new technology has a hype stage where the buzz is greater than actual adoption or business value achieved. This phase is generally followed by a trough were the buzz wears off, skepticism creeps in and adopters must consider business needs and realities before applying to specific use cases and proceeding to the plateau where technologies contribute to measurable business productivity.

According to the 2014 report, Big Data technologies were either at the peak of hype or well into the trough:

“Big data, as a whole, has crossed the Peak of Inflated Expectations, and is sliding into the Trough of Disillusionment. Once adoption increases, and reality sets in, based on first successes and failures, the peak of hype has passed. Innovation will continue, the innovation trigger is full, but the Trough of Disillusionment will be fast and brutal. Viable technologies will grow quickly, combined with a shakeout of all the vendors that simply jumped on the bandwagon. This is essentially good news. More robust and enterprise-ready solutions will appear; and big data implementations will move from being systems of innovation to mission-critical systems of differentiation.”

Further, these technologies are seen as a long way from adding business value.

“In many cases, transformation is at least two to five years away — or more. In addition, many technologies indicate that they will become obsolete before reaching the Plateau of Productivity.”

Fast-forward to August 2015 and the updated Big Data Hype Cycle report…

But there isn’t a new report. According to an August 20 blog post from Gartner analyst Nick Heudecker, “Big Data Isn’t Obsolete. It’s Normal.” [no subscription required]

“First, the big data technology profile dropped off a few Hype Cycles, but advanced into the Trough of Disillusionment in others. Second, we retired the very popular Hype Cycle for Big Data [emphasis mine]. The reason for both is simple: big data is no longer a topic unto itself. Instead, the various topics formerly encompassing big data evolved into other areas. What other areas?

  • Advanced Analytics and Data Science
  • Business Intelligence and Analytics
  • Enterprise Information Management
  • In-Memory Computing Technology
  • Information Infrastructure

The characteristics that defined big data, those pesky 3 Vs [volume, variety, velocity], are no longer exotic. They’re common.”

So – one year and Big Data goes from a hyped set of technologies still in the realm of the early adopter, to common and simply part of long-standing tech sectors such as analytics, business intelligence and information infrastructure. Why is that? And what does it mean?

In part 2, I’ll delve into adoption drivers and barriers using Gartner’s annual Hadoop adoption surveys from 2014 and 2015.