Building an IoT Application in SnapLogic: Figuring out Pipelines and Tasks

In the first post in this series, we talked about the challenges of integrating the Internet of Things into the enterprise. In the next few blog posts, we are going to build a simple IoT application that illustrates all the major aspects of working with SnapLogic and hardware.  In this post, we’re going to skip device details, but at a high level we’ll have:

  • A sensor somewhere (on-premises, from an API, etc.) that produces data that includes a “color” payload;
  • An LED on-premise, attached to our local network, conveniently hooked up to look like a REST endpoint;
  • Two pipelines, one on-premise, one in the cloud.

Hardware Considerations

Some IoT hardware is designed to be cloud-native, and will generally have a publish/subscribe relationship with a cloud server (such as MQTT).  This is very easy to work with from a security standpoint, since the output of these devices are accessible from anywhere.

Other devices instead communicate on their local network.  Assuming your local network isn’t internet accessible, this can create problems in talking to the device.  Usefully, the SnapLogic Control Plane (depicted, in a manner of speaking, as the rightmost rectangle below) comes to our rescue here.

Control-Data Plane Diagram
A “graphical depiction” of the control plane (right) communicating with various data planes, including a Hadooplex at the bottom. We see the artist is somewhat defensive about his rendering of the pachyderm likeness.

Continue reading “Building an IoT Application in SnapLogic: Figuring out Pipelines and Tasks”

SnapLogic Spring 2015 Has Arrived: iPaaS Meets IoT

snapLogic_iotAccording to a recent Gartner report, “Systems of innovation that leverage mobile and Web applications will demand coherent data integration and application integration techniques to gain just-in-time insights and activate responses across continuously evolving events. Systems of record that interact with a growing range of endpoints (including machine-generated data and IoT-related data) will involve linking of business flows with data access and analysis of event streams.”

Today we announced the general availability of the SnapLogic Spring 2015 release, which expands our focus on big data integration with new support for the Internet of Things (IoT). Along with many new and updated Snaps, the new release introduces a set of Snaps for Message Queuing Telemetry Transport (MQTT), which is a lightweight machine-to-machine connectivity protocol. The new Snaps will allow SnapLogic customers to create pipelines that connect to an MQTT broker for sensors, mobile and connected devices and stream data to analytical and other applications in real time. Later this year, we’re looking at supporting additional IoT protocols such as AMQP, CoAP, OMA Lightweight M2M, and ETSI Smart M2M.

“The Spring 2015 release is yet another milestone for our company as we continue to focus on platform innovation and ensuring our enterprise customers can connect data, applications and things faster.”

– Gaurav Dhillon, founder and CEO, SnapLogic

The Spring 2015 release introduces updates and innovations throughout the SnapLogic Elastic Integration Platform, which has recently been recognized by Ovum as an enterprise integration platform as as service (iPaaS) leader. Spring 2015 updates include:

  • Pipeline Execution Statistics: When users execute a pipeline, a duration chart now provides an overview of where the pipeline is spending its time as it is processing the stream of JSON documents. The Pipeline Execution Statistics dialog displays how much CPU time is being consumed and how much memory is being allocated for each Snap in real time, allowing for optimized pipeline design and improved system performance.

SnapLogic Pipeline Statistics

  • Low-Latency Ultra Pipelines: Administrators can now set up multiple receiving Ultra Pipelines nodes to ensure there is no single point of failure and achieve higher availability and reliability. Ultra Pipelines now have improved error handling and performance monitoring in the SnapLogic dashboard.
  • Public Monitoring API: In addition to an API for managing users and groups, a new public monitoring API has been introduced allowing customers to proactively query pipeline status and take advantage of their existing monitoring tools to track the health of SnapLogic integrations.
  • Multi-Instance Kerberos Authentication: The SnapLogic Hadooplex runs natively as a YARN application on the Hadoop cluster, whether it is running behind the firewall or in the cloud. The Spring 2015 release introduces support for multiple instances authenticated with different Kerberos users on a single Hadoop cluster.
  • Hadoop for Humans: With an easy-to-use cloud-based designer, customers can create SnapReduce pipelines that now support router and copy functions as well as advanced expressions for more complex processing and analytics use cases.
  • New and Updated Snaps: The Spring 2015 release introduces new Snaps for Anaplan, Google BigQuery, SAP Hana and Splunk. The Salesforce Snap now integrates with CipherCloud’s security and compliance platform and the release also includes many other updated Snaps for applications and technologies including: AWS Redshift, Eloqua, Excel, Expensify, HP Vertica, HDFS, JDBC, LDAP, Microsoft Active Directory, Microsoft SQL Server, MySQL, NetSuite, OpenAir, Oracle, PostgreSQL, PGP, REST, SAP BAPI, ServiceNow, SOAP, and Tableau.


All SnapLogic Elastic Integration Platform customers have been updated to the Spring 2015 release. To learn more about what’s new, visit: