Estimating Load and Performance of Integrations

One of the most common requests I hear from colleagues and customers is, “How do I estimate how many jobs I can run on a node and how fast will they run?” The immediate and most accurate answer is… it depends. While it may seem a flippant answer, it is a succinct response to a complex multidimensional problem. Let’s examine the variables. Continue reading “Estimating Load and Performance of Integrations”

Building an IoT Application in SnapLogic, Part II: Speeding Through the Last Mile

The last post in this ongoing IoT series detailed the creation of a cloud-based Ultra Pipeline to do the bulk of the work for our IoT application. We described the following application:

  • A sensor somewhere (on-premises, from an API, etc.) that produces data that includes a “color” payload;
  • An LED on-premise, attached to our local network, conveniently hooked up to look like a REST endpoint;
  • Two pipelines, one on-premise, one in the cloud.

Continue reading “Building an IoT Application in SnapLogic, Part II: Speeding Through the Last Mile”

REST GET and the SnapLogic Public APIs for Pipeline Executions

As a part of a wider analytics project I’m working on, analyzing runtime information from the SnapLogic platform, I chose to use the functionality exposed to all customers, the Public API for Pipeline Monitoring API and the REST API. These two things are combined in this post. I started by reading the documentation (of course), which shows the format of the request and response. So I created a new pipeline and dropped a REST GET Snap on the canvas:

Continue reading “REST GET and the SnapLogic Public APIs for Pipeline Executions”

Two-way SSL with SnapLogic’s REST Snap

SnapLogic_word_cloudThere are lots of ways for a client to authenticate itself against a server, including basic authentication, form-based authentication, and OAuth.

In these cases, the client communicates with the server over HTTPS, and the server’s identify is confirmed by validating its public certificate. The server doesn’t care who the client is, just as long as they have the correct credentials. Continue reading “Two-way SSL with SnapLogic’s REST Snap”

Everything So Brittle: What ESB Middleware Means in 2015

intellyx_snaplogicThis week Jason Bloomberg, industry expert and author of the book, The Agile Architecture Revolution, delivered a great presentation called: It’s the 21st Century , Why Isn’t Your Data Integration Loosely Coupled? He reviews some of the challenges with traditional middleware connectors and tight coupling, where any change in the data format or interface requirements for either end of any interaction requires an update of the connector. The result is a very brittle integration environment with too many single points of failure. (See the iPaaS requirement for fluidity in hybrid deployments.)

Jason goes on to summarize the benefits of loose coupling and review the limitations of Web Services, XML Schemas and rigid, strongly typed data formats. The presentation reviews REST, JSON and what he calls the “schema-less data trap.” He concludes with the following data integration do’s and don’ts:


  • Plan ahead for horizontal scalability and fault tolerance (cloud friendliness)
  • Favor document-centric data formats


  • Use rigid, centralized middleware
  • Rely heavily on fixed schemas

Specifically on the topic of the enterprise service bus (ESB), Jason has this to say:

  • no-esbEssentially the ESB is traditional middleware with Web Services added
  • The ESB is typically an older, single-point-of failure, hub-based deployment
  • The ESB is not “cloud friendly” and not up to the task – not designed to be horizontally scalable and state information is maintained centrally

He has this to say about SnapLogic “design-time introspection” and our Snaps, which he calls, “next-generation connectors”:

  • They can gather the metadata automatically so that integration configuration can be performed dynamically
  • SnapLogic lets you automate the configuration of the integration so you have greater flexibility as you deal with changing interactions.

You can check out the entire presentation, which also includes a SnapLogic Elastic Integration Platform demonstration by Craig Stewart, here and I’ve embedded the slides below. Be sure to also check out Jason’s review of SnapLogic Snaps: Re-Inventing Intelligent Connectors and our whitepaper: Why Buses Don’t Fly in the Cloud.

The Promise of the ESB without the Overhead

The Enterprise Service Bus (ESB) is a model for inter-application communication that was developed in the era of the Service Oriented Architecture (SOA). Today when people mention the term ESB, the first things that typically come to mind are long, expensive development cycles, inflexibility, and complexity. While designed with good intentions, most legacy ESB implementations grew into overloaded business logic monsters, causing performance issues, especially when attempting to deal with large volumes of data. What people ultimately want from an ESB is a high-speed, agile, and scalable infrastructure that is ubiquitous, able to work with modern data and services both on-premises and in the cloud.

At the heart of legacy ESB technology are the Simple Object Access Protocol (SOAP) and eXtensible Markup Language (XML), which are both becoming outdated, designed at a time when the Service Oriented Architecture (SOA) was mainstream. In the new enterprise IT world of the SMACT (social, mobile, analytics, cloud and things), especially when it comes to big data, message-based ESB technology begins looking more like a horse and buggy than a modern approach to application and data integration. The application and data worlds have evolved and the requirements have converged. In that evolution the ESB is no longer a viable option.

no_esbSnapLogic’s answer to low-latency application and data integration lies in a high-speed, guaranteed delivery system that works across disparate sources, whether on-premise or in the cloud. Designed for high-speed data streaming, transformation and delivery, SnapLogic’s Ultra Pipelines are designed to meet the demands of today’s modern systems, supporting virtually any type of interface or data representation (JSON, XML, SOAP, REST, etc.) with low-latency, real-time processing, and guaranteed data delivery.

SnapLogic has removed the heavy lifting typically found in ESB systems and implementations, moving the user from a long and costly development paradigm into a quick configuration paradigm. Whether your requirement is event-based, real-time, streaming or scheduled batch-oriented data integration, SnapLogic’s modern, JSON-centric, RESTful platform has a clear advantage over traditional ESB solutions that were built for last generation’s structured application and data management challenges.

Ultra-PipelinesGraphic_NoBG (1)But what about microservices? As the enterprise continues to migrate more toward increasingly granular services, all the more reason to adopt an agile, modern integration platform that is equipped to deal with evolutionary changes. SnapLogic’s concept of the ESB rests on a system that was designed from the ground up to deliver the ultimate in flexibility, agility, and elasticity.  As EMA Research recently noted:

“Even as pre-built Snaps eliminate work cycles and manual processes related to data integration, SnapLogic Ultra Pipelines accelerates data delivery. This parallel approach to addressing the twin challenges of big data integration and real-time processing reinforces SnapLogic’s position as a leader in the data integration and iPaaS space.”

Randy Hamilton is a Silicon Valley entrepreneur and technologist who writes periodically about industry related topics including the cloud, big data and iOT.  Randy has held positions as Instructor (Open Distributed Systems) at UC Santa Cruz and has enjoyed positions at Basho (Riak NoSQL database), Sun Microsystems, and Outlook Ventures, as well as being one of the founding members and VP Engineering at

SnapLogic Ultra Pipelines: A Modern Alternative to Legacy ESB Technology

snaplogic_ultra_pipelineSnapLogic recently introduced Ultra Pipelines, which is a turbo boost to our JSON-centric data pipeline technology, enabling reliable, real-time completion of time-sensitive enterprise tasks and processes. In this Integration Developer News review, SnapLogic product manager Tim Lui describes Ultra Pipelines this way:

“Unlike some iPaaS or on-prem integration solutions that require a trigger or an event (such as a write or update) to activate an integration, our real-time integration technology puts in place a long-running connection that is always listening and waiting for instructions. That way, Ultra Pipelines are instantly ready to execute when there is new data.”

He goes on to note, “something we find customers really like is they gain this added speed — without having to give up scalability. So, customers gain huge performance benefits and still get the elastic architecture we’ve always offered.”

In a comprehensive article about the advantages of lightweight “microservices” compared to “the heavyweight, middleware-centric enterprise service buses (ESBs) that drove many of the SOA implementations in the 2000s,” SOA author and industry expert Jason Bloomberg talks about Avoiding Dumb Pipes, observing,“SnapLogic customers can leverage their Ultra Pipelines to build microservices that serve as “web scale pipes.” (His article has started an interesting dialog on LinkedIn.)

Enterprise Management Associates also published a write up on Ultra Pipelines (SnapLogic Accelerates iPaaS Data Delivery with Ultra Pipelines), outlining how the Elastic Integration Platform is well-suited for real-time and batch-oriented application and data integration use cases. EMA concludes:

“Even as pre-built Snaps eliminate work cycles and manual processes related to data integration, SnapLogic Ultra Pipelines accelerates data delivery. This parallel approach to addressing the twin challenges of big data integration and real-time processing reinforces SnapLogic’s position as a leader in the data integration and iPaaS space.”

I’ve embedded the EMA review below. You can learn more about SnapLogic Ultra Pipelines here and for additional thoughts on why the ESB is wrong approach for managing modern cloud and big data integration requirements, be sure to read our whitepaper: Why Buses Don’t Fly in the Cloud as well as Dave Linthicum’s paper: The Death of Traditional Integration:How the Changing Nature of IT Mandates New Approaches and Technologies.