February 2019 Release: SnapLogic Resumable Pipelines

Hi! In this video, I will show what SnapLogic Resumable Pipelines are and how you can use them for guaranteed data delivery.

Resumable Pipelines is a new capability that complements some of our existing platform functionalities, including SnapLogic Ultra Pipelines.

Ultra Pipelines allow you to run real-time integrations and guaranteed delivery for a higher quality of service in data and application integrations.  However, the execution will use an at-least-once guaranteed delivery when using Ultra Pipelines.

Resumable Pipelines work a little differently in that they provide an exactly-once guaranteed delivery mechanism and are ideal for a low volume of data deliveries and application integration use cases. Snaps, including Aggregating Functions,  Group-By Functions, Sorting Functions or any function that saves the state, are supported.

An example of using a Resumable Pipeline is that it will go into a suspended state when the pipeline fails to continue executing. These suspended pipelines can be resumed manually through the UI or programmatically through the public API. When this happens, it turns a pipeline execution into a stateful execution or a stateful mode, meaning that data will be stored on the local Snaplex. Once it’s ready to resume it will reload those documents into the execution mode for continued processing.  This process supports endpoint failure.

Now let’s look at this pipeline, Service 1 is our first endpoint and Service 2 is our second endpoint. Turn on Resumable Pipelines in the pipeline properties dialog, and check the Suspendable/Resumable checkbox. Once checked, you will see a warning that the pipeline will change from streaming to non-streaming. Validation, however, will continue to use streaming behavior.

First, I will call Service 1 and Service 2, and run the pipeline so that all the services are online and working. Next, I’m going to switch to the Manager view and disable Service 1. Then, I’ll go back into the Design view and re-execute the pipeline. At this point, you’ll see Service 1 turn orange, and when we view the execution statistics, they show that there is an error in this pipeline.

The pipeline is suspended and the state is stored locally to the Snaplex. I can either resume it by clicking play here, or I can go to the Dashboard view and select the row or the execution that got suspended and click play to resume the pipeline.

When the service resumes, the documents are loaded back into the pipeline and the pipeline tries to execute. Service 2 was able to start and complete. The Join Snap, however, is waiting for Service 1, which is why the rest of the pipeline is still orange.

Let’s take a look at the Dashboard view to see that the pipeline is in a suspended state.

Now, let’s go back to the Manager view and turn the service back on.  Then, let’s resume the pipeline by clicking the play button. We can view the pipeline statistics here and the extra details show the different state changes, what time they happened, when things resume, and who took the action. In short, the pipeline resumes from the point of failure, so if it failed on Service 1, it would resume from that point to continue processing.

Finally, let’s take a look at the Designer view to go over additional features. Here we can view all the statuses that can be selected for filtering, and the duration that you see here, is the total runtime duration, so if a pipeline was suspended for 10 minutes, then it’s not going to count that 10 minutes as a part of the duration.

One final feature to note is that currently the pipeline can be suspended for two weeks, we plan on making that time period configurable in the future.