The Real World

Drea Thomas is the engineering director responsible for the front end technology at SnapLogic – which explains his clear bias toward Designer and all things visual.

When you build a data integration platform, it is easy to expect integrations to look like something from a demo:

But, as we?ve learned over the years, real-world integrations often are a lot messier and can look more like:

The Spring ?12 release of the SnapLogic Server is all about improving the user experience dealing with large and complex pipelines. The goal is to reduce the time required to build a complex pipeline by an order of magnitude. Let me outline some of the improvements we’ve made to the product to address this.

Latency

In previous versions of the product, calculating pass-through fields (something that is quite complex) was done on the server side. This makes a lot of sense because you can centralize the code in one place and re-use it for Designer, or for any application that builds pipelines using SNAPI. But, when your pipelines look like RealWorld, the pipeline definition is quite large – sometimes several megabytes. Sending it back and forth over the wire introduces significant latency.

In Spring ?12, we?ve moved some of the most frequently used server logic into the client to greatly reduce latency. In some cases, this leads to a 300% improvement in Designer performance.

Debugging

In DemoLand, every pipeline runs the first time. In the RealWorld, things don?t always work out that way. In previous versions of the product, when a pipeline resource was badly configured, or there was an error in a query or an expression, you would sometimes discover it only when you ran your pipeline. This required you to read an error log to identify which resource caused the problem and why.

In DemoLand identifying the offending resource and fixing it is fairly straightforward because you only have a handful of resources. In RealWorld, even finding the offending resource on the pipeline canvas is challenging.

In Spring ?12 we?ve done a number of things to improve the situation.

Improved Validation

Components now catch many more problems at validation time – before you even run the pipeline. And the problems that are caught produce better error messages. This tells the developer which resource failed and why.

Resource Highlighting

When a resource fails during pipeline execution, we mark it red and call out the specific error in a tooltip.

Getting Unstuck

In DemoLand, all pipelines run quickly and produce the results you expect. In the RealWorld, pipelines can take far longer than anticipated or produce unexpected results. When you want to identify the bottleneck in a pipeline, you need to see how much time is being spent by which resource and what it is doing.

Previous versions of the product allowed the user to view aggregate statistics at the end of a pipeline run, which gave a good indicator. But what do you do when a pipeline hangs, or takes a really long time to complete? In Spring ?12, we now show resource statistics in real time during pipeline execution in Designer.

Data Tracing

Previous versions of the product allowed you to generate trace files showing you the value of records and they flowed through the pipeline. This enabled you to figure out why your join was not producing the result you expected. But, the trace data was written to files and required you to view the data outside of the Designer.

This allows you to identify problems without having to switch tools, making debugging faster and easier.

In addition to the GUI improvements, there are many, many other improvements and bug fixes. See the release notes for a comprehensive list.

Whether we?ve achieved our ?order of magnitude? goal is up to our customers to judge. But, as someone who builds a lot of pipelines, I can say that Spring ?12 has made my life much easier.
I?m very proud of this release, and I?d encourage everyone to give it a try and send us feedback.

-Drea-


Subscribe to Blog Updates

Quickly connect apps, data, and devices

Start Free Trial
Contact Us Free Trial