In my last blog post, we focused on performance improvements associated with Google BigQuery Snaps in Release 4.11. This week, in a continuing look into the details of Fall 2017 Release news, we focus on additional performance gains from Release 4.11. We’re not talking just about the usual updates, but speed.
Two of our transformation Snaps, Sort and In-Memory Lookup, received significant performance improvements in the 4.11 release. The SnapLogic Development team was on a mission to eliminate unnecessary latency that might go unnoticed under mild loads but is more evident with larger data volumes. Seeing an increase in processing efficiency and an increase in the size of datasets that can be handled in-memory were a big priority.
The result is much faster processing with greater resiliency for larger data volumes. Just in case you think this blog post is just metaphors, let’s consider the numbers (fasten your seatbelts!):
Your results will, of course, vary based on Snaplex node hardware profile, data elements, data size, etc., but the above chart shows a very consistent and significant improvement of Sort performance. The Sort Snap is in the top 10 percent of most used Snaps (out of more than 400), so chances are you’ve already experienced the benefit of this particular performance improvement.
Furthermore, if you have downstream transformation Snaps like Join, Aggregate, In-Memory Lookup, etc. in your pipeline, sorting your data first will result in greater overall performance.
Speaking of the In-Memory Lookup Snap, that too got a performance boost. Not only is the In-Memory Lookup Snap faster, it can handle more data in-memory. Consider the following evidence:
Again, same small print from above applies* (*your results will vary based on hardware configuration and data), but the SnapLogic testing shown in the above chart clearly shows both a consistent and significant improvement in performance, compared against the same hardware configuration and dataset. Result? The ability to handle much larger datasets with the In-Memory Lookup Snap.
You will continue to see us focus on performance improvements big and small now and in upcoming releases. After all, every millisecond counts at scale.
So, take your new supercharger for a spin and let us know what you think.
