SnapLogic Kafka Snaps in Action

Apache Kafka

In today’s business world big data is generating a big buzz. Besides the searching, storing and scaling, one thing that clearly stands out is – stream processing. That’s where Apache Kafka comes in.

Kafka at a high level can be described as a publish and subscribe messaging system. Like any other messaging system, Kafka maintains feeds of messages into topics. Producers write data into topics and consumers read data out of these topics. For the sake of simplicity, I have linked to the Kafka documentation here.

In this blog post, I will demonstrate a simple use case where Twitter feeds to a Kafka topic and the data is written to Hadoop. Below are the detailed instructions of how users can build pipelines using the SnapLogic Elastic Integration Platform.
Continue reading “SnapLogic Kafka Snaps in Action”

Snaplex Thresholds and Pipeline Queuing

As the integration market continues to mature, there is a constant demand to support and process more complex data and process flows. When applications process large data, they often run out of resources and become unresponsive, leaving users confused and unhappy. Gauging resources and alerting users with appropriate messages are some of the most important factors of ideal software. In the Winter 2016 release of the SnapLogic Elastic Integration Platform, we introduced the concept of pipeline queuing, which allows users to define thresholds for their Snaplexes, and when thresholds are reached, any further requests to it are queued until the next resources are available. Continue reading “Snaplex Thresholds and Pipeline Queuing”