Case study: Connecting the dots at Box

“Data needs to be delivered to a user in the right place at the right time in the right volume.”

Spoken by veteran SnapLogic user Alan Leung, Senior Enterprise Systems Program Manager at Box, Alan provides insight in this case study for why a cloud-first ecosystem with self-service integration is the right solution for many enterprise companies. Just as Box is on a mission to improve and innovate cloud-based file storage, internally they have moved toward a cloud-centric infrastructure that benefits from a cloud-based integration platform.

connecting-the-dots-at-box

Read the full case study here or take a look at some highlights below:

  • Overall problem: Box needed to more efficiently integrate cloud-based applications, including Salesforce, Zuora, NetSuite, and Tableau, all of which they relied on for daily operations.
  • Challenges: The primary challenge was around APIs – each application’s endpoints for integration purposes behaved differently, limiting abilities to build useful connections quickly and resulting in a series of disjointed silos. Manual upload and download processes caused a strain on resources and a waste of time and effort.
  • Goal: To satisfy the need to aggregate the business data piling up in various applications into a cloud-based warehouse to enable self-service, predictive analytics.
  • Solution needed: A cloud-based integration platform that vastly reduced or eliminated the time-consuming manual processes the users faced.
  • Solution found: With the SnapLogic Elastic Integration Cloud, Alan and his team benefitted from:
    • A platform that did not require sophisticated technical skills
    • The agility to enable quick and efficient integration projects
    • The ability to handle both structured and unstructured data at speed
    • An enhanced ability to quickly analyze and make sense of so much data, allowing the company to “rapidly pivot [our] operations to seize opportunity across every aspects of the business.”

For a quick snapshot, Box currently has 23 applications connected through the platform, resulting in 170 data pipelines processing over 15 billion transactions daily. They also have eliminated the need to build a single interface internally; and an ongoing benefit of Box’s partnership with SnapLogic is that more Snaps are always being created and can be implemented for future integration needs.

Learn more about all of our customers here, and stay tuned for more customer stories.

Gartner Names SnapLogic a Leader in the 2017 Enterprise iPaaS Magic Quadrant

For the second year in a row, SnapLogic has been named a Leader in Gartner’s Magic Quadrant for Enterprise Integration Platform as a Service (iPaaS).

Gartner evaluated iPaaS vendors on “completeness of vision” and “ability to execute.” Those named to the Leaders quadrant, as Gartner noted in the report, “have a solid reputation, with notable market presence and a proven track record in enabling … their platforms are well-proven and functionally rich, with regular releases to rapidly address this fast-evolving market.”

In a press release issued today, SnapLogic CTO James Markarian said of the recognition: “Since our inception, we have been laser-focused on delivering a modern enterprise integration platform that is specifically designed to manage the data and application integration demands of today’s hybrid enterprise technology environments. Our Enterprise Integration Cloud eliminates the complexity of legacy integrations, providing a platform that supports fast and easy self-service integration.”

The Enterprise iPaaS Magic Quadrant is embedded below. We’d encourage you to download the complete report as it provides a comprehensive review of all the vendors and the growing market.

Gartner 2017 iPaaS MQ

Thanks to all of SnapLogic’s customers, partners, and employees for the ongoing support and for making SnapLogic’s Enterprise Integration Cloud a leading self-service integration platform connecting applications, data, and things.

Podcast: James Markarian and David Linthicum on New Approaches to Cloud Integration

SnapLogic CTO James Markarian recently joined cloud expert David Linthicum as a guest on the Doppler Cloud Podcast. The two discussed the mass movement to the cloud and how this is changing how companies approach both application and data integration.

In this 20-minute podcast, “Data Integration from Different Perspectives,” the pair discuss how to navigate the new realities of hybrid app integration, data and analytics moving to the cloud, user demand for self-service technologies, the emerging impact of AI and ML, and more.

You can listen to the full podcast here, and below:

 

Cloud fluency: Does your data integration solution speak the truth?

There’s been a lot of cloud-washing in the enterprise data integration space — vendors are heavily promoting their cloud solutions, yet for many, only a skinny part of their monolithic apps has been “cloudified.”

In an era of “alternative facts,” it’s important to make technology decisions based on truths. Here is an important one: A big data integration solution built on genuine, made-for-the-cloud platform as a service (PaaS) technology offers important benefits including:

  1. Self-service integration by “citizen integrators,” without reliance on IT
  2. For IT organizations, the ability to easily connect multiple data sets, to achieve a bespoke enterprise tech environment

These are in addition to the traditional benefits of cloud solutions: no on-premise installation; continuous, no-fuss upgrades; and the latest software innovation, delivered automatically.

Why “built for the cloud” matters

You can’t get these benefits with “cloudified” software that was originally invented in 1992. Of course, I’m referring to Informatica; while the company promotes its cloud capabilities, the software largely retains a monolithic architecture that resides on-premises, and does most of its work there, too.

In contrast, SnapLogic is purpose-built for the cloud, meaning there are no legacy components that prevent the data and application integration service from running at cloud speed. Data streams between applications, databases, files, social and big data sources via the Snaplex, a self-upgrading, elastic execution grid.

In more everyday terms, SnapLogic has 100% cloud fluency. Our technology was made for the cloud, born in the cloud, and it lives in the cloud.

The consumerization of data integration

Further to point 1 above, “citizen integrators,” industry futurists like R. “Ray” Wang have been talking about the consumerization of IT for more than half a decade. And that is exactly what SnapLogic has mastered. Our great breakthrough, our big innovation, is that we have consumerized the dungeon-like, dark problem of data integration.

Integration used to be a big, boring problem relegated to the back office. We’ve brought it from the dungeon to the front office and into the light. It is amazing to see how people use our product. They go from one user to hundreds of users as they get access to data in a secure, organized and appropriately access-controlled manner. But you don’t have a cast of thousands of IT people enabling all this; users merely help themselves. This is the right model for the modern enterprise.

“An ERP of one”

As for the second major benefit of a true cloud solution — a bespoke enterprise tech environment, at a fraction of the time and cost of traditional means — here’s a customer quote from a young CEO of a hot company that’s a household name.

“Look, we’ve got an ‘ERP of one’ by using SnapLogic — a totally customized enterprise information environment. We can buy the best-of-the-best SaaS offerings, and then with SnapLogic, integrate them into a bespoke ERP system that would cost a bajillion dollars to build ourselves. We can custom mix and match the capabilities that uniquely fit us. We got the bespoke suit at off-the-rack prices by using SnapLogic to customize our enterprise environment.”

To my mind, that’s the big payoff, and an excellent way to think about SnapLogic’s value. We are able to give our customer an “ERP of one” faster and cheaper than they could have ever imagined. This is where the world is going, because of the vanishingly low prices of compute power and storage, and cloud computing.

Today you literally can, without a huge outlay, build your own enterprise technology world. But you need the glue to realize the vision, to bring it all together. That glue is SnapLogic.

Find out more about how and why SnapLogic puts best-of-breed enterprise integration within every organization’s grasp. Register for this upcoming webinar featuring a conversation with myself, industry analyst and data integration expert David Linthicum, and Gaurav Dhillon, SnapLogic’s CEO and also an Informatica alumnus: “We left Informatica. Now you can, too.”

SNP_Thumb_Informatica

Gaurav Dhillon is CEO at SnapLogic. Follow him on Twitter @gdhillon.

Finally viable: Best-of-breed enterprise environments

It’s one of the oldest, most contentious rivalries in the enterprise application arena: What’s better, best-of-breed environments or single-vendor suites? Since the turn of the century, suite vendors have argued that their approach avoids the steep data integration challenges that can be inherent with best-of-breed. On the flip side, point solution vendors say that enterprise suites pack in a lot of “dead wood” but don’t offer the real functionality, or customization potential, that is needed.

However, unlike religion and politics, this is one argument that is headed toward extinction. The biggest barrier to best-of-breed strategies — data integration — is, hands down, easier by an order of magnitude today, thanks to built-for-the-cloud app integration solutions that eliminate previous barriers. As a result, best-of-breed application environments aren’t just viable, they’re readily attainable.

Two dimensions of data integration

There are two ways in which data integration has dramatically improved with native cloud solutions: on the back end, between the applications themselves, and on the front end, from the user experience perspective.

On the back end, one of the first-order implications of a robust data model is the number of connectors a data integration solution provides. SnapLogic has hundreds of Snaps (connectors) and that’s not coincidental. Our library of Snaps proves our suitability to the modern world; it’s an order of magnitude easier to build and support a SnapLogic connector than an Informatica connector — the integration tool of choice for last-century best-of-breed environments — because our data model fits the modern world.

As a result, customers are up and running with SnapLogic in a day or two. In minutes we can show customers what SnapLogic is capable of doing. This is in comparison to Informatica and other legacy integration technologies; here, developers or consultants can work for weeks or months on the same integration project and still have nothing. They can’t deliver quickly due to the limitations of the underlying technology.

The ease of big data integration with SnapLogic has profound implications on the user experience. Instead of having to beg analysts to do ETLs (extract, transfer, and load) to pull the data set they need, SnapLogic users can get whatever data they want, themselves. They can then analyze it and get answers far faster than under previous best-of-breed regimes.

These are not subtle differences.

The economics of cloud-based integration

The subscription-based pricing model of cloud-based integration services further democratizes data access. Instead of putting the burden on IT to buy and implement an integrated application suite — which can cost upwards of $100 million in a large enterprise — cloud-based integration technology can be acquired at a nominal per-user fee, charged to a corporate credit card. Lines of business have taken advantage of this ease of access, making their own technology moves with the full knowledge and support of IT.

For IT organizations that have embraced their new mission of enablement, the appeal of cloud-based data integration is clear. In addition to allowing business users to work the way they want to, the cloud-based solution is infinitely easier to customize, and deploy and support globally. And it offers an obvious answer to the question, “Do I want to continue feeling the pain of using integrated app suites or do I want to join the new century?”

Find out more about how and why SnapLogic puts best-of-breed integration within every organization’s grasp. Register for this upcoming webinar featuring a conversation with myself, industry analyst and data integration expert David Linthicum, and Gaurav Dhillon, SnapLogic’s CEO and also an Informatica alumnus: “We left Informatica. Now you can, too.”

SNP_Thumb_Informatica

James Markarian is CTO at SnapLogic. Follow him on Twitter @jamesmarkarian.

Deep Dive into SnapLogic Winter 2017 Snaps Release

By Pavan Venkatesh

Data streams with Confluent and migration to Hadoop: In my previous blog post, I explained how future data movement trends will look. In this post, I’ll dig into some of the exciting things we announced as part of the Winter 2017 (4.8) Snaps release. This will also address future data movement trends for customers who want to move data to the cloud from different systems or migrate to Hadoop.

Major highlights in 2017 Winter release (4.8) include:

  • Support of Confluent Kafka – A distributed messaging system for streaming data
  • Teradata to Hadoop – A quick and easy way to migrate data
  • Enhancements to the Teradata Snap Pack: On the TPT front, customers can quickly load/update/delete data in Teradata
  • The RedShift Multi-Execute Snap – Allows multiple statements to be sequentially executed, so customers can maintain business logic
  • Enhancements to the MongoDB Snap pack (Delete and Update) and the DynamoDB Snap pack (Delete and Delete-item)
  • Workday Read output enhancements – Now it’s easier for the downstream systems to consume
  • Netsuite Snap Pack improvements -Users can now submit asynchronous operations
  • Security feature enhancements – Including SSL for MongoDB Snap Pack and invalidating database connection pools when account properties are modified
  • Major performance improvement while writing to an S3 bucket using S3 File Writer – Users can now configure a buffer size in the Snap so larger blocks are sent to S3 quickly

Confluent Kafka Snap Pack

Kafka is a distributed messaging system based on publish/subscribe model with high throughput and scalability. It is mainly used for ingestion from multiple sources and then sent to multiple downstream systems. Use cases include website activity tracking, fraud analytics, log aggregation, sales analytics, and others. Confluent is the company that provides the enterprise capability and offering for open source Kafka.

Here at SnapLogic we have built Kafka Producer and Consumer Snaps as part of the Confluent Snap Pack. A deep dive into Kafka architecture and its working will be a good segue before going into the Snap Pack or pipeline details.

kafka-cluster

Kafka consists of single or multiple Producers that can produce messages from a single or multiple upstream systems, and single or multiple Consumers that consume messages as part of downstream systems. A Kafka cluster constitutes one or more servers called Brokers. Messages (key and value or just the value) will be fed into higher level abstraction called Topics. Each Topic can have multiple messages from different Producers. User can also define different Topics for new category of messages. These Producers write messages to Topics and Consumers consume from one or more Topics. Also Topics are partitioned, replicated, and persisted across Brokers. Messages in the Topics are ordered within a partition and each of these will have a sequential ID number called offset. Zookeeper usually maintains these offsets but Confluent calls it coordination kernel.

Kafka also allows configuring a Consumer group where multiple Consumers are part of it, when consuming from a Topic.

With over 400 Snaps supporting various on-prem (relational databases, files, nosql databases, and others) and cloud products (Netsuite, SalesForce, Workday, RedShift, Anaplan, and others), the Snaplogic Elastic Integration Cloud in combination with the Confluent Kafka Snap Pack will be a powerful combination for moving data to different systems in a fast and streaming manner. Customers can realize benefits and generate business outcomes in a quick manner.

With respect to the Confluent Kafka Snap Pack, we support Confluent Version 3.0.1 (Kafka v0.9). These Snaps abstract the complexities and users only have to provide configuration details to build a pipeline which moves data easily. One thing to note is that when multiple Consumer Snaps are used in a pipeline and have been configured with the same consumer group, then each Consumer Snap will be assigned a different subset of partitions in the Topic.

kafka-producer

kafka-consumer

pipeline1

In the above example, I built a pipeline where sales leads (messages) stored in local files and MySQL are sent to a Topic in Confluent Kafka via Confluent Kafka Producer Snaps. The downstream system Redshift will consume these messages from that Topic via the Confluent Kafka Consumer Snap and bulk load it to RedShift for historical or auditing needs. These messages are also sent to Tableau as another Consumer to run analytics on how many leads were generated this year, so customer can compare this against last year.

Easy migrations from Teradata to Hadoop

There has been a major shift where customers are moving from expensive Teradata solutions to Hadoop or other data warehouse. Until now, there has not been an easy solution in transferring large amounts of data from Teradata to big data Hadoop. With this release we have developed a Teradata Export to HDFS Snap with two goals in mind: 1) ease of use and 2) high performance. This Snap uses the Teradata Connector for Hadoop (TDCH v1.5.1). Customers just have to download this connector from the Teradata website in addition to the regular jdbc jars. No installation required on either Teradata or Hadoop nodes.

TDCH utilizes MapReduce (MR) as its execution engine where the queries gets submitted to this framework, and the distributed processes launched by the MapReduce framework make JDBC connections to the Teradata database. The data fetched will be directly loaded into the defined HDFS location. The degree of parallelism for these TDCH jobs is defined by the number of mappers (a Snap configuration) used by the MapReduce job. The number of mappers also defines the number of files created in HDFS location.

The Snap account details with a sample query to extract data from Teradata and load it to HDFS is shown below.

edit-account

terradata-export

 

The pipeline to this effect is as follows:

pipeline2

As you can see above, you use just one Snap to export data from Teradata and load it into HDFS. Customers can later use HDFS Reader Snap to read files that are exported.

Winter 2017 release has equipped customers with lots of benefits, from data streams, easy migrations, to enhancing security functionality, and performance benefits. More information on the SnapLogic Winter 2017 (4.8) release can be found in the release notes.

Pavan Venkatesh is Senior Product Manager at SnapLogic. Follow him on Twitter @pavankv.

From helicopter to enabler: The new face of enterprise IT

Can an IT organization effectively run a 2017 business on 25-year-old technology? As someone who played a large hand in developing the data integration technology in question — at Informatica, where I was CTO for nearly two decades — I can tell you that the answer is simple: “No.”

A vastly different primordial landscape

That said, I know that when Informatica was created, it was the best technology for data integration at the time. The world was a lot simpler in 1992: there were five databases that mattered, and they were all pretty similar. There were just a few ERP systems: Oracle, SAP and a young PeopleSoft. Informatica was ideally suited to that software baseline, and the scale-up UNIX platforms of that era. The web, obviously, was not in the picture.

IT organizations were also a lot simpler in 1992. If any business person wanted new tech functionality — a new workstation added to a network, or a new report from a client/server system — they put their request into the IT queue, because that was the only way to get it.

IT is still important; it’s just different

Fast-forward 25 years to 2017. Almost everything about that primordial technology landscape, when Informatica roamed the world, is different. For example, now there’s the web, the cloud, NoSQL databases, and best of breed application strategies that are actually viable. None of these existed when Informatica started. Every assumption from that time — the compute platform, scale-up/scale-out, data types, data volumes and data formats — is different.

IT organizations are radically different, too. The command-and-control IT organization of the past has transformed into a critical enablement function. IT still enables core operations by securing the enterprise and establishing a multitude of technology governance frameworks. But the actual procurement of end-user technology, such as analyzing data aggregated from across systems and across the enterprise, is increasingly in the hands of business users.

In other words, the role of IT is changing, but the importance of IT isn’t. It’s like parenting; as your kids grow your role changes. It’s less about helicoptering and more about enabling. Parents don’t become less important, but how we deliver value evolves.

This is a good analog to the changes in enterprise IT. The IT organization wants to enable users because it’s pretty impossible to keep up with the blistering pace of business growth and change. If the IT organization tries to control too much, at some point it starts holding the business back.

Smart IT organizations have realized their role in the modern enterprise is to help their business partners become more successful. SnapLogic delivers a vital piece of required technology; we help IT organizations to give their users the self-service data integration services they need, instead of waiting for analysts to run an ETL through Informatica to pull the requested data together. By enabling self-service, SnapLogic is helping lines of business — most companies’ biggest growth drivers — to reach their full potential. If you’re a parent reading this, I know it will sound familiar.

Here’s another way to find out more about why IT organizations are embracing SnapLogic as a critical enabler: readSnapLogic’s new whitepaper that captures my conversation with Gaurav Dhillon, SnapLogic’s CEO and also an Informatica alumnus: “We left Informatica. Now you can, too.”

snp-informatica-wp-1000x744