Acquire Prep DeliverThis month’s webinar was all about big data and how customers can use the SnapLogic Elastic Integration Platform and SnapReduce 2.0. In yesterday’s live discussion and demonstration, we talked with SnapLogic Chief Scientist Greg Benson, who is Professor of Computer Science at University of San Francisco and has worked on research in distributed systems, parallel programming, OS kernels and programming languages for +20 years. The webinar went into the details of SnapReduce 2.0 for big data integration (more on that later), but first we talked about Hadoop in terms of where it’s been, where it’s going and what the implications are for traditional enterprise data warehousing. Here’s a brief recap below:

  • The Big Data journey: Greg talked about early initiatives and use cases and how so much “data exhaust” was getting left on the floor.
  • Hadoop and data warehousing: Many believe Hadoop and the Hadoop ecosystem will eventually replace what relational data warehouses do today because of the economics of Hadoop and what has now become possible in terms of data storage. Right now, though, they’re complementary.
  • Implications on data integration: There was a good discussion about why old tech won’t work in the new era of SMAC and the variety of sources and use cases for both streaming and batch data processing.
  • The need to acquire, prepare and deliver big data: This includes both batch and streaming processing for a new generation of ETL/ELT.

Following the big data discussion, Greg and the team moved on to SnapReduce 2.0 and the concept of elastic scale out, with a Q&A session to address customer and prospect questions. Check out the presentation slides and questions below:

How do you make SnapLogic work across two clouds…say Salesforce in one cloud and social data in another cloud?
The first thing to understand is that the SnapLogic Snaplex respects data gravity. From this question, it looks like “services” are seen as separate clouds. SnapLogic easily connects separate services and applications and can do so either in our cloud or through a Snaplex running on premises or in a VPC. As we covered in the webinar, with SnapReduce, the Snaplex can also now run natively as a YARN application within a Hadoop cluster.

Is it possible to do transformations on data before it is actually written on to HDFS?
Yes, absolutely. When streaming data into HDFS, the data can be filtered or transformed before writing to HDFS.

Are the data flows (pipelines) converted into jar files or something like pig?
MapReduce ode is generated directly and issues to Hadoop as a jar.

Can SnapLogic directly write a .tde file for Tableau or is it a CSV file which Tableau later converts to its native format?
The SnapLogic Tableau Snap directly writes to a TDE.

Once I have read data from HDFS using HDFS Reader, would I be able to do join with data sitting on a source / database (viz. Oracle) / SQL Server)? If so, where will that pipeline run?
Yes, this can be done and in this scenario the pipeline would run in Hadoop, but on a single Hadoop node. It would not run as a MapReduce job.

Let us know in the Comments section below if you have any more questions, and be sure to watch the webinar recording if you couldn’t make yesterday’s live event and sign up for the Early Access Program. In the meantime, here’s a brief SnapReduce 2.0 demonstration:

“A unified approach to connecting growing data volumes, types and sources as well as new and legacy business applications and proliferating APIs is essential to becoming a truly elastic enterprise.” – Gaurav Dhillon, SnapLogic CEO

SnapLogic_ipaasCreating a unified platform for connecting data, applications and APIs - this is what we’re focused on here at SnapLogic and we’ve had a busy few weeks with the announcement of our Summer 2014 Release. Check out some of the highlights below, and take a look at our recent webinars to learn more about the specific topics that are most relevant to you and your cloud integration and big data integration challenges, questions and needs.

  • Integration Developers News addresses how SnapLogic helps iPaaS “cross the chasm” with visibility, security and pre-built integrations for apps, data, APIs and Hadoop.
  • Jason Bloomberg’s article, Because Bulldozers and Buses Don’t Fly, covers traditional ETL and ESB strategies, as well as the elasticity required for today’s cloud integration initiatives. Jason described SnapLogic as being “at the heart of cloud-centric architecture” and says that “it’s the SnapLogic Designer toll that is downright sexy.”
  • 451 quote451 Research impact report reviews SnapLogic’s growth, readying for ‘big data’ and the summer release. Check out Carl Lehmann’s outlook on cloud integration, at right.
  • Gaurav talks SaaS at Enterprise Founders Upfront, speaking about his career and experiences in integration and what he sees happening in the industry.
  • SnapLogic was named an AlwaysOn Global 250 Winner, recognized as an elastic integration leader for creating technology innovations for the global silicon valley. As Tony Perkins, founder and editor of AlwaysOn says, “As the expansion of the mobile and cloud markets continue, the companies on this year’s AlwaysOn Global 250 are racing to provide business and consumer users with the best products and services. The new generation of hardware and software solutions is making it possible for businesses to become even more efficient and profitable.”

Additionally, a few internal highlights from the past few months have been:

You can also find more information on our newest Snaps for cloud integration, and watch the Summer 2014 recorded webinar for more details on all of the new and improved features of last month’s release.

Over the past few weeks I’ve outlined what we believe are the 6 primary requirements for integration platform as a service (iPaaS):

  1. Fully-Functional Cloud-based Service
  2. Single Platform for Big Data, Application and API Integration
  3. Elastic Scale
  4. Built on Modern Standards
  5. Broad Cloud and On-Premises Connectivity
  6. Self-Service for Citizen Integrators

5 Signs You Need a New Cloud Integration StrategyThe final post in the series focuses on what is probably the most disruptive change to traditional approaches to data and application integration – the demand for self-service from the business and the rise of the so-called Citizen Integrator.

Integration tools have historically been the domain of the specialists. The black belts. The plumbers. They have also required distinct skill sets and domain knowledge – from enterprise application integration (EAI) and enterprise service buses (ESBs) and services oriented architecture (SOA) to extract, transformation and load (ETL) and enterprise data warehousing (EDW) and business intelligence (BI). Different teams. Different tools. Different worlds.

With apologies to S.E. Hinton, that was then, this is now…

A few years ago, Gartner predicted that Citizen Developers will build at least 25% of new business applications by 2014. They’ve also predicted that in a few years, the CMO will spend more on IT than the CIO and the majority of new integration flows will be developed outside the control of IT departments. Of course, it’s not a new phenomenon that people want access to trusted information – including when, where and how they want it – but the The Nexus of Forces (also known has SMAC - social, mobile, analytics, cloud) has shaken up traditional roles and responsibilities when it comes to the business and IT relationship. In the impatient enterprise, waiting days, weeks or months to get access to this information is not an option.

Simplified_iPaaS_GraphicWith cloud deployments comes cloud expectations, and speed and time to value are the primary drivers for integration platform as a service (iPaaS). Enter the Citizen Integrator.

Citizen Integrators generally have a solid background in data, but they are not classically trained on the legacy tools of the trade. And if they do have experience with ETL/ELT or EAI tools, newer approaches to solving both old and new integration challenges may require some re-thinking (as explained in this iRobot video). If the Citizen Integrator has an applications background, they are typically well-versed in SaaS and have a good grasp of specific end-points and business processes. If they have an analytics background, the Citizen Integrator may have used business intelligence tools and may now even have the coveted “Data Scientist” title.  On average, however, Citizen Integrators are in the line business. The have solid Excel skills and a role that requires more and more access to data from multiple systems. Digital marketers are a good example of a team that may look to tools like SnapLogic, AWS Redshift and Tableau for self-service integration of data from multiple channels, low-cost data warehousing and advanced data visualization. And they want the entire infrastructure up and running in days, not months or years.

If this is where you see integration going at your company – whether the focus is connecting data, applications or APIs, here are a few features that we believe a cloud integration service is going to require in order to meet the needs of your Citizen Integrators:

“By running natively on Hadoop, SnapLogic delivers powerful application integration and extends the reach, performance and utilization of big data platforms.” – Gaurav Dhillon, CEO, SnapLogic

Webinar_icon_August2014A big focus for us this month, and something our customers are becoming more and more interested in, is big data and how SnapReduce 2.0 comes into play. Next week we’ll be hosting a live webinar called “Hadoop for Humans: Introducing SnapReduce 2.0” that will address the ways in which SnapReduce brings usability, connectivity and soft-ware defined architecture advantages to our Elastic Integration Platform. Join us next Wednesday, August 13th at 10:00am PST/1:00pm EST to find out how SnapLogic delivers:

  • Improved Big Data Acquisition: Go beyond developer tools like Sqoop and Flume and access more than 160 pre-built connects (Snaps) within Hadoop using SnapLogic’s easy-to-use graphical user interface.
  • Better Big Data Preparation: Double your data scientist productivity by simplifying complex data shaping tasks such as transformations, joins and unions.
  • Universal Big Data Delivery: Make Hadoop analytics results easily available to off-cluster applications and data stores such as statistical packages and business intelligence (BI) or data visualization tools with powerful data integration and transformations, aggregation, joins and sorts as well as Snaps for Amazon Redshift, Oracle, SAP, MongoDB, Netezza and many more.

This interactive webinar will feature SnapLogic Chief Scientist Greg Benson, who is Professor of Computer Science at the University of San Francisco and has worked on research in distributed systems, parallel programming, OS kernels and programming languages for 20 years. Check out this blog post featuring a discussion with Greg on SnapReduce and how it eases the big data journey for more information before next week’s webinar. You can register for the webinar here, and be sure to take a look at our Big Data and SnapReduce pages to learn more.

We receFB_Posts_SummerLaunch14_720x266ntly added a series of new training videos to highlight some features and enhancements of the SnapLogic Elastic Integration Platform. Check out the videos below to learn more about some of the user interface updates from our Summer 2014 release, and how to automatically map known fields with the SnapLogic Data Mapper.

SnapLogic Summer 2014 User Interface Updates

This video features some of the new enhancements we recently made to the SnapLogic Elastic Integration Platform user experience including new features in the pipeline and dashboard tabs.

The SnapLogic Data Mapper

In this video, see how you can use fields and data types with the SmartLink button to automatically map known fields with the SnapLogic Data Mapper. In coming springs, there will be additional learning that will pick up other data that has been mapped. This video also covers Expression Builder which gives access to more comprehensive information about capabilities to manipulate data.

Check out our full video site for additional trainings and demonstrations.

Today’s post is from SnapLogic summer intern, Rishabh Mehan: My name is Rishabh Mehan and I’m currently a student at New York Institute of Technology. I’ve been doing computer programming/software development for 8 years and this summer I’ve been working at SnapLogic as an intern. My main focus has been the new SnapLogic Developer Community, which went live with our Summer 2014 release.

Screen Shot 2014-07-31 at 2.49.37 PMOne of the things that excited me the most about what we’re working on at SnapLogic (other than Elastic Integration, Big Data and powering cloud analytics of course) is the fact that we’re setting out to enable our customers with the potential to move to the cloud and expand the kind of data and application integrations that are possible.

Our new SnapLogic Developer Community was created to make it easier for developers to expand the current list of Snaps according to their needs, as well as to have the ability to create a completely new Snap. With a very simple approach, the SnapLogic Developer Community provides a knowledge base and environment to share ideas for developing on our cloud integration platform.

The Developer Community provides a base for collaborative learning, and our team and other developers will always be there to help you, as well as ask you for help. This is how developers work. The current structure of the Community is diversified into three segments:

  1. Get Set
  2. Get Started
  3. Get Collaborative

Get Set

  • Brief overview of the architecture
  • Introduction to the technology and terminology
  • Snaps and pipelines
  • Set up the on-premises Snaplex

Get StartedScreen Shot 2014-07-31 at 3.43.39 PM

  • Set up your developer environment
  • Snap Development
  • Demo Snaps and guides
  • Documentation for your reference

Get Collaborative

  • Community forum to discuss your issues
  • Post your responses and help others
  • Learn about what other developers are doing

After being provisioned as a Developer in your SnapLogic organization, you are all set to enter the Developer Community and go through each and every document available.

Additionally, we have developed easy multi-platform installers for developers, which help in setting up your own on-premises Snaplex and develop without depending on any other resources. The package also provides you with the Snap Developer Kit (SDK) and our Snaps for developers. You can easily reference them, use them and – if you’d like to – modify them.

Here’s an example of a SnapLogic Windows Installer:


All of the documentation will guide you through the process, so even if you don’t know anything about the development of Snaps, you really don’t have to worry. So login and get started today. We’re looking forward to hearing your feedback!

Gaurav Founders UpfrontA few weeks ago the folks at Work-Bench invited SnapLogic founder and CEO Gaurav Dhillon to speak about his experience in the integration industry and what he sees for today and moving forward, as a guest at their Enterprise Founders Upfront event in New York. The #FoundersUpfront series was created to allow founders of tech companies to share their experiences and perspectives when starting a company, executing its growth and development and and ultimately scaling a successful enterprise technology business. Gaurav was invited in part for of his role in co-founding Informatica back in the 90′s, and switching gears in the application and data integration to found SnapLogic because of his vision for a truly cloud-based integration platform as a service (iPaaS).

During the Founders Upfront interview, Gaurav spoke with Jon Lehr, Venture Director of Work-Bench and founder of the New York Enterprise Tech Meetup, and addressed the processes and ever-changing trends in enterprise sales over the past few years, as well as current SaaS friendly industries in the market today.

A few additional topics below as covered by the Work-Bench recap blog below:

  1. The early days of Informatica and how whether at a startup or a large company, everyone works hard in technology but sometimes it takes more than that to get products out in front.
  2. Which industry is leading cloud adoption including the financial services industry typically considered to be quick to adopt new technologies, while the retail industry is surprisingly – and aggressively – also SaaS savvy. Gaurav used an example here of SnapLogic customer Target, which created mobile apps and got more sophisticated in their understanding of customers.
  3. The disruption of legacy tech incumbents and recognizing that - when comparing cloud computing and existing business units – in order to survive, “you have to take huge risks and sail your company into harm’s way.”

You can also check out the full conversation between Jon and Gaurav, or watch Gaurav talk about the SnapLogic Elastic Integration Platform as a way to solve the Integrator’s Dilemma here. Enjoy!