A few weeks ago, Rishabh Mehan wrote about the new SnapLogic Community – for Developers By Developers. The initial goal of the community was focused on API development and custom Snap building for customers and partners. The site has expanded and we’re starting to see some great participation. Here’s a brief summary: You can now access all of our documentation, Elastic Integration Platform and Snap Development: You can now download an on-premises Snaplex (known as a “Groundplex”), the Java SDK, public Snap Packs and installers for Mac, Linux and Windows. The Community forum for asking questions and sharing best practices has expanded dramatically. You can now select a category and post your question or answer to the forum. The SnapLogic team has started using the tag: didyouknow when posting tips and tricks. Here are some examples:
- Which is the most used Snap?
- How do you remove white spaces from a string?
- If I can’t use pipeline parameters directly in my SQL select statement, how do I do it?
- Can you comment on the suitability for using Snaplogic for moving (not transforming) large binary (100s MB or GB) files using FTP?
- If I can’t use pipeline parameters directly in my SQL select statement, how do I do it?
In your questions (and answers) you can now add screenshots, documents and formatting to ensure it’s clear and detailed. To help us (and others) know what’s most important, you can vote up and down questions and answers: The team is adding features to the SnapLogic Developer Community every day and we’re excited to see our customers and partners joining the discussion. As we continue to see more and more Citizen Integrators logging into our cloud integration service, we’re looking forward to with you. Simply login to Developer.SnapLogic.com with your SnapLogic username and password.
This month’s webinar was all about big data and how customers can use the SnapLogic Elastic Integration Platform and SnapReduce 2.0. In yesterday’s live discussion and demonstration, we talked with SnapLogic Chief Scientist Greg Benson, who is Professor of Computer Science at University of San Francisco and has worked on research in distributed systems, parallel programming, OS kernels and programming languages for +20 years. The webinar went into the details of SnapReduce 2.0 for big data integration (more on that later), but first we talked about Hadoop in terms of where it’s been, where it’s going and what the implications are for traditional enterprise data warehousing. Here’s a brief recap below:
- The Big Data journey: Greg talked about early initiatives and use cases and how so much “data exhaust” was getting left on the floor.
- Hadoop and data warehousing: Many believe Hadoop and the Hadoop ecosystem will eventually replace what relational data warehouses do today because of the economics of Hadoop and what has now become possible in terms of data storage. Right now, though, they’re complementary.
- Implications on data integration: There was a good discussion about why old tech won’t work in the new era of SMAC and the variety of sources and use cases for both streaming and batch data processing.
- The need to acquire, prepare and deliver big data: This includes both batch and streaming processing for a new generation of ETL/ELT.
Following the big data discussion, Greg and the team moved on to SnapReduce 2.0 and the concept of elastic scale out, with a Q&A session to address customer and prospect questions. Check out the presentation slides and questions below:
How do you make SnapLogic work across two clouds…say Salesforce in one cloud and social data in another cloud?
The first thing to understand is that the SnapLogic Snaplex respects data gravity. From this question, it looks like “services” are seen as separate clouds. SnapLogic easily connects separate services and applications and can do so either in our cloud or through a Snaplex running on premises or in a VPC. As we covered in the webinar, with SnapReduce, the Snaplex can also now run natively as a YARN application within a Hadoop cluster.
Is it possible to do transformations on data before it is actually written on to HDFS?
Yes, absolutely. When streaming data into HDFS, the data can be filtered or transformed before writing to HDFS.
Are the data flows (pipelines) converted into jar files or something like pig?
MapReduce ode is generated directly and issues to Hadoop as a jar.
Can SnapLogic directly write a .tde file for Tableau or is it a CSV file which Tableau later converts to its native format?
The SnapLogic Tableau Snap directly writes to a TDE.
Once I have read data from HDFS using HDFS Reader, would I be able to do join with data sitting on a source / database (viz. Oracle) / SQL Server)? If so, where will that pipeline run?
Yes, this can be done and in this scenario the pipeline would run in Hadoop, but on a single Hadoop node. It would not run as a MapReduce job.
Let us know in the Comments section below if you have any more questions, and be sure to watch the webinar recording if you couldn’t make yesterday’s live event and sign up for the Early Access Program. In the meantime, here’s a brief SnapReduce 2.0 demonstration:
“A unified approach to connecting growing data volumes, types and sources as well as new and legacy business applications and proliferating APIs is essential to becoming a truly elastic enterprise.” – Gaurav Dhillon, SnapLogic CEO
Creating a unified platform for connecting data, applications and APIs - this is what we’re focused on here at SnapLogic and we’ve had a busy few weeks with the announcement of our Summer 2014 Release. Check out some of the highlights below, and take a look at our recent webinars to learn more about the specific topics that are most relevant to you and your cloud integration and big data integration challenges, questions and needs.
- Integration Developers News addresses how SnapLogic helps iPaaS “cross the chasm” with visibility, security and pre-built integrations for apps, data, APIs and Hadoop.
- Jason Bloomberg’s article, Because Bulldozers and Buses Don’t Fly, covers traditional ETL and ESB strategies, as well as the elasticity required for today’s cloud integration initiatives. Jason described SnapLogic as being “at the heart of cloud-centric architecture” and says that “it’s the SnapLogic Designer toll that is downright sexy.”
- 451 Research impact report reviews SnapLogic’s growth, readying for ‘big data’ and the summer release. Check out Carl Lehmann’s outlook on cloud integration, at right.
- Gaurav talks SaaS at Enterprise Founders Upfront, speaking about his career and experiences in integration and what he sees happening in the industry.
- SnapLogic was named an AlwaysOn Global 250 Winner, recognized as an elastic integration leader for creating technology innovations for the global silicon valley. As Tony Perkins, founder and editor of AlwaysOn says, “As the expansion of the mobile and cloud markets continue, the companies on this year’s AlwaysOn Global 250 are racing to provide business and consumer users with the best products and services. The new generation of hardware and software solutions is making it possible for businesses to become even more efficient and profitable.”
Additionally, a few internal highlights from the past few months have been:
You can also find more information on our newest Snaps for cloud integration, and watch the Summer 2014 recorded webinar for more details on all of the new and improved features of last month’s release.
Over the past few weeks I’ve outlined what we believe are the 6 primary requirements for integration platform as a service (iPaaS):
- Fully-Functional Cloud-based Service
- Single Platform for Big Data, Application and API Integration
- Elastic Scale
- Built on Modern Standards
- Broad Cloud and On-Premises Connectivity
- Self-Service for Citizen Integrators
The final post in the series focuses on what is probably the most disruptive change to traditional approaches to data and application integration – the demand for self-service from the business and the rise of the so-called Citizen Integrator.
Integration tools have historically been the domain of the specialists. The black belts. The plumbers. They have also required distinct skill sets and domain knowledge – from enterprise application integration (EAI) and enterprise service buses (ESBs) and services oriented architecture (SOA) to extract, transformation and load (ETL) and enterprise data warehousing (EDW) and business intelligence (BI). Different teams. Different tools. Different worlds.
With apologies to S.E. Hinton, that was then, this is now…
A few years ago, Gartner predicted that Citizen Developers will build at least 25% of new business applications by 2014. They’ve also predicted that in a few years, the CMO will spend more on IT than the CIO and the majority of new integration flows will be developed outside the control of IT departments. Of course, it’s not a new phenomenon that people want access to trusted information – including when, where and how they want it – but the The Nexus of Forces (also known has SMAC - social, mobile, analytics, cloud) has shaken up traditional roles and responsibilities when it comes to the business and IT relationship. In the impatient enterprise, waiting days, weeks or months to get access to this information is not an option.
With cloud deployments comes cloud expectations, and speed and time to value are the primary drivers for integration platform as a service (iPaaS). Enter the Citizen Integrator.
Citizen Integrators generally have a solid background in data, but they are not classically trained on the legacy tools of the trade. And if they do have experience with ETL/ELT or EAI tools, newer approaches to solving both old and new integration challenges may require some re-thinking (as explained in this iRobot video). If the Citizen Integrator has an applications background, they are typically well-versed in SaaS and have a good grasp of specific end-points and business processes. If they have an analytics background, the Citizen Integrator may have used business intelligence tools and may now even have the coveted “Data Scientist” title. On average, however, Citizen Integrators are in the line business. The have solid Excel skills and a role that requires more and more access to data from multiple systems. Digital marketers are a good example of a team that may look to tools like SnapLogic, AWS Redshift and Tableau for self-service integration of data from multiple channels, low-cost data warehousing and advanced data visualization. And they want the entire infrastructure up and running in days, not months or years.
If this is where you see integration going at your company – whether the focus is connecting data, applications or APIs, here are a few features that we believe a cloud integration service is going to require in order to meet the needs of your Citizen Integrators:
“By running natively on Hadoop, SnapLogic delivers powerful application integration and extends the reach, performance and utilization of big data platforms.” – Gaurav Dhillon, CEO, SnapLogic
A big focus for us this month, and something our customers are becoming more and more interested in, is big data and how SnapReduce 2.0 comes into play. Next week we’ll be hosting a live webinar called “Hadoop for Humans: Introducing SnapReduce 2.0” that will address the ways in which SnapReduce brings usability, connectivity and soft-ware defined architecture advantages to our Elastic Integration Platform. Join us next Wednesday, August 13th at 10:00am PST/1:00pm EST to find out how SnapLogic delivers:
- Improved Big Data Acquisition: Go beyond developer tools like Sqoop and Flume and access more than 160 pre-built connects (Snaps) within Hadoop using SnapLogic’s easy-to-use graphical user interface.
- Better Big Data Preparation: Double your data scientist productivity by simplifying complex data shaping tasks such as transformations, joins and unions.
- Universal Big Data Delivery: Make Hadoop analytics results easily available to off-cluster applications and data stores such as statistical packages and business intelligence (BI) or data visualization tools with powerful data integration and transformations, aggregation, joins and sorts as well as Snaps for Amazon Redshift, Oracle, SAP, MongoDB, Netezza and many more.
This interactive webinar will feature SnapLogic Chief Scientist Greg Benson, who is Professor of Computer Science at the University of San Francisco and has worked on research in distributed systems, parallel programming, OS kernels and programming languages for 20 years. Check out this blog post featuring a discussion with Greg on SnapReduce and how it eases the big data journey for more information before next week’s webinar. You can register for the webinar here, and be sure to take a look at our Big Data and SnapReduce pages to learn more.
We recently added a series of new training videos to highlight some features and enhancements of the SnapLogic Elastic Integration Platform. Check out the videos below to learn more about some of the user interface updates from our Summer 2014 release, and how to automatically map known fields with the SnapLogic Data Mapper.
SnapLogic Summer 2014 User Interface Updates
This video features some of the new enhancements we recently made to the SnapLogic Elastic Integration Platform user experience including new features in the pipeline and dashboard tabs.
The SnapLogic Data Mapper
In this video, see how you can use fields and data types with the SmartLink button to automatically map known fields with the SnapLogic Data Mapper. In coming springs, there will be additional learning that will pick up other data that has been mapped. This video also covers Expression Builder which gives access to more comprehensive information about capabilities to manipulate data.
Check out our full video site for additional trainings and demonstrations.
Today’s post is from SnapLogic summer intern, Rishabh Mehan: My name is Rishabh Mehan and I’m currently a student at New York Institute of Technology. I’ve been doing computer programming/software development for 8 years and this summer I’ve been working at SnapLogic as an intern. My main focus has been the new SnapLogic Developer Community, which went live with our Summer 2014 release.
One of the things that excited me the most about what we’re working on at SnapLogic (other than Elastic Integration, Big Data and powering cloud analytics of course) is the fact that we’re setting out to enable our customers with the potential to move to the cloud and expand the kind of data and application integrations that are possible.
Our new SnapLogic Developer Community was created to make it easier for developers to expand the current list of Snaps according to their needs, as well as to have the ability to create a completely new Snap. With a very simple approach, the SnapLogic Developer Community provides a knowledge base and environment to share ideas for developing on our cloud integration platform.
The Developer Community provides a base for collaborative learning, and our team and other developers will always be there to help you, as well as ask you for help. This is how developers work. The current structure of the Community is diversified into three segments:
- Get Set
- Get Started
- Get Collaborative
- Brief overview of the architecture
- Introduction to the technology and terminology
- Snaps and pipelines
- Set up the on-premises Snaplex
- Set up your developer environment
- Snap Development
- Demo Snaps and guides
- Documentation for your reference
- Community forum to discuss your issues
- Post your responses and help others
- Learn about what other developers are doing
After being provisioned as a Developer in your SnapLogic organization, you are all set to enter the Developer Community and go through each and every document available.
Additionally, we have developed easy multi-platform installers for developers, which help in setting up your own on-premises Snaplex and develop without depending on any other resources. The package also provides you with the Snap Developer Kit (SDK) and our Snaps for developers. You can easily reference them, use them and – if you’d like to – modify them.
Here’s an example of a SnapLogic Windows Installer:
All of the documentation will guide you through the process, so even if you don’t know anything about the development of Snaps, you really don’t have to worry. So login and get started today. We’re looking forward to hearing your feedback!