SnapLogic announced the availability of new pre-built intelligent connectors – called Snaps – for Microsoft Azure Data Lake Store. The new Snaps provide fast, self-service data ingestion and transformation from virtually any source – whether on-premises, in the cloud or in hybrid environments – to Microsoft’s highly-scalable, cloud-based repository for big data analytics workloads. This latest integration between SnapLogic and Microsoft Azure helps enterprise customers gain new insights and unlock business value from their cloud-based big data initiatives.
The suite of Snaps available in the Spark mode enable us to ingest and land data from a Hadoop ecosystem and transform the data by leveraging the parallel operations such as map, filter, reduce or join on a Resilient Distributed Datasets (RDD), which is a fault-tolerant collection of elements that can be operated on in parallel.
There are various formats available for data storage in HDFS. These file formats support one or more compression formats that affect the size of data stored in the HDFS file system. The choice of file formats and compression depends on various factors like desired performance for read or write specific use case, desired compression level for storing the data. Continue reading “Ingestion, Transformation and Data Flow Snaps in Spark”
This month, we are planning the delivery of our latest Snap Release.
NetSuite Update will be added to the NetSuite Snap Pack. This Snap provides the ability to update the records of an object in NetSuite.
This release also introduces the Splunk Search Snap, which executes a search query using Splunk’s REST API.
As with all releases, we make continual improvements on our Snaps. Changes in this release focus on Database Snaps, NetSuite Snap Pack, Script Snap, and others. See the Release Notes for more information.
To kick off our bi-weekly TechTalk webinar series featuring tips and tricks for the SnapLogic Elastic Integration Platform, yesterday’s session focused on scaling integrations. As discussed in the webinar, there are two basic approaches here – the first is using Child Pipelines when the work you want to parallelize is compute heavy. The second approach is to use a Splitter and Join Snap when the work is computationally light.
Our TechTalks are designed for SnapLogic customers and partners and serve as interactive ask-the-experts training sessions. We invite you to register here and come with questions. Here’s what you can expect:
- A 30-minute session focused on a specific topic where your questions will be answered by our cloud integration specialists
- A review of some of the recent Q&As submitted on the SnapLogic Developer Community Forum
At SnapLogic, we’re always focused on delivering new Snaps and updating our library of over 160 existing Snaps. Snaps are the building blocks of a pipeline that perform a single function such as read, write or act on data. They are found in the Snap catalog, accessible on the left hand side of the SnapLogic Designer. Drag a Snap from the Snap catalog onto the workspace to use it in a pipeline.
We are pleased to announce the addition of the following Snaps for the SnapLogic Elastic Integration Platform:
PostgreSQL Snap Pack
With this Snap Pack, you can lookup records, fetch data, and execute SQL statements to delete, update or insert data within a specified PostgreSQL table.
The latest Snap added to the JIRA Snap Pack, JIRA Transition, adds support for performing a transition on a JIRA issue by supplying the updates in an input document.
SAP IDoc Listener
The SAP Snap Pack was expanded with the SAP IDoc Listener. This Snap receives SAP IDoc requests and sends the IDoc data contained in the requests to the output view.
The following Snap Packs are available as Beta Releases:
- Reltio Snap Pack: Read, write, and delete resource objects in Reltio.
- Xactly Snap Pack: Support for searching Xactly is available.
With the August 2014 Snap Release, we are also delivering minor updates and fixes for Oracle Stored Procedure, File Reader/Writer/Delete, Multi File Reader, ZipFile Read/Write, Fixed Width Parser. These updates will be pushed this evening, with no required down time. Look for the Release Notes at the time of release for more information. We also have some additional resources below for you to check out:
- Cloud Integration? There’s a Snap for That!
- The SnapLogic Developer Community
- Documentation for the SnapLogic Elastic Integration Platform
Today’s post is from SnapLogic summer intern, Rishabh Mehan: My name is Rishabh Mehan and I’m currently a student at New York Institute of Technology. I’ve been doing computer programming/software development for 8 years and this summer I’ve been working at SnapLogic as an intern. My main focus has been the new SnapLogic Developer Community, which went live with our Summer 2014 release.
One of the things that excited me the most about what we’re working on at SnapLogic (other than Elastic Integration, Big Data and powering cloud analytics of course) is the fact that we’re setting out to enable our customers with the potential to move to the cloud and expand the kind of data and application integrations that are possible.
Our new SnapLogic Developer Community was created to make it easier for developers to expand the current list of Snaps according to their needs, as well as to have the ability to create a completely new Snap. With a very simple approach, the SnapLogic Developer Community provides a knowledge base and environment to share ideas for developing on our cloud integration platform.
The Developer Community provides a base for collaborative learning, and our team and other developers will always be there to help you, as well as ask you for help. This is how developers work. The current structure of the Community is diversified into three segments:
- Get Set
- Get Started
- Get Collaborative
- Brief overview of the architecture
- Introduction to the technology and terminology
- Snaps and pipelines
- Set up the on-premises Snaplex
- Set up your developer environment
- Snap Development
- Demo Snaps and guides
- Documentation for your reference
- Community forum to discuss your issues
- Post your responses and help others
- Learn about what other developers are doing
After being provisioned as a Developer in your SnapLogic organization, you are all set to enter the Developer Community and go through each and every document available.
Additionally, we have developed easy multi-platform installers for developers, which help in setting up your own on-premises Snaplex and develop without depending on any other resources. The package also provides you with the Snap Developer Kit (SDK) and our Snaps for developers. You can easily reference them, use them and – if you’d like to – modify them.
Here’s an example of a SnapLogic Windows Installer:
All of the documentation will guide you through the process, so even if you don’t know anything about the development of Snaps, you really don’t have to worry. So login and get started today. We’re looking forward to hearing your feedback!
We’re always coming out with new Snaps, which are the building blocks of a SnapLogic pipeline. Snaps perform a single function such as read, write, or acting on data. They are found in the Snap catalog, accessible on the left hand side of Designer. Simply drag a Snap from the Snap catalog onto the workspace to use it in a pipeline, which is a data flow. Snaps are classified as either Core Snaps, meaning that they are available to all customers, or by the Snap Pack for particular applications, technologies or utilities. We also have a Snap SDK for customers and partners to build their own custom Snaps.
Application and Database Snaps
Here is a list of Snap Packs from A-Z. Note that this list is always growing as we introduce new Snaps every few weeks. Each Snap Pack contains many Snaps. For example, here’s a screenshot of what’s in the Active Directory Snap, which was released in May 2014.
- Active Directory
- AWS (Account Creation, EMR Executor, Hive Executor)
- AWS Redshift
- Google Analytics
- Google DFA
- Google Directory
- Microsoft Dynamics AX
- Microsoft Exchange
- Microsoft SQL Server
- Oracle RDBMS
- Oracle EBS
- Hadoop (HDFS Reader and Writer)
Core Snaps are available to all customers with the SnapLogic Elastic Integration Platform. They include the following Snap Packs: Binary, Email, Fixed Width, Flow, REST, Script, SOAP, Transform and XML. Here are examples of Core XML Snaps in the Snap Pack:
Here are examples of SnapLogic Transform Snaps:
For more information on SnapLogic Snaps, be sure to visit our SnapStore and you can learn more about the key concepts in this post. Here’s a demonstration of the SnapLogic Elastic Integration Platform in action to help you get a better understanding of Snaps, Pipelines, Patterns as well as our Monitoring Dashboards.