Yahoo Pipes for the rest of your data….
Pasha Sadri from Yahoo gave the final keynote presentation at the mySQL conference on his work with Yahoo Pipes. He began his presentation talking about problems he was having grinding through craigslist and other RSS feeds to extract useful information. He thought about how he could build something that solved similar problems for others trying to do the same kinds of things.
Pipes is a RSS mashup tool that allow you to combine RSS feed and apply transformations on the data and create a new feed. If you haven’t seen it, you ought to check it out. Several good technical reviews here, here and here.
When I first saw Pipes, I was shocked at how familiar it seemed. They were taking data from different sources, applying transformations and generating new feeds. Their browser based UI contained data feed and transformation components that all could be interconnected and configured very easily.
It was familiar because that exactly what we were building here at SnapLogic. Sure there are lots of differences, but suddenly our ideas didn’t seem so strange. One of our biggest challenges seems to collapse overnight. Before Pipes, we worried that people wouldn’t really understand the idea of mashing up data feeds. With Pipes, they were doing it with RSS, so describing SnapLogic as ‘Yahoo Pipes for all kinds of data’ was perfectly clear.
It was great to hear Pasha’s presentation. If you weren’t there, you can see a similar talk that he gave at Google earlier this month. I didn’t get a chance to talk to Pasha yesterday, but I’m going to try to track him down to show him what we’ve been up to.