Yahoo Pipes, impressive!

Every once in a while you are surfing the web and come across something that you really think, whoa. For me this is most of the time something cool in graphical design or user interaction. But a week ago I came across Yahoo Pipes and just at that moment I also had a good reason to use it.

Now the best way to start is to describe what Yahoo Pipes is. The description from the Yahoo Pipes website is:

Pipes is a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.

So a visual way to mashup different sources of data. So what you for example could do is combine Flickr, Facebook and Last.fm data into one nice tailored data feed and then you won’t have to combine all the sources of data in you web application. But you’ll be able to just do one JSON call and get all you data in the format that you want.

My use case
The reason I found this interesting is because I ran into a problem with the last.fm API. The problem is in the artist.getSimilar method. This method can be used to get a number of artist similar to the artist name provided. This works quite well when you look at the normal XML return value. But when you switch to the JSON output it doesn’t work correctly anymore. The reason for this, is that this method in XML returns a similarartists tag. This tag has a collection of child elements with the name artist. But the similarartists also has a attribute with the name artist. Probably in the last.fm implementation the switch between XML to JSON format is automatic. And in JSON all the child elements will be discarded. So basically rendering the JSON output of the artist.getSimilar method useless.

In Pipes
So I was interested to see whether Yahoo Pipes would be able to fix this problem by parsing the XML and then outputting the proper values. So I created an account, logged in and started dragging and dropping elements around. I created the following flow:

I used the following components to parse the last.fm XML and create a API call that would be usable in JSON.

URL Builder
The first call is a URL builder. This builder has a base URL which is that URL of the last.fm API. In the URL Builder you can add as many query parameters as you want. In order to make the parameters configurable you can hook in Text Input fields. For this functionality I added text input for the artist name and for the API key. This way other users can also use my generated pipe if they like to.

Fetch Data
The URL Builder is connected to the input of a Fetch Data block. What this block does is read in the information from the URL build by the URL Builder and parses it for you. In the “Path to item list” you can already select what exact data it should get from the XML feed. In my case I want all similarartist.artist.

But that still leaves me with the problem that the first Object is just the artist name. In outputting the data that’s really not nice and I want to get rid of that first one.

Split, Count, Simple Math & Tail
In order to get rid of the first item from this list I start of by splitting the output of the Fetch Data block. On the left output of the split I count the number of results. The total amount of items is then put into a Simple Math block. The simple math block then just subtracts 1 from the total count. The other part of the Split block is fed into the Tail block. What the Tail block does is that it outputs only the last items from the provided data. So in the Tail the (total amount – 1) is provided. Why they labeled this field “Emit after position” I don’t know. But I don’t think this is a very proper description.

Sub-element
The last step before outputting the data, is that I only output the item.name. This is because this is the only important element for me at this time and I don’t want to clutter the pipe output with a lot of data that I don’t have any use for.

After saving the Pipe you are then able to connect to the pipe. You can even provide that exact path to use. My pipe can by accessed over here:
http://pipes.yahoo.com/fruitmaster/getsimilarartists?artist=Justice&_render=JSON

Now I do have to say that not all is well in the land of Pipes. A few times the Yahoo online editor didn’t completely get what I was doing and wasn’t working as it should. This was easily resolved by reloading the page. But still it was kind of annoying. Also the documentation is sometimes kinda vague and the preview data in the editor also sometimes fails. But all in all, it’s a nice piece of software that Yahoo set up!

This entry was posted on Sunday, February 15th, 2009 and is filed under JavaScript. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.