I’ve been having a fun time with the idea of pulling information from other sources into my website, and I think this morning I was able to put together a toolkit that makes it fairly easy going forward.
Just about all of the information sources I might want to pull from provide RSS feeds, so I started looking at Yahoo Pipes as a way to filter, adjust, and otherwise correct feeds before they’re published here. It turns out that Pipes is far more powerful than I expected, and it’s fun to use, too. It does have the same kind of feel as piping data through a bunch of Unix commands, but it also has a bit of a functional programming feel to it too (although I tend to regard anything that has map and reduce as functional-like).
In any case, my first creation is a pipe to process an RSS feed from github. I use github as a hosting service for open source or otherwise public source code that I’m working on, assuming that I’m the one that gets to make that decision.
I decided that including all events from github would be a bit overwhelming, so instead I’m only showing events where I push local changes that I have. I then re-write the title of the post to be a little shorter. You can see the pipe at http://pipes.yahoo.com/ebroder/githubpush. I’ve generalized it a bit so that you can substitute any username.
Finally, to pull in the result of the output from Yahoo Pipes, I use WP-o-Matic to pull the RSS entries into here.
I’m looking forward to poking at some other sites with Yahoo Pipes and seeing how much I can collate into this site.
Oh – a few other details. First, I’m categorizing all entries based on their source. So far, that means “posts”, “github”, or “twitter”. If you only want to see original content on this site, just go to the posts category page.
Second, with the new evil plan to write more, but also restrict access more, it doesn’t make sense any more to crosspost all of my entries publically to LiveJournal. LJ folks, you’ll just have to deal with visiting a site outside your little circle.