neostephenism

Rebuilding a Broken RSS Feed

One of the promises of RSS is that if a website publishes articles, you can subscribe to them. In practice, that only works if the site's RSS feed is actually maintained.

While building my RSS reader, I discovered that 2nd Life Media Alamogordo's RSS feed wasn't publishing their articles at all—it was serving content from an entirely different customer. Since I wanted the publication in my daily reading queue, I decided to fix the problem myself.

The result is 2nd Life Media Feed, a small Python project that reconstructs a proper RSS feed directly from the public website.

Instead of relying on the site's broken XML, the script:

Because it tracks previously-seen URLs, rerunning the scraper is inexpensive. Most runs simply check for updates and exit without downloading anything new.

Once the feed is generated, a scheduled task automatically publishes it to GitHub, where it's served through jsDelivr so RSS readers receive it with the correct XML content type. The entire pipeline runs automatically every hour.

This turned into a much more interesting engineering problem than I expected.

Along the way I ran into:

None of those bugs were in the scraper itself. They were all delivery problems that only became visible after the scraper was already working.

I think projects like this are one of the underrated strengths of the open web. If a website doesn't expose its content in a usable way, you're not forced to wait for the publisher to fix it. If the information is already public, you can often build the missing layer yourself.

In this case, that missing layer was just RSS.

The project is open source:

https://github.com/shibpshman/2ndlife-media-feed

It's a small utility, but it's also a reminder of why I still like RSS. When something breaks, the protocol is simple enough that you can usually repair it yourself instead of waiting for a platform to decide your problem is worth solving.

How It Works

The scraper is intentionally simple. It doesn't try to reverse engineer APIs or automate a browser, it just consumes the same HTML any visitor receives.

RSS Pipeline

The scraper itself is only a few hundred lines of Python. It starts by downloading the homepage, extracting every URL that matches the site's article pattern, and comparing those links against a JSON state file containing every article it's already archived. That means subsequent runs only fetch genuinely new content instead of rescanning the entire site.
Each new article is then parsed for:

Rather than assuming the site's HTML is perfectly consistent, the parser tries several increasingly generic selectors before falling back to harvesting meaningful paragraphs. That makes it fairly resilient to minor layout changes.
Once every article has been cached, the script rebuilds an RSS 2.0 feed from the fifty most recent entries using feedgen. Because the feed is generated from the cache rather than today's scrape alone, older articles remain available even if they disappear from the homepage.

I also wrote a separate one-time archive scraper. Instead of looking only at the homepage, it walks the site's paginated archive, downloads every historical article, converts the HTML into plain text, and stores the results in a date-organized archive. That let me backfill hundreds of older stories before switching to the incremental scraper for day-to-day updates.
The end result is a feed that behaves exactly like a native RSS feed, even though the website never provided one.