A simple rss reader
Tue Dec 3, 2019 · 874 words · 5 min

A while ago I went looking for good rss / atom readers. I had a few requirements:

The first software I found that might fit my requirements was Newsboat. It has a lot of features and is actively maintained. The main problem with newsboat is that it has too many features. A filter language, killfiles (?), query feeds, podcast support, interfaces for multiple online feedreaders, etc. I.e. a bunch of bloat that I will never use. To top it all off, it's written in C++. Funnily enough, in the midst of all those features I couldn't find a way to download multiple feeds in parallel. Every time I wanted to fetch new feeds it sequentially downloaded each one in my list. Another anti-feature was the lack of feed groups. To be fair I didn't spend that much time with the software so I may have just missed those two features, but it was already too late for Newsboat.

Next, I took a look at snownews. I didn't actually compile it, but after a cursory review, I deemed it too complicated as well. Why did it take around 7K lines of C code (including http digest handling code?!) to accomplish what should be a simple task? I think that snownews tries to do to much, when there are already great tools written to do things like download files and render html to the terminal. I knew there was a better way.

So, about 4 hours later, I came up with rss, a barebones feed reader. Its dependencies are curl, to fetch the feed, xsltproc, to parse the feed, and w3m to render the feed's posts. Of those curl and xsltproc are pretty universal. Additionally w3m is likely to be installed by heavy terminal users already, so I didn't consider depending on it a deal breaker. If someone knows a better html viewer for the terminal, let me know. Also, rss's manpage is generated by scdoc, but that counts as an optional dependency :^).

All of the heavy lifting is done by xsltproc and this stylesheet. xslt was an interesting language to work with, but it was definitely the right tool for the job. With just a 60 line stylesheet I could seperate a single xml feed into a file / directory structure like this:

Ruby News
`-- 2019
    |-- 08
    |   `-- 28
    |       |-- 09:00:00 - Ruby 2.5.6 Released
    |       `-- 09:00:00 - Ruby 2.6.4 Released
    `-- 10
        |-- 01
        |   |-- 11:00:00 - CVE-2019-15845: A NUL injection vulnerability of File.fnmatch and Fil
        |   |-- 11:00:00 - CVE-2019-16201: Regular Expression Denial of Service vulnerability of
        |   |-- 11:00:00 - CVE-2019-16254: HTTP response splitting in WEBrick (Additional fix)
        |   |-- 11:00:00 - CVE-2019-16255: A code injection vulnerability of Shell#[] and Shell#
        |   |-- 11:00:00 - Ruby 2.4.8 Released
        |   |-- 11:00:00 - Ruby 2.5.7 Released
        |   `-- 11:00:00 - Ruby 2.6.5 Released
        `-- 02
            `-- 09:00:00 - Ruby 2.4.9 Released

(Ruby's news feed)

The remainder of the features are implemented with basic unix utilities. For instance, the "read" list, is just a list of paths that gets appended to as you read posts. In order to find unread posts, a list of all posts is generated with find, concatenated with the "read" list, sorted, and uniq -u'd.

I was able to cut down on complexity in other places too. In order to specify the feeds you want to subscribe to, you write a shell script that calls the function feed. rss will source that script when necessary and the resulting calls to feed actually fetch the feed directly.

It was also trivial to add parallel fetching using shell job control. I was able to enforce a limit on concurrent jobs using the jobs builtin and wc.

while [ $(jobs -p | wc -l) -ge $job_limit ]; do
  sleep 0.5
done

The final innovation was to allow rss to be extensible. This was accomplished by having a second shell script that gets sourced on startup. Functions defined in the extension script that begin with rss_ can be called from the command line. These functions also have the entirety of rss to call themselves. Using the primitives provided by rss it is easy to implement a command that fetches and then lists new posts, or fetches with a different concurrent limit.

All of these features are accomplished in around 200 lines of POSIX shell script, so if your interest is piqued, then feel free to dive in. Use it as is or as a starting point for your own ultimate feed reader! I've been using rss with a few extension functions for a couple of months now and so far it has accomplished everything I need.

Note: as of writing the provided zsh completions sort of work and the bash completions don't at all. Submit a patch!

In case you missed it, here is a link to the repo: https://git.sr.ht/~lattis/rss


posts · about · home