Dave Winer suggests a complex scheme (not unlike the Radio coffee mug links) involving localhost urls and arbitrary ports for OS rss subscription management.
This seems overly complex, especially if you've got control of the browser. Much simpler would be for the browser to recognize an rss file (by mime type, or possibly involving some slightly cleverer handling of xml) and hand the URL off to your designated subscription manager. I'm pretty sure I could do this now with only configuration files (other than adding command line subscription support to my rss reader).
(Yes, I've noticed that my syndicate link comes back as text/plain. I expect to be fixing that in the near future. :-)
Hmm, you could *almost* use CSS to override something like this in a per-user way, if it were tagged right (but it would be "wrong" without an override, and even more complex.) Really, mime-type first, with a fallback to extension (ie. the spec should tell the server to do both, the client does whichever it can) should be quite enough.